@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -2
- package/README.md +39 -0
- package/dist/cjs/auth/index.d.ts +3 -0
- package/dist/cjs/auth/index.js +13 -0
- package/dist/cjs/auth/profile-loader.d.ts +18 -0
- package/dist/cjs/auth/profile-loader.js +208 -0
- package/dist/cjs/client-factory.d.ts +4 -0
- package/dist/cjs/client-factory.js +10 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/index.d.ts +3 -1
- package/dist/cjs/index.js +8 -2
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/auth/index.d.ts +3 -0
- package/dist/esm/auth/index.js +2 -0
- package/dist/esm/auth/profile-loader.d.ts +18 -0
- package/dist/esm/auth/profile-loader.js +169 -0
- package/dist/esm/client-factory.d.ts +4 -0
- package/dist/esm/client-factory.js +9 -0
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/index.d.ts +3 -1
- package/dist/esm/index.js +2 -1
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/dist/types/auth/index.d.ts +3 -0
- package/dist/types/auth/profile-loader.d.ts +18 -0
- package/dist/types/client-factory.d.ts +4 -0
- package/dist/types/index.d.ts +3 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -482
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
|
@@ -1,430 +1,430 @@
|
|
|
1
|
-
# Module 1: Foundations
|
|
2
|
-
|
|
3
|
-
**Level:** Beginner
|
|
4
|
-
**Estimated Time:** 15 minutes
|
|
5
|
-
|
|
6
|
-
## Overview
|
|
7
|
-
|
|
8
|
-
This module introduces the fundamental concepts of file operations in the Fluent Connect SDK. You'll learn about data source abstractions, when to use different storage systems, and the unified interface pattern that makes working with files consistent across S3, SFTP, and other storage backends.
|
|
9
|
-
|
|
10
|
-
## Learning Objectives
|
|
11
|
-
|
|
12
|
-
By the end of this module, you will:
|
|
13
|
-
|
|
14
|
-
- ✅ Understand the data source abstraction pattern
|
|
15
|
-
- ✅ Know when to use S3 vs SFTP vs local files
|
|
16
|
-
- ✅ Understand the unified interface benefits
|
|
17
|
-
- ✅ Learn the common file operation patterns
|
|
18
|
-
|
|
19
|
-
## What Are Data Sources?
|
|
20
|
-
|
|
21
|
-
In the SDK, a **data source** is an abstraction layer that provides a consistent interface for file operations across different storage systems. Instead of learning separate APIs for S3, SFTP, FTP, local files, etc., you use the same methods regardless of where files are stored.
|
|
22
|
-
|
|
23
|
-
### The Problem Without Abstraction
|
|
24
|
-
|
|
25
|
-
```typescript
|
|
26
|
-
// ❌ WITHOUT SDK: Every storage system has different API
|
|
27
|
-
import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
|
|
28
|
-
import * as SftpClient from 'ssh2-sftp-client';
|
|
29
|
-
import * as fs from 'fs/promises';
|
|
30
|
-
|
|
31
|
-
// S3 - complex AWS SDK
|
|
32
|
-
const s3 = new S3Client({ region: 'us-east-1' });
|
|
33
|
-
const s3Result = await s3.send(
|
|
34
|
-
new GetObjectCommand({
|
|
35
|
-
Bucket: 'my-bucket',
|
|
36
|
-
Key: 'file.csv',
|
|
37
|
-
})
|
|
38
|
-
);
|
|
39
|
-
const s3Body = await s3Result.Body?.transformToString();
|
|
40
|
-
|
|
41
|
-
// SFTP - SDK SftpDataSource
|
|
42
|
-
const sftp = new SftpDataSource(
|
|
43
|
-
{
|
|
44
|
-
type: 'SFTP_CSV',
|
|
45
|
-
connectionId: 'my-sftp',
|
|
46
|
-
name: 'My SFTP',
|
|
47
|
-
settings: {
|
|
48
|
-
host: 'example.com',
|
|
49
|
-
username: 'user',
|
|
50
|
-
password: 'pass',
|
|
51
|
-
},
|
|
52
|
-
},
|
|
53
|
-
logger
|
|
54
|
-
);
|
|
55
|
-
// No connect needed - handled internally
|
|
56
|
-
const sftpResult = await sftp.downloadFile('/remote/file.csv');
|
|
57
|
-
|
|
58
|
-
// Local files - yet another API
|
|
59
|
-
const localResult = await fs.readFile('/local/file.csv', 'utf-8');
|
|
60
|
-
|
|
61
|
-
// Three completely different APIs for the same operation!
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
### The Solution: Unified Interface
|
|
65
|
-
|
|
66
|
-
```typescript
|
|
67
|
-
// ✅ WITH SDK: Same interface for all storage systems
|
|
68
|
-
import {
|
|
69
|
-
S3DataSource,
|
|
70
|
-
SftpDataSource,
|
|
71
|
-
createConsoleLogger,
|
|
72
|
-
toStructuredLogger
|
|
73
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
74
|
-
|
|
75
|
-
const logger = createConsoleLogger();
|
|
76
|
-
|
|
77
|
-
// S3 data source
|
|
78
|
-
const s3Source = new S3DataSource(
|
|
79
|
-
{
|
|
80
|
-
type: 'S3_CSV',
|
|
81
|
-
connectionId: 'my-s3',
|
|
82
|
-
name: 'My S3 Bucket',
|
|
83
|
-
s3Config: {
|
|
84
|
-
region: 'us-east-1',
|
|
85
|
-
bucket: 'my-bucket',
|
|
86
|
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
|
|
87
|
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
|
|
88
|
-
},
|
|
89
|
-
},
|
|
90
|
-
logger
|
|
91
|
-
);
|
|
92
|
-
|
|
93
|
-
// SFTP data source
|
|
94
|
-
const sftpSource = new SftpDataSource(
|
|
95
|
-
{
|
|
96
|
-
type: 'SFTP_CSV',
|
|
97
|
-
connectionId: 'my-sftp',
|
|
98
|
-
name: 'My SFTP Server',
|
|
99
|
-
settings: {
|
|
100
|
-
host: 'example.com',
|
|
101
|
-
username: 'user',
|
|
102
|
-
password: 'pass',
|
|
103
|
-
},
|
|
104
|
-
},
|
|
105
|
-
logger
|
|
106
|
-
);
|
|
107
|
-
|
|
108
|
-
// Same method signature for both!
|
|
109
|
-
const s3File = await s3Source.downloadFile('file.csv');
|
|
110
|
-
const sftpFile = await sftpSource.downloadFile('file.csv');
|
|
111
|
-
|
|
112
|
-
// ✅ Consistent, predictable, easy to swap
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
## When to Use Each Storage System
|
|
116
|
-
|
|
117
|
-
### S3 (AWS Simple Storage Service)
|
|
118
|
-
|
|
119
|
-
**Best For:**
|
|
120
|
-
|
|
121
|
-
- Cloud-native applications
|
|
122
|
-
- Serverless workflows (Lambda, Versori)
|
|
123
|
-
- Large datasets and data lakes
|
|
124
|
-
- High availability requirements
|
|
125
|
-
- Scalable storage needs
|
|
126
|
-
- Public or presigned URL access
|
|
127
|
-
|
|
128
|
-
**Advantages:**
|
|
129
|
-
|
|
130
|
-
- ✅ Unlimited scalability
|
|
131
|
-
- ✅ High durability (99.999999999%)
|
|
132
|
-
- ✅ Low cost for large datasets
|
|
133
|
-
- ✅ Presigned URLs for temporary access
|
|
134
|
-
- ✅ Event notifications (S3 → Lambda)
|
|
135
|
-
- ✅ No connection pooling needed
|
|
136
|
-
|
|
137
|
-
**Disadvantages:**
|
|
138
|
-
|
|
139
|
-
- ❌ Requires AWS account
|
|
140
|
-
- ❌ Internet dependency
|
|
141
|
-
- ❌ Per-request pricing model
|
|
142
|
-
- ❌ No native "move" operation (copy + delete)
|
|
143
|
-
|
|
144
|
-
**Use Case Examples:**
|
|
145
|
-
|
|
146
|
-
```typescript
|
|
147
|
-
// Data lake ingestion
|
|
148
|
-
await s3.uploadFile('data-lake/orders/2025-01-15.parquet', parquetData);
|
|
149
|
-
|
|
150
|
-
// Serverless processing
|
|
151
|
-
const csvFiles = await s3.listFiles({ prefix: 'incoming/', maxKeys: 1000 });
|
|
152
|
-
// Filter CSV files after listing (pattern matching not supported in S3 listFiles)
|
|
153
|
-
const csvOnly = csvFiles.filter(f => f.name.endsWith('.csv'));
|
|
154
|
-
|
|
155
|
-
// Presigned URL generation
|
|
156
|
-
const url = await s3PresignService.generatePresignedDownloadUrl('report.pdf', 3600);
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
### SFTP (SSH File Transfer Protocol)
|
|
160
|
-
|
|
161
|
-
**Best For:**
|
|
162
|
-
|
|
163
|
-
- B2B integrations with partners
|
|
164
|
-
- EDI system integrations
|
|
165
|
-
- Secure file transfers
|
|
166
|
-
- Compliance requirements (HIPAA, PCI-DSS)
|
|
167
|
-
- Systems with existing SFTP infrastructure
|
|
168
|
-
|
|
169
|
-
**Advantages:**
|
|
170
|
-
|
|
171
|
-
- ✅ Industry standard for B2B
|
|
172
|
-
- ✅ Strong encryption (SSH)
|
|
173
|
-
- ✅ Widely supported
|
|
174
|
-
- ✅ Native move/rename operations
|
|
175
|
-
- ✅ Directory structure support
|
|
176
|
-
- ✅ Username/password or key-based auth
|
|
177
|
-
|
|
178
|
-
**Disadvantages:**
|
|
179
|
-
|
|
180
|
-
- ❌ Connection management overhead
|
|
181
|
-
- ❌ Requires server maintenance
|
|
182
|
-
- ❌ Connection pooling complexity
|
|
183
|
-
- ❌ Network latency sensitivity
|
|
184
|
-
- ❌ Manual scaling
|
|
185
|
-
|
|
186
|
-
**Use Case Examples:**
|
|
187
|
-
|
|
188
|
-
```typescript
|
|
189
|
-
// Partner file drop
|
|
190
|
-
const partnerFiles = await sftp.listFiles(); // Pattern from config
|
|
191
|
-
for (const file of partnerFiles) {
|
|
192
|
-
const content = await sftp.downloadFile(file.path);
|
|
193
|
-
await processPartnerData(content);
|
|
194
|
-
await sftp.moveFile(file.path, `/archive/${file.name}`);
|
|
195
|
-
}
|
|
196
|
-
|
|
197
|
-
// Compliance-required audit trail
|
|
198
|
-
await sftp.uploadFile('/outgoing/audit-report.txt', reportData);
|
|
199
|
-
```
|
|
200
|
-
|
|
201
|
-
### Local Files
|
|
202
|
-
|
|
203
|
-
**Best For:**
|
|
204
|
-
|
|
205
|
-
- Development and testing
|
|
206
|
-
- Single-machine scripts
|
|
207
|
-
- Small datasets
|
|
208
|
-
- Quick prototypes
|
|
209
|
-
|
|
210
|
-
**Advantages:**
|
|
211
|
-
|
|
212
|
-
- ✅ No external dependencies
|
|
213
|
-
- ✅ Fast access
|
|
214
|
-
- ✅ Simple setup
|
|
215
|
-
|
|
216
|
-
**Disadvantages:**
|
|
217
|
-
|
|
218
|
-
- ❌ Not scalable
|
|
219
|
-
- ❌ No high availability
|
|
220
|
-
- ❌ Single point of failure
|
|
221
|
-
|
|
222
|
-
## Data Source Comparison
|
|
223
|
-
|
|
224
|
-
| Feature | S3 | SFTP | Local Files |
|
|
225
|
-
| ---------------------- | ------------- | ------------------ | ------------------- |
|
|
226
|
-
| **Scalability** | Unlimited | Limited by server | Limited by disk |
|
|
227
|
-
| **Availability** | 99.99% SLA | Depends on server | Single machine |
|
|
228
|
-
| **Cost** | Pay per use | Server + bandwidth | Infrastructure only |
|
|
229
|
-
| **Setup Complexity** | Medium | High | Low |
|
|
230
|
-
| **B2B Integration** | Rare | Common | N/A |
|
|
231
|
-
| **Cloud-Native** | Yes | No | No |
|
|
232
|
-
| **Connection Pooling** | N/A | Required | N/A |
|
|
233
|
-
| **Move Operation** | Copy + Delete | Native | Native |
|
|
234
|
-
| **Presigned URLs** | Yes | No | No |
|
|
235
|
-
|
|
236
|
-
## Unified Interface Pattern
|
|
237
|
-
|
|
238
|
-
All data sources in the SDK implement a common interface with these core operations:
|
|
239
|
-
|
|
240
|
-
### Core File Operations
|
|
241
|
-
|
|
242
|
-
```typescript
|
|
243
|
-
interface DataSource {
|
|
244
|
-
// List files matching criteria
|
|
245
|
-
listFiles(options?: ListOptions): Promise<FileInfo[]>;
|
|
246
|
-
|
|
247
|
-
// Read file content
|
|
248
|
-
downloadFile(path: string): Promise<string>;
|
|
249
|
-
|
|
250
|
-
// Write file content
|
|
251
|
-
uploadFile(path: string, content: string | Buffer): Promise<void>;
|
|
252
|
-
|
|
253
|
-
// Move file (rename or relocate)
|
|
254
|
-
moveFile(source: string, dest: string, createDirs?: boolean): Promise<void>;
|
|
255
|
-
|
|
256
|
-
// Copy file
|
|
257
|
-
copyFile(source: string, dest: string): Promise<void>;
|
|
258
|
-
|
|
259
|
-
// Delete file
|
|
260
|
-
deleteFile(path: string): Promise<void>;
|
|
261
|
-
|
|
262
|
-
// Check file existence
|
|
263
|
-
fileExists(path: string): Promise<boolean>;
|
|
264
|
-
|
|
265
|
-
// Clean up resources
|
|
266
|
-
dispose(): Promise<void>;
|
|
267
|
-
}
|
|
268
|
-
```
|
|
269
|
-
|
|
270
|
-
### Benefits of Unified Interface
|
|
271
|
-
|
|
272
|
-
1. **Portability**: Swap S3 for SFTP without code changes
|
|
273
|
-
2. **Testability**: Mock data sources easily
|
|
274
|
-
3. **Consistency**: Same method names, same error handling
|
|
275
|
-
4. **Composability**: Build higher-level abstractions
|
|
276
|
-
5. **Documentation**: Learn once, use everywhere
|
|
277
|
-
|
|
278
|
-
## Common File Operation Patterns
|
|
279
|
-
|
|
280
|
-
### Pattern 1: Read-Process-Archive
|
|
281
|
-
|
|
282
|
-
The most common workflow in data integration:
|
|
283
|
-
|
|
284
|
-
```typescript
|
|
285
|
-
// 1. List files to process
|
|
286
|
-
const files = await dataSource.listFiles();
|
|
287
|
-
|
|
288
|
-
// 2. Process each file
|
|
289
|
-
for (const file of files) {
|
|
290
|
-
// 2a. Read file
|
|
291
|
-
const content = await dataSource.downloadFile(file.path);
|
|
292
|
-
|
|
293
|
-
// 2b. Process data
|
|
294
|
-
await processData(content);
|
|
295
|
-
|
|
296
|
-
// 2c. Archive to prevent reprocessing
|
|
297
|
-
const archivePath = `/archive/${file.name}`;
|
|
298
|
-
await dataSource.moveFile(file.path, archivePath, true);
|
|
299
|
-
}
|
|
300
|
-
```
|
|
301
|
-
|
|
302
|
-
### Pattern 2: Transform-and-Upload
|
|
303
|
-
|
|
304
|
-
Process data and write to storage:
|
|
305
|
-
|
|
306
|
-
```typescript
|
|
307
|
-
// 1. Transform data
|
|
308
|
-
const jsonData = await transformToJSON(sourceData);
|
|
309
|
-
|
|
310
|
-
// 2. Upload result
|
|
311
|
-
const outputPath = `output/${new Date().toISOString()}.json`;
|
|
312
|
-
await dataSource.uploadFile(outputPath, JSON.stringify(jsonData));
|
|
313
|
-
```
|
|
314
|
-
|
|
315
|
-
### Pattern 3: Multi-Folder Processing
|
|
316
|
-
|
|
317
|
-
Process files from multiple logical folders:
|
|
318
|
-
|
|
319
|
-
```typescript
|
|
320
|
-
const folders = ['orders', 'inventory', 'customers'];
|
|
321
|
-
|
|
322
|
-
for (const folder of folders) {
|
|
323
|
-
const files = await dataSource.listFiles({ prefix: folder });
|
|
324
|
-
|
|
325
|
-
for (const file of files) {
|
|
326
|
-
const content = await dataSource.downloadFile(file.name);
|
|
327
|
-
await processByType(folder, content);
|
|
328
|
-
}
|
|
329
|
-
}
|
|
330
|
-
```
|
|
331
|
-
|
|
332
|
-
### Pattern 4: Conditional Processing
|
|
333
|
-
|
|
334
|
-
Only process files meeting certain criteria:
|
|
335
|
-
|
|
336
|
-
```typescript
|
|
337
|
-
const allFiles = await dataSource.listFiles();
|
|
338
|
-
|
|
339
|
-
// Filter by file extension after listing
|
|
340
|
-
const csvFiles = allFiles.filter(f => f.name.endsWith('.csv'));
|
|
341
|
-
|
|
342
|
-
for (const file of csvFiles) {
|
|
343
|
-
// Only process files from today
|
|
344
|
-
if (file.name.includes(new Date().toISOString().split('T')[0])) {
|
|
345
|
-
const content = await dataSource.downloadFile(file.name);
|
|
346
|
-
await processData(content);
|
|
347
|
-
}
|
|
348
|
-
}
|
|
349
|
-
```
|
|
350
|
-
|
|
351
|
-
## Key Concepts Summary
|
|
352
|
-
|
|
353
|
-
### Data Source Abstraction
|
|
354
|
-
|
|
355
|
-
- Provides unified interface across storage systems
|
|
356
|
-
- Same methods work on S3, SFTP, local files
|
|
357
|
-
- Eliminates vendor lock-in
|
|
358
|
-
|
|
359
|
-
### Storage System Selection
|
|
360
|
-
|
|
361
|
-
- **S3**: Cloud-native, scalable, serverless workflows
|
|
362
|
-
- **SFTP**: B2B integrations, enterprise systems, compliance
|
|
363
|
-
- **Local**: Development, testing, simple scripts
|
|
364
|
-
|
|
365
|
-
### Common Operations
|
|
366
|
-
|
|
367
|
-
- **List**: Find files to process
|
|
368
|
-
- **Download**: Read file content
|
|
369
|
-
- **Upload**: Write file content
|
|
370
|
-
- **Move**: Archive or reorganize
|
|
371
|
-
- **Copy**: Backup or duplicate
|
|
372
|
-
- **Delete**: Clean up temporary files
|
|
373
|
-
|
|
374
|
-
### Common Patterns
|
|
375
|
-
|
|
376
|
-
- Read-Process-Archive (most common)
|
|
377
|
-
- Transform-and-Upload
|
|
378
|
-
- Multi-Folder Processing
|
|
379
|
-
- Conditional Processing
|
|
380
|
-
|
|
381
|
-
## Practice Exercise
|
|
382
|
-
|
|
383
|
-
Try identifying which storage system to use:
|
|
384
|
-
|
|
385
|
-
**Scenario 1**: You need to integrate with a retail partner who sends daily order files via their existing file transfer system.
|
|
386
|
-
|
|
387
|
-
<details>
|
|
388
|
-
<summary>Answer</summary>
|
|
389
|
-
|
|
390
|
-
**SFTP** - B2B integrations typically use SFTP as the industry standard.
|
|
391
|
-
|
|
392
|
-
</details>
|
|
393
|
-
|
|
394
|
-
**Scenario 2**: You're building a serverless data pipeline on AWS Lambda that processes inventory updates from an S3 bucket.
|
|
395
|
-
|
|
396
|
-
<details>
|
|
397
|
-
<summary>Answer</summary>
|
|
398
|
-
|
|
399
|
-
**S3** - Cloud-native, serverless workflows work best with S3.
|
|
400
|
-
|
|
401
|
-
</details>
|
|
402
|
-
|
|
403
|
-
**Scenario 3**: You're writing a one-time migration script to convert 100 CSV files on your laptop to JSON.
|
|
404
|
-
|
|
405
|
-
<details>
|
|
406
|
-
<summary>Answer</summary>
|
|
407
|
-
|
|
408
|
-
**Local Files** - Simple, single-machine tasks don't need cloud storage.
|
|
409
|
-
|
|
410
|
-
</details>
|
|
411
|
-
|
|
412
|
-
## Key Takeaways
|
|
413
|
-
|
|
414
|
-
- 🎯 Data sources provide a unified interface for file operations across storage systems
|
|
415
|
-
- 🎯 Choose S3 for cloud-native scalability, SFTP for B2B integrations, local files for development
|
|
416
|
-
- 🎯 The same methods work on all data sources: `downloadFile()`, `uploadFile()`, `moveFile()`, etc.
|
|
417
|
-
- 🎯 Common patterns: Read-Process-Archive, Transform-and-Upload, Multi-Folder Processing
|
|
418
|
-
- 🎯 Unified interface enables portability, testability, and consistency
|
|
419
|
-
|
|
420
|
-
## Next Steps
|
|
421
|
-
|
|
422
|
-
Continue to [Module 2: Quick Start](./file-operations-02-quick-start.md) to see working examples with S3 and SFTP.
|
|
423
|
-
|
|
424
|
-
---
|
|
425
|
-
|
|
426
|
-
**Related Resources:**
|
|
427
|
-
|
|
428
|
-
- [Quick Reference](../../../02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md) - All operations cheat sheet
|
|
429
|
-
- [S3 Operations](./file-operations-03-s3-operations.md) - Complete S3 guide
|
|
430
|
-
- [SFTP Operations](./file-operations-04-sftp-operations.md) - Complete SFTP guide
|
|
1
|
+
# Module 1: Foundations
|
|
2
|
+
|
|
3
|
+
**Level:** Beginner
|
|
4
|
+
**Estimated Time:** 15 minutes
|
|
5
|
+
|
|
6
|
+
## Overview
|
|
7
|
+
|
|
8
|
+
This module introduces the fundamental concepts of file operations in the Fluent Connect SDK. You'll learn about data source abstractions, when to use different storage systems, and the unified interface pattern that makes working with files consistent across S3, SFTP, and other storage backends.
|
|
9
|
+
|
|
10
|
+
## Learning Objectives
|
|
11
|
+
|
|
12
|
+
By the end of this module, you will:
|
|
13
|
+
|
|
14
|
+
- ✅ Understand the data source abstraction pattern
|
|
15
|
+
- ✅ Know when to use S3 vs SFTP vs local files
|
|
16
|
+
- ✅ Understand the unified interface benefits
|
|
17
|
+
- ✅ Learn the common file operation patterns
|
|
18
|
+
|
|
19
|
+
## What Are Data Sources?
|
|
20
|
+
|
|
21
|
+
In the SDK, a **data source** is an abstraction layer that provides a consistent interface for file operations across different storage systems. Instead of learning separate APIs for S3, SFTP, FTP, local files, etc., you use the same methods regardless of where files are stored.
|
|
22
|
+
|
|
23
|
+
### The Problem Without Abstraction
|
|
24
|
+
|
|
25
|
+
```typescript
|
|
26
|
+
// ❌ WITHOUT SDK: Every storage system has different API
|
|
27
|
+
import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
|
|
28
|
+
import * as SftpClient from 'ssh2-sftp-client';
|
|
29
|
+
import * as fs from 'fs/promises';
|
|
30
|
+
|
|
31
|
+
// S3 - complex AWS SDK
|
|
32
|
+
const s3 = new S3Client({ region: 'us-east-1' });
|
|
33
|
+
const s3Result = await s3.send(
|
|
34
|
+
new GetObjectCommand({
|
|
35
|
+
Bucket: 'my-bucket',
|
|
36
|
+
Key: 'file.csv',
|
|
37
|
+
})
|
|
38
|
+
);
|
|
39
|
+
const s3Body = await s3Result.Body?.transformToString();
|
|
40
|
+
|
|
41
|
+
// SFTP - SDK SftpDataSource
|
|
42
|
+
const sftp = new SftpDataSource(
|
|
43
|
+
{
|
|
44
|
+
type: 'SFTP_CSV',
|
|
45
|
+
connectionId: 'my-sftp',
|
|
46
|
+
name: 'My SFTP',
|
|
47
|
+
settings: {
|
|
48
|
+
host: 'example.com',
|
|
49
|
+
username: 'user',
|
|
50
|
+
password: 'pass',
|
|
51
|
+
},
|
|
52
|
+
},
|
|
53
|
+
logger
|
|
54
|
+
);
|
|
55
|
+
// No connect needed - handled internally
|
|
56
|
+
const sftpResult = await sftp.downloadFile('/remote/file.csv');
|
|
57
|
+
|
|
58
|
+
// Local files - yet another API
|
|
59
|
+
const localResult = await fs.readFile('/local/file.csv', 'utf-8');
|
|
60
|
+
|
|
61
|
+
// Three completely different APIs for the same operation!
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
### The Solution: Unified Interface
|
|
65
|
+
|
|
66
|
+
```typescript
|
|
67
|
+
// ✅ WITH SDK: Same interface for all storage systems
|
|
68
|
+
import {
|
|
69
|
+
S3DataSource,
|
|
70
|
+
SftpDataSource,
|
|
71
|
+
createConsoleLogger,
|
|
72
|
+
toStructuredLogger
|
|
73
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
74
|
+
|
|
75
|
+
const logger = createConsoleLogger();
|
|
76
|
+
|
|
77
|
+
// S3 data source
|
|
78
|
+
const s3Source = new S3DataSource(
|
|
79
|
+
{
|
|
80
|
+
type: 'S3_CSV',
|
|
81
|
+
connectionId: 'my-s3',
|
|
82
|
+
name: 'My S3 Bucket',
|
|
83
|
+
s3Config: {
|
|
84
|
+
region: 'us-east-1',
|
|
85
|
+
bucket: 'my-bucket',
|
|
86
|
+
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
|
|
87
|
+
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
|
|
88
|
+
},
|
|
89
|
+
},
|
|
90
|
+
logger
|
|
91
|
+
);
|
|
92
|
+
|
|
93
|
+
// SFTP data source
|
|
94
|
+
const sftpSource = new SftpDataSource(
|
|
95
|
+
{
|
|
96
|
+
type: 'SFTP_CSV',
|
|
97
|
+
connectionId: 'my-sftp',
|
|
98
|
+
name: 'My SFTP Server',
|
|
99
|
+
settings: {
|
|
100
|
+
host: 'example.com',
|
|
101
|
+
username: 'user',
|
|
102
|
+
password: 'pass',
|
|
103
|
+
},
|
|
104
|
+
},
|
|
105
|
+
logger
|
|
106
|
+
);
|
|
107
|
+
|
|
108
|
+
// Same method signature for both!
|
|
109
|
+
const s3File = await s3Source.downloadFile('file.csv');
|
|
110
|
+
const sftpFile = await sftpSource.downloadFile('file.csv');
|
|
111
|
+
|
|
112
|
+
// ✅ Consistent, predictable, easy to swap
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
## When to Use Each Storage System
|
|
116
|
+
|
|
117
|
+
### S3 (AWS Simple Storage Service)
|
|
118
|
+
|
|
119
|
+
**Best For:**
|
|
120
|
+
|
|
121
|
+
- Cloud-native applications
|
|
122
|
+
- Serverless workflows (Lambda, Versori)
|
|
123
|
+
- Large datasets and data lakes
|
|
124
|
+
- High availability requirements
|
|
125
|
+
- Scalable storage needs
|
|
126
|
+
- Public or presigned URL access
|
|
127
|
+
|
|
128
|
+
**Advantages:**
|
|
129
|
+
|
|
130
|
+
- ✅ Unlimited scalability
|
|
131
|
+
- ✅ High durability (99.999999999%)
|
|
132
|
+
- ✅ Low cost for large datasets
|
|
133
|
+
- ✅ Presigned URLs for temporary access
|
|
134
|
+
- ✅ Event notifications (S3 → Lambda)
|
|
135
|
+
- ✅ No connection pooling needed
|
|
136
|
+
|
|
137
|
+
**Disadvantages:**
|
|
138
|
+
|
|
139
|
+
- ❌ Requires AWS account
|
|
140
|
+
- ❌ Internet dependency
|
|
141
|
+
- ❌ Per-request pricing model
|
|
142
|
+
- ❌ No native "move" operation (copy + delete)
|
|
143
|
+
|
|
144
|
+
**Use Case Examples:**
|
|
145
|
+
|
|
146
|
+
```typescript
|
|
147
|
+
// Data lake ingestion
|
|
148
|
+
await s3.uploadFile('data-lake/orders/2025-01-15.parquet', parquetData);
|
|
149
|
+
|
|
150
|
+
// Serverless processing
|
|
151
|
+
const csvFiles = await s3.listFiles({ prefix: 'incoming/', maxKeys: 1000 });
|
|
152
|
+
// Filter CSV files after listing (pattern matching not supported in S3 listFiles)
|
|
153
|
+
const csvOnly = csvFiles.filter(f => f.name.endsWith('.csv'));
|
|
154
|
+
|
|
155
|
+
// Presigned URL generation
|
|
156
|
+
const url = await s3PresignService.generatePresignedDownloadUrl('report.pdf', 3600);
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
### SFTP (SSH File Transfer Protocol)
|
|
160
|
+
|
|
161
|
+
**Best For:**
|
|
162
|
+
|
|
163
|
+
- B2B integrations with partners
|
|
164
|
+
- EDI system integrations
|
|
165
|
+
- Secure file transfers
|
|
166
|
+
- Compliance requirements (HIPAA, PCI-DSS)
|
|
167
|
+
- Systems with existing SFTP infrastructure
|
|
168
|
+
|
|
169
|
+
**Advantages:**
|
|
170
|
+
|
|
171
|
+
- ✅ Industry standard for B2B
|
|
172
|
+
- ✅ Strong encryption (SSH)
|
|
173
|
+
- ✅ Widely supported
|
|
174
|
+
- ✅ Native move/rename operations
|
|
175
|
+
- ✅ Directory structure support
|
|
176
|
+
- ✅ Username/password or key-based auth
|
|
177
|
+
|
|
178
|
+
**Disadvantages:**
|
|
179
|
+
|
|
180
|
+
- ❌ Connection management overhead
|
|
181
|
+
- ❌ Requires server maintenance
|
|
182
|
+
- ❌ Connection pooling complexity
|
|
183
|
+
- ❌ Network latency sensitivity
|
|
184
|
+
- ❌ Manual scaling
|
|
185
|
+
|
|
186
|
+
**Use Case Examples:**
|
|
187
|
+
|
|
188
|
+
```typescript
|
|
189
|
+
// Partner file drop
|
|
190
|
+
const partnerFiles = await sftp.listFiles(); // Pattern from config
|
|
191
|
+
for (const file of partnerFiles) {
|
|
192
|
+
const content = await sftp.downloadFile(file.path);
|
|
193
|
+
await processPartnerData(content);
|
|
194
|
+
await sftp.moveFile(file.path, `/archive/${file.name}`);
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
// Compliance-required audit trail
|
|
198
|
+
await sftp.uploadFile('/outgoing/audit-report.txt', reportData);
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
### Local Files
|
|
202
|
+
|
|
203
|
+
**Best For:**
|
|
204
|
+
|
|
205
|
+
- Development and testing
|
|
206
|
+
- Single-machine scripts
|
|
207
|
+
- Small datasets
|
|
208
|
+
- Quick prototypes
|
|
209
|
+
|
|
210
|
+
**Advantages:**
|
|
211
|
+
|
|
212
|
+
- ✅ No external dependencies
|
|
213
|
+
- ✅ Fast access
|
|
214
|
+
- ✅ Simple setup
|
|
215
|
+
|
|
216
|
+
**Disadvantages:**
|
|
217
|
+
|
|
218
|
+
- ❌ Not scalable
|
|
219
|
+
- ❌ No high availability
|
|
220
|
+
- ❌ Single point of failure
|
|
221
|
+
|
|
222
|
+
## Data Source Comparison
|
|
223
|
+
|
|
224
|
+
| Feature | S3 | SFTP | Local Files |
|
|
225
|
+
| ---------------------- | ------------- | ------------------ | ------------------- |
|
|
226
|
+
| **Scalability** | Unlimited | Limited by server | Limited by disk |
|
|
227
|
+
| **Availability** | 99.99% SLA | Depends on server | Single machine |
|
|
228
|
+
| **Cost** | Pay per use | Server + bandwidth | Infrastructure only |
|
|
229
|
+
| **Setup Complexity** | Medium | High | Low |
|
|
230
|
+
| **B2B Integration** | Rare | Common | N/A |
|
|
231
|
+
| **Cloud-Native** | Yes | No | No |
|
|
232
|
+
| **Connection Pooling** | N/A | Required | N/A |
|
|
233
|
+
| **Move Operation** | Copy + Delete | Native | Native |
|
|
234
|
+
| **Presigned URLs** | Yes | No | No |
|
|
235
|
+
|
|
236
|
+
## Unified Interface Pattern
|
|
237
|
+
|
|
238
|
+
All data sources in the SDK implement a common interface with these core operations:
|
|
239
|
+
|
|
240
|
+
### Core File Operations
|
|
241
|
+
|
|
242
|
+
```typescript
|
|
243
|
+
interface DataSource {
|
|
244
|
+
// List files matching criteria
|
|
245
|
+
listFiles(options?: ListOptions): Promise<FileInfo[]>;
|
|
246
|
+
|
|
247
|
+
// Read file content
|
|
248
|
+
downloadFile(path: string): Promise<string>;
|
|
249
|
+
|
|
250
|
+
// Write file content
|
|
251
|
+
uploadFile(path: string, content: string | Buffer): Promise<void>;
|
|
252
|
+
|
|
253
|
+
// Move file (rename or relocate)
|
|
254
|
+
moveFile(source: string, dest: string, createDirs?: boolean): Promise<void>;
|
|
255
|
+
|
|
256
|
+
// Copy file
|
|
257
|
+
copyFile(source: string, dest: string): Promise<void>;
|
|
258
|
+
|
|
259
|
+
// Delete file
|
|
260
|
+
deleteFile(path: string): Promise<void>;
|
|
261
|
+
|
|
262
|
+
// Check file existence
|
|
263
|
+
fileExists(path: string): Promise<boolean>;
|
|
264
|
+
|
|
265
|
+
// Clean up resources
|
|
266
|
+
dispose(): Promise<void>;
|
|
267
|
+
}
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
### Benefits of Unified Interface
|
|
271
|
+
|
|
272
|
+
1. **Portability**: Swap S3 for SFTP without code changes
|
|
273
|
+
2. **Testability**: Mock data sources easily
|
|
274
|
+
3. **Consistency**: Same method names, same error handling
|
|
275
|
+
4. **Composability**: Build higher-level abstractions
|
|
276
|
+
5. **Documentation**: Learn once, use everywhere
|
|
277
|
+
|
|
278
|
+
## Common File Operation Patterns
|
|
279
|
+
|
|
280
|
+
### Pattern 1: Read-Process-Archive
|
|
281
|
+
|
|
282
|
+
The most common workflow in data integration:
|
|
283
|
+
|
|
284
|
+
```typescript
|
|
285
|
+
// 1. List files to process
|
|
286
|
+
const files = await dataSource.listFiles();
|
|
287
|
+
|
|
288
|
+
// 2. Process each file
|
|
289
|
+
for (const file of files) {
|
|
290
|
+
// 2a. Read file
|
|
291
|
+
const content = await dataSource.downloadFile(file.path);
|
|
292
|
+
|
|
293
|
+
// 2b. Process data
|
|
294
|
+
await processData(content);
|
|
295
|
+
|
|
296
|
+
// 2c. Archive to prevent reprocessing
|
|
297
|
+
const archivePath = `/archive/${file.name}`;
|
|
298
|
+
await dataSource.moveFile(file.path, archivePath, true);
|
|
299
|
+
}
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
### Pattern 2: Transform-and-Upload
|
|
303
|
+
|
|
304
|
+
Process data and write to storage:
|
|
305
|
+
|
|
306
|
+
```typescript
|
|
307
|
+
// 1. Transform data
|
|
308
|
+
const jsonData = await transformToJSON(sourceData);
|
|
309
|
+
|
|
310
|
+
// 2. Upload result
|
|
311
|
+
const outputPath = `output/${new Date().toISOString()}.json`;
|
|
312
|
+
await dataSource.uploadFile(outputPath, JSON.stringify(jsonData));
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
### Pattern 3: Multi-Folder Processing
|
|
316
|
+
|
|
317
|
+
Process files from multiple logical folders:
|
|
318
|
+
|
|
319
|
+
```typescript
|
|
320
|
+
const folders = ['orders', 'inventory', 'customers'];
|
|
321
|
+
|
|
322
|
+
for (const folder of folders) {
|
|
323
|
+
const files = await dataSource.listFiles({ prefix: folder });
|
|
324
|
+
|
|
325
|
+
for (const file of files) {
|
|
326
|
+
const content = await dataSource.downloadFile(file.name);
|
|
327
|
+
await processByType(folder, content);
|
|
328
|
+
}
|
|
329
|
+
}
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
### Pattern 4: Conditional Processing
|
|
333
|
+
|
|
334
|
+
Only process files meeting certain criteria:
|
|
335
|
+
|
|
336
|
+
```typescript
|
|
337
|
+
const allFiles = await dataSource.listFiles();
|
|
338
|
+
|
|
339
|
+
// Filter by file extension after listing
|
|
340
|
+
const csvFiles = allFiles.filter(f => f.name.endsWith('.csv'));
|
|
341
|
+
|
|
342
|
+
for (const file of csvFiles) {
|
|
343
|
+
// Only process files from today
|
|
344
|
+
if (file.name.includes(new Date().toISOString().split('T')[0])) {
|
|
345
|
+
const content = await dataSource.downloadFile(file.name);
|
|
346
|
+
await processData(content);
|
|
347
|
+
}
|
|
348
|
+
}
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
## Key Concepts Summary
|
|
352
|
+
|
|
353
|
+
### Data Source Abstraction
|
|
354
|
+
|
|
355
|
+
- Provides unified interface across storage systems
|
|
356
|
+
- Same methods work on S3, SFTP, local files
|
|
357
|
+
- Eliminates vendor lock-in
|
|
358
|
+
|
|
359
|
+
### Storage System Selection
|
|
360
|
+
|
|
361
|
+
- **S3**: Cloud-native, scalable, serverless workflows
|
|
362
|
+
- **SFTP**: B2B integrations, enterprise systems, compliance
|
|
363
|
+
- **Local**: Development, testing, simple scripts
|
|
364
|
+
|
|
365
|
+
### Common Operations
|
|
366
|
+
|
|
367
|
+
- **List**: Find files to process
|
|
368
|
+
- **Download**: Read file content
|
|
369
|
+
- **Upload**: Write file content
|
|
370
|
+
- **Move**: Archive or reorganize
|
|
371
|
+
- **Copy**: Backup or duplicate
|
|
372
|
+
- **Delete**: Clean up temporary files
|
|
373
|
+
|
|
374
|
+
### Common Patterns
|
|
375
|
+
|
|
376
|
+
- Read-Process-Archive (most common)
|
|
377
|
+
- Transform-and-Upload
|
|
378
|
+
- Multi-Folder Processing
|
|
379
|
+
- Conditional Processing
|
|
380
|
+
|
|
381
|
+
## Practice Exercise
|
|
382
|
+
|
|
383
|
+
Try identifying which storage system to use:
|
|
384
|
+
|
|
385
|
+
**Scenario 1**: You need to integrate with a retail partner who sends daily order files via their existing file transfer system.
|
|
386
|
+
|
|
387
|
+
<details>
|
|
388
|
+
<summary>Answer</summary>
|
|
389
|
+
|
|
390
|
+
**SFTP** - B2B integrations typically use SFTP as the industry standard.
|
|
391
|
+
|
|
392
|
+
</details>
|
|
393
|
+
|
|
394
|
+
**Scenario 2**: You're building a serverless data pipeline on AWS Lambda that processes inventory updates from an S3 bucket.
|
|
395
|
+
|
|
396
|
+
<details>
|
|
397
|
+
<summary>Answer</summary>
|
|
398
|
+
|
|
399
|
+
**S3** - Cloud-native, serverless workflows work best with S3.
|
|
400
|
+
|
|
401
|
+
</details>
|
|
402
|
+
|
|
403
|
+
**Scenario 3**: You're writing a one-time migration script to convert 100 CSV files on your laptop to JSON.
|
|
404
|
+
|
|
405
|
+
<details>
|
|
406
|
+
<summary>Answer</summary>
|
|
407
|
+
|
|
408
|
+
**Local Files** - Simple, single-machine tasks don't need cloud storage.
|
|
409
|
+
|
|
410
|
+
</details>
|
|
411
|
+
|
|
412
|
+
## Key Takeaways
|
|
413
|
+
|
|
414
|
+
- 🎯 Data sources provide a unified interface for file operations across storage systems
|
|
415
|
+
- 🎯 Choose S3 for cloud-native scalability, SFTP for B2B integrations, local files for development
|
|
416
|
+
- 🎯 The same methods work on all data sources: `downloadFile()`, `uploadFile()`, `moveFile()`, etc.
|
|
417
|
+
- 🎯 Common patterns: Read-Process-Archive, Transform-and-Upload, Multi-Folder Processing
|
|
418
|
+
- 🎯 Unified interface enables portability, testability, and consistency
|
|
419
|
+
|
|
420
|
+
## Next Steps
|
|
421
|
+
|
|
422
|
+
Continue to [Module 2: Quick Start](./file-operations-02-quick-start.md) to see working examples with S3 and SFTP.
|
|
423
|
+
|
|
424
|
+
---
|
|
425
|
+
|
|
426
|
+
**Related Resources:**
|
|
427
|
+
|
|
428
|
+
- [Quick Reference](../../../02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md) - All operations cheat sheet
|
|
429
|
+
- [S3 Operations](./file-operations-03-s3-operations.md) - Complete S3 guide
|
|
430
|
+
- [SFTP Operations](./file-operations-04-sftp-operations.md) - Complete SFTP guide
|