@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -2
- package/README.md +39 -0
- package/dist/cjs/auth/index.d.ts +3 -0
- package/dist/cjs/auth/index.js +13 -0
- package/dist/cjs/auth/profile-loader.d.ts +18 -0
- package/dist/cjs/auth/profile-loader.js +208 -0
- package/dist/cjs/client-factory.d.ts +4 -0
- package/dist/cjs/client-factory.js +10 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/index.d.ts +3 -1
- package/dist/cjs/index.js +8 -2
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/auth/index.d.ts +3 -0
- package/dist/esm/auth/index.js +2 -0
- package/dist/esm/auth/profile-loader.d.ts +18 -0
- package/dist/esm/auth/profile-loader.js +169 -0
- package/dist/esm/client-factory.d.ts +4 -0
- package/dist/esm/client-factory.js +9 -0
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/index.d.ts +3 -1
- package/dist/esm/index.js +2 -1
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/dist/types/auth/index.d.ts +3 -0
- package/dist/types/auth/profile-loader.d.ts +18 -0
- package/dist/types/client-factory.d.ts +4 -0
- package/dist/types/index.d.ts +3 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -482
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md
CHANGED
|
@@ -1,518 +1,518 @@
|
|
|
1
|
-
---
|
|
2
|
-
template_id: tpl-extract-graphql-to-s3-parquet
|
|
3
|
-
canonical_filename: graphql-parquet-extraction.md
|
|
4
|
-
sdk_version: latest
|
|
5
|
-
runtime: versori
|
|
6
|
-
direction: extraction
|
|
7
|
-
source: fluent-graphql
|
|
8
|
-
destination: s3-parquet
|
|
9
|
-
entity: inventory|orders
|
|
10
|
-
format: parquet
|
|
11
|
-
logging: versori
|
|
12
|
-
status: stable
|
|
13
|
-
---
|
|
14
|
-
|
|
15
|
-
# Template: Extraction - GraphQL to S3 Parquet
|
|
16
|
-
|
|
17
|
-
**SDK**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
18
|
-
**Deployment Target:** Versori Platform
|
|
19
|
-
|
|
20
|
-
---
|
|
21
|
-
|
|
22
|
-
---
|
|
23
|
-
|
|
24
|
-
## 📋 Implementation Prompt
|
|
25
|
-
|
|
26
|
-
```
|
|
27
|
-
I need a Versori extractor that:
|
|
28
|
-
|
|
29
|
-
1) Queries Fluent Commerce GraphQL with auto-pagination (Relay cursors)
|
|
30
|
-
2) Supports incremental runs via KV state (date overlap buffer optional)
|
|
31
|
-
3) Transforms results using UniversalMapper per mapping config
|
|
32
|
-
4) Writes partitioned Parquet files to S3 with metadata
|
|
33
|
-
5) Uses native Versori log (LoggingService removed - use native log)
|
|
34
|
-
|
|
35
|
-
Use the loaded docs to fill in SDK specifics and best practices.
|
|
36
|
-
Keep the structure identical to the template; only adapt where needed.
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
---
|
|
40
|
-
|
|
41
|
-
## Versori Workflows Structure
|
|
42
|
-
|
|
43
|
-
**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
|
|
44
|
-
|
|
45
|
-
**Trigger Types:**
|
|
46
|
-
- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
|
|
47
|
-
- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
|
|
48
|
-
- **`http()`** → External API calls (chained from webhook/schedule)
|
|
49
|
-
- **`fn()`** → Internal processing (chained from webhook/schedule)
|
|
50
|
-
|
|
51
|
-
### Recommended Project Structure
|
|
52
|
-
|
|
53
|
-
```
|
|
54
|
-
graphql-parquet-extraction/
|
|
55
|
-
├── index.ts # Entry point - exports all workflows
|
|
56
|
-
└── src/
|
|
57
|
-
├── workflows/
|
|
58
|
-
│ ├── webhook/
|
|
59
|
-
│ │ └── adhoc-extraction.ts # Webhook: Manual extraction
|
|
60
|
-
│ │
|
|
61
|
-
│ └── scheduled/
|
|
62
|
-
│ └── daily-extraction.ts # Scheduled: Daily extraction
|
|
63
|
-
│
|
|
64
|
-
├── services/
|
|
65
|
-
│ └── extraction.service.ts # Shared orchestration logic (reusable)
|
|
66
|
-
│
|
|
67
|
-
└── config/
|
|
68
|
-
└── extraction-config.json # Configuration
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
**Benefits:**
|
|
72
|
-
- ✅ Clear trigger separation (`webhook/` vs `scheduled/`)
|
|
73
|
-
- ✅ Descriptive file names (easy to browse and understand)
|
|
74
|
-
- ✅ Scalable (add new workflows without cluttering)
|
|
75
|
-
- ✅ Reusable code in `services/` (DRY principle)
|
|
76
|
-
- ✅ Easy to modify individual workflows without affecting others
|
|
77
|
-
|
|
78
|
-
---
|
|
79
|
-
|
|
80
|
-
## 💻 STEP 3: Implementation
|
|
81
|
-
|
|
82
|
-
- Source: GraphQL (`createClient().graphql` with auto-pagination)
|
|
83
|
-
- Transform: `UniversalMapper` (mapping per entity)
|
|
84
|
-
- Destination: S3 Parquet (`S3DataSource.writeParquetContent` + `uploadFile`)
|
|
85
|
-
- State: `VersoriKVAdapter` + `StateService` for incremental cursor
|
|
86
|
-
- Execution modes: scheduled workflow, ad hoc webhook
|
|
87
|
-
|
|
88
|
-
### 📦 SDK Imports (Verified - Versori Optimized)
|
|
89
|
-
|
|
90
|
-
```typescript
|
|
91
|
-
import {
|
|
92
|
-
createClient,
|
|
93
|
-
UniversalMapper,
|
|
94
|
-
S3DataSource,
|
|
95
|
-
VersoriKVAdapter,
|
|
96
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
97
|
-
|
|
98
|
-
// Versori platform
|
|
99
|
-
import { schedule, webhook, http, fn } from '@versori/run';
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
> Note: Use native Versori `log` from context; do not import `LoggingService` on Versori.
|
|
103
|
-
|
|
104
|
-
---
|
|
105
|
-
|
|
106
|
-
# Versori: GraphQL Extraction → Partitioned Parquet on S3
|
|
107
|
-
|
|
108
|
-
**FC Connect SDK Use Case Guide**
|
|
109
|
-
|
|
110
|
-
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
111
|
-
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
112
|
-
|
|
113
|
-
**Context**: Versori workflow that extracts records from Fluent via GraphQL with auto-pagination, transforms with UniversalMapper, and writes partitioned Parquet files to S3.
|
|
114
|
-
|
|
115
|
-
**Type**: Extraction
|
|
116
|
-
|
|
117
|
-
**Complexity**: Medium
|
|
118
|
-
|
|
119
|
-
**Runtime**: Versori Platform (Scheduled or Manual Webhook)
|
|
120
|
-
|
|
121
|
-
**Estimated Lines**: ~600
|
|
122
|
-
|
|
123
|
-
## What You'll Build
|
|
124
|
-
|
|
125
|
-
- Versori scheduled workflow (daily) and manual webhook trigger
|
|
126
|
-
- Fluent client via Versori connection
|
|
127
|
-
- GraphQL query with auto-pagination (progress callbacks)
|
|
128
|
-
- UniversalMapper transform for export schema
|
|
129
|
-
- Partitioning by date/entity/size with multiple Parquet files
|
|
130
|
-
- Upload to S3 target with metadata
|
|
131
|
-
- Incremental extraction using Versori KV state
|
|
132
|
-
|
|
133
|
-
## 📦 SDK Imports (Verified - Versori Optimized)
|
|
134
|
-
|
|
135
|
-
**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
|
|
136
|
-
|
|
137
|
-
**✅ VERSORI PLATFORM - Use Native Logs:**
|
|
138
|
-
|
|
139
|
-
- Use `log` from context: `const { log } = ctx;`
|
|
140
|
-
- Don't import or use LoggingService for Versori connectors
|
|
141
|
-
- Native Versori logs are simpler and automatically integrated with platform monitoring
|
|
142
|
-
|
|
143
|
-
```typescript
|
|
144
|
-
import {
|
|
145
|
-
createClient,
|
|
146
|
-
UniversalMapper,
|
|
147
|
-
S3DataSource,
|
|
148
|
-
VersoriKVAdapter,
|
|
149
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
150
|
-
|
|
151
|
-
await createClient(ctx); // Versori context-aware client
|
|
152
|
-
await client.graphql({ query, variables, pagination });
|
|
153
|
-
new UniversalMapper(mapping, { customResolvers });
|
|
154
|
-
await s3.writeParquetContent(data, { compression: 'SNAPPY' });
|
|
155
|
-
await s3.uploadFile(key, buffer, { contentType: 'application/octet-stream' });
|
|
156
|
-
const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
## Project Setup
|
|
160
|
-
|
|
161
|
-
```bash
|
|
162
|
-
# Create project
|
|
163
|
-
mkdir versori-graphql-parquet && cd $_
|
|
164
|
-
|
|
165
|
-
# Initialize
|
|
166
|
-
npm init -y
|
|
167
|
-
|
|
168
|
-
# Install dependencies
|
|
169
|
-
npm install @fluentcommerce/fc-connect-sdk @versori/run
|
|
170
|
-
|
|
171
|
-
# Structure
|
|
172
|
-
mkdir -p src
|
|
173
|
-
```
|
|
174
|
-
|
|
175
|
-
### Package Configuration (package.json)
|
|
176
|
-
|
|
177
|
-
```json
|
|
178
|
-
{
|
|
179
|
-
"name": "versori-graphql-parquet",
|
|
180
|
-
"version": "1.0.0",
|
|
181
|
-
"description": "Versori workflow: GraphQL extraction → partitioned Parquet on S3",
|
|
182
|
-
"versori": {
|
|
183
|
-
"workflows": "./src/index.ts"
|
|
184
|
-
},
|
|
185
|
-
"type": "module",
|
|
186
|
-
"scripts": {
|
|
187
|
-
"deploy": "versori deploy",
|
|
188
|
-
"logs": "versori logs"
|
|
189
|
-
},
|
|
190
|
-
"dependencies": {
|
|
191
|
-
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
192
|
-
"@versori/run": "latest"
|
|
193
|
-
},
|
|
194
|
-
"devDependencies": {
|
|
195
|
-
"typescript": "^5.0.0",
|
|
196
|
-
"@types/node": "^20.0.0"
|
|
197
|
-
}
|
|
198
|
-
}
|
|
199
|
-
```
|
|
200
|
-
|
|
201
|
-
### TypeScript Configuration (tsconfig.json)
|
|
202
|
-
|
|
203
|
-
```json
|
|
204
|
-
{
|
|
205
|
-
"compilerOptions": {
|
|
206
|
-
"module": "ES2022",
|
|
207
|
-
"target": "ES2024",
|
|
208
|
-
"moduleResolution": "node"
|
|
209
|
-
}
|
|
210
|
-
}
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
### Activation Variables (Versori)
|
|
214
|
-
|
|
215
|
-
```bash
|
|
216
|
-
# Fluent
|
|
217
|
-
retailerId=your-retailer-id
|
|
218
|
-
|
|
219
|
-
# Target S3
|
|
220
|
-
targetBucket=fluent-exports
|
|
221
|
-
targetRegion=us-east-1
|
|
222
|
-
targetAccessKeyId=AKIAXXXXXXXXXXXX
|
|
223
|
-
targetSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
224
|
-
targetPrefix=exports/
|
|
225
|
-
|
|
226
|
-
# Extraction controls
|
|
227
|
-
entityType=inventory # inventory | orders
|
|
228
|
-
pageSize=200
|
|
229
|
-
maxRecords=20000
|
|
230
|
-
incremental=true
|
|
231
|
-
partitionBy=date # date | entity | none
|
|
232
|
-
partitionSize=10000 # records per Parquet file (cap)
|
|
233
|
-
stateKey=export:last-run # KV key for last export timestamp
|
|
234
|
-
```
|
|
235
|
-
|
|
236
|
-
---
|
|
237
|
-
|
|
238
|
-
## Complete Workflow (src/index.ts)
|
|
239
|
-
|
|
240
|
-
```typescript
|
|
241
|
-
import { schedule, webhook, http, fn } from '@versori/run';
|
|
242
|
-
import {
|
|
243
|
-
createClient,
|
|
244
|
-
UniversalMapper,
|
|
245
|
-
S3DataSource,
|
|
246
|
-
VersoriKVAdapter,
|
|
247
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
248
|
-
|
|
249
|
-
// Inline query templates (vendor-agnostic)
|
|
250
|
-
const queries = {
|
|
251
|
-
inventory: {
|
|
252
|
-
connectionPath: 'inventoryPositions',
|
|
253
|
-
query: `query GetInventory($retailerId: ID!, $first: Int!, $after: String, $updatedFrom: DateTime) {
|
|
254
|
-
inventoryPositions(retailerId: $retailerId, first: $first, after: $after, updatedOn: { from: $updatedFrom }) {
|
|
255
|
-
edges { node { id ref productRef locationRef qty availableQty reservedQty status updatedOn } cursor }
|
|
256
|
-
pageInfo { hasNextPage }
|
|
257
|
-
}
|
|
258
|
-
}`,
|
|
259
|
-
},
|
|
260
|
-
orders: {
|
|
261
|
-
connectionPath: 'orders',
|
|
262
|
-
query: `query GetOrders($retailerId: ID!, $first: Int!, $after: String, $createdFrom: DateTime) {
|
|
263
|
-
orders(retailerId: $retailerId, first: $first, after: $after, createdOn: { from: $createdFrom }) {
|
|
264
|
-
edges { node { id ref status totalPrice totalTaxPrice createdOn customer { ref email firstName lastName } } cursor }
|
|
265
|
-
pageInfo { hasNextPage }
|
|
266
|
-
}
|
|
267
|
-
}`,
|
|
268
|
-
},
|
|
269
|
-
} as const;
|
|
270
|
-
|
|
271
|
-
// Inline mapping templates (minimal generic fields)
|
|
272
|
-
const mappings = {
|
|
273
|
-
inventory: {
|
|
274
|
-
fields: {
|
|
275
|
-
sku: { source: 'productRef', resolver: 'sdk.uppercase' },
|
|
276
|
-
location: { source: 'locationRef' },
|
|
277
|
-
quantity: { source: 'qty', resolver: 'sdk.parseInt' },
|
|
278
|
-
available: { source: 'availableQty', resolver: 'sdk.parseInt' },
|
|
279
|
-
status: { source: 'status', resolver: 'sdk.lowercase' },
|
|
280
|
-
updated_at: { source: 'updatedOn', resolver: 'sdk.formatDate' },
|
|
281
|
-
},
|
|
282
|
-
},
|
|
283
|
-
orders: {
|
|
284
|
-
fields: {
|
|
285
|
-
order_id: { source: 'ref' },
|
|
286
|
-
status: { source: 'status', resolver: 'sdk.lowercase' },
|
|
287
|
-
total: { source: 'totalPrice', resolver: 'sdk.parseFloat' },
|
|
288
|
-
created_at: { source: 'createdOn', resolver: 'sdk.formatDate' },
|
|
289
|
-
customer_email: { source: 'customer.email' },
|
|
290
|
-
customer_name: {
|
|
291
|
-
source: null,
|
|
292
|
-
resolver: 'custom.fullName', // Custom resolver defined below
|
|
293
|
-
},
|
|
294
|
-
},
|
|
295
|
-
},
|
|
296
|
-
} as const;
|
|
297
|
-
|
|
298
|
-
async function runExport(ctx: any) {
|
|
299
|
-
// ✅ VERSORI PLATFORM: Extract native log from context
|
|
300
|
-
const { log, openKv, activation } = ctx;
|
|
301
|
-
|
|
302
|
-
const entityType = (activation?.getVariable('entityType') || 'inventory').toLowerCase();
|
|
303
|
-
const retailerId = activation?.getVariable('retailerId');
|
|
304
|
-
const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
|
|
305
|
-
const maxRecords = parseInt(activation?.getVariable('maxRecords') || '20000', 10);
|
|
306
|
-
const incremental = (activation?.getVariable('incremental') || 'true') === 'true';
|
|
307
|
-
const partitionBy = (activation?.getVariable('partitionBy') || 'date').toLowerCase();
|
|
308
|
-
const partitionSize = parseInt(activation?.getVariable('partitionSize') || '10000', 10);
|
|
309
|
-
|
|
310
|
-
// Target S3
|
|
311
|
-
const s3 = new S3DataSource(
|
|
312
|
-
{
|
|
313
|
-
type: 'S3_CSV',
|
|
314
|
-
name: 'Target S3',
|
|
315
|
-
s3Config: {
|
|
316
|
-
bucket: activation?.getVariable('targetBucket'),
|
|
317
|
-
region: activation?.getVariable('targetRegion') || 'us-east-1',
|
|
318
|
-
accessKeyId: activation?.getVariable('targetAccessKeyId'),
|
|
319
|
-
secretAccessKey: activation?.getVariable('targetSecretAccessKey'),
|
|
320
|
-
},
|
|
321
|
-
},
|
|
322
|
-
log
|
|
323
|
-
);
|
|
324
|
-
|
|
325
|
-
// Fluent client from Versori connection
|
|
326
|
-
const client = await createClient(ctx);
|
|
327
|
-
const queryCfg = (queries as any)[entityType];
|
|
328
|
-
if (!queryCfg) {
|
|
329
|
-
log.error('[GraphQLParquetExtraction] Configuration error: Unknown entityType', {
|
|
330
|
-
entityType,
|
|
331
|
-
recommendation: `Set entityType to one of: ${Object.keys(queries).join(', ')}`
|
|
332
|
-
});
|
|
333
|
-
throw new Error(`Unknown entityType: ${entityType}`);
|
|
334
|
-
}
|
|
335
|
-
|
|
336
|
-
const variables: any = { retailerId, first: pageSize };
|
|
337
|
-
|
|
338
|
-
// Incremental state via Versori KV
|
|
339
|
-
const kv = new VersoriKVAdapter(openKv(':project:'));
|
|
340
|
-
const stateKey = activation?.getVariable('stateKey') || 'export:last-run';
|
|
341
|
-
|
|
342
|
-
if (incremental) {
|
|
343
|
-
const state = await kv.get([stateKey]);
|
|
344
|
-
const lastIso = (state?.value as any)?.lastExportIso;
|
|
345
|
-
if (lastIso) {
|
|
346
|
-
if (entityType === 'inventory') variables.updatedFrom = lastIso;
|
|
347
|
-
if (entityType === 'orders') variables.createdFrom = lastIso;
|
|
348
|
-
}
|
|
349
|
-
}
|
|
350
|
-
|
|
351
|
-
// Execute GraphQL with auto-pagination
|
|
352
|
-
const start = Date.now();
|
|
353
|
-
const result = await client.graphql({
|
|
354
|
-
query: queryCfg.query,
|
|
355
|
-
variables,
|
|
356
|
-
pagination: {
|
|
357
|
-
maxRecords,
|
|
358
|
-
onProgress: (page: number, count: number) => {
|
|
359
|
-
if (page % 10 === 0) log.info('Pagination progress', { page, count });
|
|
360
|
-
},
|
|
361
|
-
},
|
|
362
|
-
});
|
|
363
|
-
|
|
364
|
-
const connection = (result.data as any)[queryCfg.connectionPath];
|
|
365
|
-
const nodes = Array.isArray(connection?.edges) ? connection.edges.map((e: any) => e.node) : [];
|
|
366
|
-
|
|
367
|
-
log.info('Extracted records', { count: nodes.length, ms: Date.now() - start });
|
|
368
|
-
|
|
369
|
-
if (nodes.length === 0) return { uploaded: [], total: 0 };
|
|
370
|
-
|
|
371
|
-
// Transform
|
|
372
|
-
const mapper = new UniversalMapper((mappings as any)[entityType] || { fields: {} }, {
|
|
373
|
-
customResolvers: {
|
|
374
|
-
'custom.fullName': (v: any, s: any) =>
|
|
375
|
-
`${s?.customer?.firstName || ''} ${s?.customer?.lastName || ''}`.trim(),
|
|
376
|
-
},
|
|
377
|
-
});
|
|
378
|
-
|
|
379
|
-
const transformed: any[] = [];
|
|
380
|
-
for (const n of nodes) {
|
|
381
|
-
const res = await mapper.map(n);
|
|
382
|
-
if (res.success) transformed.push(res.data);
|
|
383
|
-
}
|
|
384
|
-
|
|
385
|
-
// Partitioning
|
|
386
|
-
const groups: any[][] = [];
|
|
387
|
-
if (partitionBy === 'none') {
|
|
388
|
-
for (let i = 0; i < transformed.length; i += partitionSize)
|
|
389
|
-
groups.push(transformed.slice(i, i + partitionSize));
|
|
390
|
-
} else if (partitionBy === 'entity') {
|
|
391
|
-
const byKey = new Map<string, any[]>();
|
|
392
|
-
for (const rec of transformed) {
|
|
393
|
-
const key = String(rec.status || 'default');
|
|
394
|
-
const arr = byKey.get(key) || [];
|
|
395
|
-
arr.push(rec);
|
|
396
|
-
byKey.set(key, arr);
|
|
397
|
-
}
|
|
398
|
-
for (const arr of byKey.values())
|
|
399
|
-
for (let i = 0; i < arr.length; i += partitionSize)
|
|
400
|
-
groups.push(arr.slice(i, i + partitionSize));
|
|
401
|
-
} else {
|
|
402
|
-
// date partition (YYYY/MM/DD)
|
|
403
|
-
const byDate = new Map<string, any[]>();
|
|
404
|
-
for (const rec of transformed) {
|
|
405
|
-
const iso = String(rec.updated_at || rec.created_at || new Date().toISOString());
|
|
406
|
-
const d = new Date(iso);
|
|
407
|
-
const key = `${d.getUTCFullYear()}/${String(d.getUTCMonth() + 1).padStart(2, '0')}/${String(d.getUTCDate()).padStart(2, '0')}`;
|
|
408
|
-
const arr = byDate.get(key) || [];
|
|
409
|
-
arr.push(rec);
|
|
410
|
-
byDate.set(key, arr);
|
|
411
|
-
}
|
|
412
|
-
for (const [dateKey, arr] of byDate.entries()) {
|
|
413
|
-
for (let i = 0; i < arr.length; i += partitionSize)
|
|
414
|
-
groups.push(Object.assign(arr.slice(i, i + partitionSize), { __dateKey: dateKey }));
|
|
415
|
-
}
|
|
416
|
-
}
|
|
417
|
-
|
|
418
|
-
// Upload Parquet parts
|
|
419
|
-
const uploaded: string[] = [];
|
|
420
|
-
const prefix = activation?.getVariable('targetPrefix') || 'exports/';
|
|
421
|
-
let idx = 0;
|
|
422
|
-
|
|
423
|
-
for (const group of groups) {
|
|
424
|
-
const dateKey = (group as any).__dateKey as string | undefined;
|
|
425
|
-
const ts = new Date().toISOString().replace(/[:.]/g, '-');
|
|
426
|
-
const folder =
|
|
427
|
-
partitionBy === 'date' && dateKey
|
|
428
|
-
? `${prefix}${entityType}/${dateKey}/`
|
|
429
|
-
: `${prefix}${entityType}/`;
|
|
430
|
-
const key = `${folder}${entityType}_${ts}_part-${idx++}.parquet`;
|
|
431
|
-
|
|
432
|
-
const buffer = await (s3 as any).writeParquetContent(group as any[], {
|
|
433
|
-
compression: 'SNAPPY',
|
|
434
|
-
rowGroupSize: 10000,
|
|
435
|
-
});
|
|
436
|
-
|
|
437
|
-
await s3.uploadFile(key, buffer, {
|
|
438
|
-
contentType: 'application/octet-stream',
|
|
439
|
-
metadata: {
|
|
440
|
-
'entity-type': entityType,
|
|
441
|
-
'record-count': String(group.length),
|
|
442
|
-
'exported-at': new Date().toISOString(),
|
|
443
|
-
},
|
|
444
|
-
});
|
|
445
|
-
|
|
446
|
-
uploaded.push(key);
|
|
447
|
-
}
|
|
448
|
-
|
|
449
|
-
// Save incremental state
|
|
450
|
-
if (incremental) {
|
|
451
|
-
await kv.set([stateKey], { lastExportIso: new Date().toISOString() });
|
|
452
|
-
}
|
|
453
|
-
|
|
454
|
-
return { uploaded, total: transformed.length };
|
|
455
|
-
}
|
|
456
|
-
|
|
457
|
-
// Scheduled export (e.g., daily 2 AM UTC)
|
|
458
|
-
export const scheduledGraphqlExport = schedule('graphql-export-daily', '0 2 * * *').then(
|
|
459
|
-
http('export-to-parquet', { connection: 'fluent_commerce' }, async ctx => {
|
|
460
|
-
const result = await runExport(ctx);
|
|
461
|
-
return { success: true, ...result };
|
|
462
|
-
})
|
|
463
|
-
);
|
|
464
|
-
|
|
465
|
-
// Manual trigger for testing
|
|
466
|
-
export const exportNow = webhook('export-now').then(
|
|
467
|
-
http('export-now-run', { connection: 'fluent_commerce' }, async ctx => {
|
|
468
|
-
const result = await runExport(ctx);
|
|
469
|
-
return { success: true, ...result };
|
|
470
|
-
})
|
|
471
|
-
);
|
|
472
|
-
```
|
|
473
|
-
|
|
474
|
-
---
|
|
475
|
-
|
|
476
|
-
## Deployment (Versori)
|
|
477
|
-
|
|
478
|
-
```bash
|
|
479
|
-
npm run deploy
|
|
480
|
-
npm run logs
|
|
481
|
-
|
|
482
|
-
# Test manual endpoint:
|
|
483
|
-
# curl -X POST https://{workspace}.versori.run/export-now -H "Content-Type: application/json" -d '{}'
|
|
484
|
-
```
|
|
485
|
-
|
|
486
|
-
## Testing
|
|
487
|
-
|
|
488
|
-
- Small run: set `pageSize=50`, `maxRecords=200`, `incremental=false` and trigger `export-now`.
|
|
489
|
-
- Verify S3 keys under `targetPrefix/entityType/YYYY/MM/DD/`.
|
|
490
|
-
- Check metadata on uploaded objects for `record-count` and `exported-at`.
|
|
491
|
-
|
|
492
|
-
## Common Issues & Solutions
|
|
493
|
-
|
|
494
|
-
- **Auto-pagination not running**
|
|
495
|
-
- Ensure GraphQL query uses `$first` and `$after` variables and you pass `first` in variables.
|
|
496
|
-
- **KV state not updating**
|
|
497
|
-
- Verify `stateKey` and that `:project:` scope is used with `openKv(':project:')`.
|
|
498
|
-
- **Large uploads slow**
|
|
499
|
-
- Use SNAPPY compression (default shown), reduce partition size, and consider parallel uploads.
|
|
500
|
-
- **Time gaps in incremental**
|
|
501
|
-
- Add an overlap (export last 5-10 minutes again) and dedupe in downstream consumers.
|
|
502
|
-
|
|
503
|
-
## Production Checklist
|
|
504
|
-
|
|
505
|
-
- Activation variables set and secured (no secrets in code)
|
|
506
|
-
- Connection configured for Fluent Commerce
|
|
507
|
-
- S3 permissions validated (PutObject, ListBucket)
|
|
508
|
-
- Partition strategy agreed with consumers (date/entity/size)
|
|
509
|
-
- Incremental strategy defined with overlap
|
|
510
|
-
- Observability (logs/metrics) and alerting on failures
|
|
511
|
-
- Cost/performance tuned (page size, row groups, partition size)
|
|
512
|
-
|
|
513
|
-
## Related Guides
|
|
514
|
-
|
|
515
|
-
- Auto-pagination: `fc-connect-sdk/docs/guides/pagination.md`
|
|
516
|
-
- Universal Mapping: `fc-connect-sdk/docs/guides/mapping/readme.md`
|
|
517
|
-
- Versori KV patterns: `docs/use-cases/versori/03-kv-state-management.md`
|
|
518
|
-
- Error handling: `docs/use-cases/patterns/error-handling-retry.md`
|
|
1
|
+
---
|
|
2
|
+
template_id: tpl-extract-graphql-to-s3-parquet
|
|
3
|
+
canonical_filename: graphql-parquet-extraction.md
|
|
4
|
+
sdk_version: latest
|
|
5
|
+
runtime: versori
|
|
6
|
+
direction: extraction
|
|
7
|
+
source: fluent-graphql
|
|
8
|
+
destination: s3-parquet
|
|
9
|
+
entity: inventory|orders
|
|
10
|
+
format: parquet
|
|
11
|
+
logging: versori
|
|
12
|
+
status: stable
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Template: Extraction - GraphQL to S3 Parquet
|
|
16
|
+
|
|
17
|
+
**SDK**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
18
|
+
**Deployment Target:** Versori Platform
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## 📋 Implementation Prompt
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
I need a Versori extractor that:
|
|
28
|
+
|
|
29
|
+
1) Queries Fluent Commerce GraphQL with auto-pagination (Relay cursors)
|
|
30
|
+
2) Supports incremental runs via KV state (date overlap buffer optional)
|
|
31
|
+
3) Transforms results using UniversalMapper per mapping config
|
|
32
|
+
4) Writes partitioned Parquet files to S3 with metadata
|
|
33
|
+
5) Uses native Versori log (LoggingService removed - use native log)
|
|
34
|
+
|
|
35
|
+
Use the loaded docs to fill in SDK specifics and best practices.
|
|
36
|
+
Keep the structure identical to the template; only adapt where needed.
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Versori Workflows Structure
|
|
42
|
+
|
|
43
|
+
**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
|
|
44
|
+
|
|
45
|
+
**Trigger Types:**
|
|
46
|
+
- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
|
|
47
|
+
- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
|
|
48
|
+
- **`http()`** → External API calls (chained from webhook/schedule)
|
|
49
|
+
- **`fn()`** → Internal processing (chained from webhook/schedule)
|
|
50
|
+
|
|
51
|
+
### Recommended Project Structure
|
|
52
|
+
|
|
53
|
+
```
|
|
54
|
+
graphql-parquet-extraction/
|
|
55
|
+
├── index.ts # Entry point - exports all workflows
|
|
56
|
+
└── src/
|
|
57
|
+
├── workflows/
|
|
58
|
+
│ ├── webhook/
|
|
59
|
+
│ │ └── adhoc-extraction.ts # Webhook: Manual extraction
|
|
60
|
+
│ │
|
|
61
|
+
│ └── scheduled/
|
|
62
|
+
│ └── daily-extraction.ts # Scheduled: Daily extraction
|
|
63
|
+
│
|
|
64
|
+
├── services/
|
|
65
|
+
│ └── extraction.service.ts # Shared orchestration logic (reusable)
|
|
66
|
+
│
|
|
67
|
+
└── config/
|
|
68
|
+
└── extraction-config.json # Configuration
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
**Benefits:**
|
|
72
|
+
- ✅ Clear trigger separation (`webhook/` vs `scheduled/`)
|
|
73
|
+
- ✅ Descriptive file names (easy to browse and understand)
|
|
74
|
+
- ✅ Scalable (add new workflows without cluttering)
|
|
75
|
+
- ✅ Reusable code in `services/` (DRY principle)
|
|
76
|
+
- ✅ Easy to modify individual workflows without affecting others
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## 💻 STEP 3: Implementation
|
|
81
|
+
|
|
82
|
+
- Source: GraphQL (`createClient().graphql` with auto-pagination)
|
|
83
|
+
- Transform: `UniversalMapper` (mapping per entity)
|
|
84
|
+
- Destination: S3 Parquet (`S3DataSource.writeParquetContent` + `uploadFile`)
|
|
85
|
+
- State: `VersoriKVAdapter` + `StateService` for incremental cursor
|
|
86
|
+
- Execution modes: scheduled workflow, ad hoc webhook
|
|
87
|
+
|
|
88
|
+
### 📦 SDK Imports (Verified - Versori Optimized)
|
|
89
|
+
|
|
90
|
+
```typescript
|
|
91
|
+
import {
|
|
92
|
+
createClient,
|
|
93
|
+
UniversalMapper,
|
|
94
|
+
S3DataSource,
|
|
95
|
+
VersoriKVAdapter,
|
|
96
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
97
|
+
|
|
98
|
+
// Versori platform
|
|
99
|
+
import { schedule, webhook, http, fn } from '@versori/run';
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
> Note: Use native Versori `log` from context; do not import `LoggingService` on Versori.
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
# Versori: GraphQL Extraction → Partitioned Parquet on S3
|
|
107
|
+
|
|
108
|
+
**FC Connect SDK Use Case Guide**
|
|
109
|
+
|
|
110
|
+
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
111
|
+
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
112
|
+
|
|
113
|
+
**Context**: Versori workflow that extracts records from Fluent via GraphQL with auto-pagination, transforms with UniversalMapper, and writes partitioned Parquet files to S3.
|
|
114
|
+
|
|
115
|
+
**Type**: Extraction
|
|
116
|
+
|
|
117
|
+
**Complexity**: Medium
|
|
118
|
+
|
|
119
|
+
**Runtime**: Versori Platform (Scheduled or Manual Webhook)
|
|
120
|
+
|
|
121
|
+
**Estimated Lines**: ~600
|
|
122
|
+
|
|
123
|
+
## What You'll Build
|
|
124
|
+
|
|
125
|
+
- Versori scheduled workflow (daily) and manual webhook trigger
|
|
126
|
+
- Fluent client via Versori connection
|
|
127
|
+
- GraphQL query with auto-pagination (progress callbacks)
|
|
128
|
+
- UniversalMapper transform for export schema
|
|
129
|
+
- Partitioning by date/entity/size with multiple Parquet files
|
|
130
|
+
- Upload to S3 target with metadata
|
|
131
|
+
- Incremental extraction using Versori KV state
|
|
132
|
+
|
|
133
|
+
## 📦 SDK Imports (Verified - Versori Optimized)
|
|
134
|
+
|
|
135
|
+
**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
|
|
136
|
+
|
|
137
|
+
**✅ VERSORI PLATFORM - Use Native Logs:**
|
|
138
|
+
|
|
139
|
+
- Use `log` from context: `const { log } = ctx;`
|
|
140
|
+
- Don't import or use LoggingService for Versori connectors
|
|
141
|
+
- Native Versori logs are simpler and automatically integrated with platform monitoring
|
|
142
|
+
|
|
143
|
+
```typescript
|
|
144
|
+
import {
|
|
145
|
+
createClient,
|
|
146
|
+
UniversalMapper,
|
|
147
|
+
S3DataSource,
|
|
148
|
+
VersoriKVAdapter,
|
|
149
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
150
|
+
|
|
151
|
+
await createClient(ctx); // Versori context-aware client
|
|
152
|
+
await client.graphql({ query, variables, pagination });
|
|
153
|
+
new UniversalMapper(mapping, { customResolvers });
|
|
154
|
+
await s3.writeParquetContent(data, { compression: 'SNAPPY' });
|
|
155
|
+
await s3.uploadFile(key, buffer, { contentType: 'application/octet-stream' });
|
|
156
|
+
const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
## Project Setup
|
|
160
|
+
|
|
161
|
+
```bash
|
|
162
|
+
# Create project
|
|
163
|
+
mkdir versori-graphql-parquet && cd $_
|
|
164
|
+
|
|
165
|
+
# Initialize
|
|
166
|
+
npm init -y
|
|
167
|
+
|
|
168
|
+
# Install dependencies
|
|
169
|
+
npm install @fluentcommerce/fc-connect-sdk @versori/run
|
|
170
|
+
|
|
171
|
+
# Structure
|
|
172
|
+
mkdir -p src
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
### Package Configuration (package.json)
|
|
176
|
+
|
|
177
|
+
```json
|
|
178
|
+
{
|
|
179
|
+
"name": "versori-graphql-parquet",
|
|
180
|
+
"version": "1.0.0",
|
|
181
|
+
"description": "Versori workflow: GraphQL extraction → partitioned Parquet on S3",
|
|
182
|
+
"versori": {
|
|
183
|
+
"workflows": "./src/index.ts"
|
|
184
|
+
},
|
|
185
|
+
"type": "module",
|
|
186
|
+
"scripts": {
|
|
187
|
+
"deploy": "versori deploy",
|
|
188
|
+
"logs": "versori logs"
|
|
189
|
+
},
|
|
190
|
+
"dependencies": {
|
|
191
|
+
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
192
|
+
"@versori/run": "latest"
|
|
193
|
+
},
|
|
194
|
+
"devDependencies": {
|
|
195
|
+
"typescript": "^5.0.0",
|
|
196
|
+
"@types/node": "^20.0.0"
|
|
197
|
+
}
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
### TypeScript Configuration (tsconfig.json)
|
|
202
|
+
|
|
203
|
+
```json
|
|
204
|
+
{
|
|
205
|
+
"compilerOptions": {
|
|
206
|
+
"module": "ES2022",
|
|
207
|
+
"target": "ES2024",
|
|
208
|
+
"moduleResolution": "node"
|
|
209
|
+
}
|
|
210
|
+
}
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
### Activation Variables (Versori)
|
|
214
|
+
|
|
215
|
+
```bash
|
|
216
|
+
# Fluent
|
|
217
|
+
retailerId=your-retailer-id
|
|
218
|
+
|
|
219
|
+
# Target S3
|
|
220
|
+
targetBucket=fluent-exports
|
|
221
|
+
targetRegion=us-east-1
|
|
222
|
+
targetAccessKeyId=AKIAXXXXXXXXXXXX
|
|
223
|
+
targetSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
224
|
+
targetPrefix=exports/
|
|
225
|
+
|
|
226
|
+
# Extraction controls
|
|
227
|
+
entityType=inventory # inventory | orders
|
|
228
|
+
pageSize=200
|
|
229
|
+
maxRecords=20000
|
|
230
|
+
incremental=true
|
|
231
|
+
partitionBy=date # date | entity | none
|
|
232
|
+
partitionSize=10000 # records per Parquet file (cap)
|
|
233
|
+
stateKey=export:last-run # KV key for last export timestamp
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
---
|
|
237
|
+
|
|
238
|
+
## Complete Workflow (src/index.ts)
|
|
239
|
+
|
|
240
|
+
```typescript
|
|
241
|
+
import { schedule, webhook, http, fn } from '@versori/run';
|
|
242
|
+
import {
|
|
243
|
+
createClient,
|
|
244
|
+
UniversalMapper,
|
|
245
|
+
S3DataSource,
|
|
246
|
+
VersoriKVAdapter,
|
|
247
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
248
|
+
|
|
249
|
+
// Inline query templates (vendor-agnostic)
|
|
250
|
+
const queries = {
|
|
251
|
+
inventory: {
|
|
252
|
+
connectionPath: 'inventoryPositions',
|
|
253
|
+
query: `query GetInventory($retailerId: ID!, $first: Int!, $after: String, $updatedFrom: DateTime) {
|
|
254
|
+
inventoryPositions(retailerId: $retailerId, first: $first, after: $after, updatedOn: { from: $updatedFrom }) {
|
|
255
|
+
edges { node { id ref productRef locationRef qty availableQty reservedQty status updatedOn } cursor }
|
|
256
|
+
pageInfo { hasNextPage }
|
|
257
|
+
}
|
|
258
|
+
}`,
|
|
259
|
+
},
|
|
260
|
+
orders: {
|
|
261
|
+
connectionPath: 'orders',
|
|
262
|
+
query: `query GetOrders($retailerId: ID!, $first: Int!, $after: String, $createdFrom: DateTime) {
|
|
263
|
+
orders(retailerId: $retailerId, first: $first, after: $after, createdOn: { from: $createdFrom }) {
|
|
264
|
+
edges { node { id ref status totalPrice totalTaxPrice createdOn customer { ref email firstName lastName } } cursor }
|
|
265
|
+
pageInfo { hasNextPage }
|
|
266
|
+
}
|
|
267
|
+
}`,
|
|
268
|
+
},
|
|
269
|
+
} as const;
|
|
270
|
+
|
|
271
|
+
// Inline mapping templates (minimal generic fields)
|
|
272
|
+
const mappings = {
|
|
273
|
+
inventory: {
|
|
274
|
+
fields: {
|
|
275
|
+
sku: { source: 'productRef', resolver: 'sdk.uppercase' },
|
|
276
|
+
location: { source: 'locationRef' },
|
|
277
|
+
quantity: { source: 'qty', resolver: 'sdk.parseInt' },
|
|
278
|
+
available: { source: 'availableQty', resolver: 'sdk.parseInt' },
|
|
279
|
+
status: { source: 'status', resolver: 'sdk.lowercase' },
|
|
280
|
+
updated_at: { source: 'updatedOn', resolver: 'sdk.formatDate' },
|
|
281
|
+
},
|
|
282
|
+
},
|
|
283
|
+
orders: {
|
|
284
|
+
fields: {
|
|
285
|
+
order_id: { source: 'ref' },
|
|
286
|
+
status: { source: 'status', resolver: 'sdk.lowercase' },
|
|
287
|
+
total: { source: 'totalPrice', resolver: 'sdk.parseFloat' },
|
|
288
|
+
created_at: { source: 'createdOn', resolver: 'sdk.formatDate' },
|
|
289
|
+
customer_email: { source: 'customer.email' },
|
|
290
|
+
customer_name: {
|
|
291
|
+
source: null,
|
|
292
|
+
resolver: 'custom.fullName', // Custom resolver defined below
|
|
293
|
+
},
|
|
294
|
+
},
|
|
295
|
+
},
|
|
296
|
+
} as const;
|
|
297
|
+
|
|
298
|
+
async function runExport(ctx: any) {
|
|
299
|
+
// ✅ VERSORI PLATFORM: Extract native log from context
|
|
300
|
+
const { log, openKv, activation } = ctx;
|
|
301
|
+
|
|
302
|
+
const entityType = (activation?.getVariable('entityType') || 'inventory').toLowerCase();
|
|
303
|
+
const retailerId = activation?.getVariable('retailerId');
|
|
304
|
+
const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
|
|
305
|
+
const maxRecords = parseInt(activation?.getVariable('maxRecords') || '20000', 10);
|
|
306
|
+
const incremental = (activation?.getVariable('incremental') || 'true') === 'true';
|
|
307
|
+
const partitionBy = (activation?.getVariable('partitionBy') || 'date').toLowerCase();
|
|
308
|
+
const partitionSize = parseInt(activation?.getVariable('partitionSize') || '10000', 10);
|
|
309
|
+
|
|
310
|
+
// Target S3
|
|
311
|
+
const s3 = new S3DataSource(
|
|
312
|
+
{
|
|
313
|
+
type: 'S3_CSV',
|
|
314
|
+
name: 'Target S3',
|
|
315
|
+
s3Config: {
|
|
316
|
+
bucket: activation?.getVariable('targetBucket'),
|
|
317
|
+
region: activation?.getVariable('targetRegion') || 'us-east-1',
|
|
318
|
+
accessKeyId: activation?.getVariable('targetAccessKeyId'),
|
|
319
|
+
secretAccessKey: activation?.getVariable('targetSecretAccessKey'),
|
|
320
|
+
},
|
|
321
|
+
},
|
|
322
|
+
log
|
|
323
|
+
);
|
|
324
|
+
|
|
325
|
+
// Fluent client from Versori connection
|
|
326
|
+
const client = await createClient(ctx);
|
|
327
|
+
const queryCfg = (queries as any)[entityType];
|
|
328
|
+
if (!queryCfg) {
|
|
329
|
+
log.error('[GraphQLParquetExtraction] Configuration error: Unknown entityType', {
|
|
330
|
+
entityType,
|
|
331
|
+
recommendation: `Set entityType to one of: ${Object.keys(queries).join(', ')}`
|
|
332
|
+
});
|
|
333
|
+
throw new Error(`Unknown entityType: ${entityType}`);
|
|
334
|
+
}
|
|
335
|
+
|
|
336
|
+
const variables: any = { retailerId, first: pageSize };
|
|
337
|
+
|
|
338
|
+
// Incremental state via Versori KV
|
|
339
|
+
const kv = new VersoriKVAdapter(openKv(':project:'));
|
|
340
|
+
const stateKey = activation?.getVariable('stateKey') || 'export:last-run';
|
|
341
|
+
|
|
342
|
+
if (incremental) {
|
|
343
|
+
const state = await kv.get([stateKey]);
|
|
344
|
+
const lastIso = (state?.value as any)?.lastExportIso;
|
|
345
|
+
if (lastIso) {
|
|
346
|
+
if (entityType === 'inventory') variables.updatedFrom = lastIso;
|
|
347
|
+
if (entityType === 'orders') variables.createdFrom = lastIso;
|
|
348
|
+
}
|
|
349
|
+
}
|
|
350
|
+
|
|
351
|
+
// Execute GraphQL with auto-pagination
|
|
352
|
+
const start = Date.now();
|
|
353
|
+
const result = await client.graphql({
|
|
354
|
+
query: queryCfg.query,
|
|
355
|
+
variables,
|
|
356
|
+
pagination: {
|
|
357
|
+
maxRecords,
|
|
358
|
+
onProgress: (page: number, count: number) => {
|
|
359
|
+
if (page % 10 === 0) log.info('Pagination progress', { page, count });
|
|
360
|
+
},
|
|
361
|
+
},
|
|
362
|
+
});
|
|
363
|
+
|
|
364
|
+
const connection = (result.data as any)[queryCfg.connectionPath];
|
|
365
|
+
const nodes = Array.isArray(connection?.edges) ? connection.edges.map((e: any) => e.node) : [];
|
|
366
|
+
|
|
367
|
+
log.info('Extracted records', { count: nodes.length, ms: Date.now() - start });
|
|
368
|
+
|
|
369
|
+
if (nodes.length === 0) return { uploaded: [], total: 0 };
|
|
370
|
+
|
|
371
|
+
// Transform
|
|
372
|
+
const mapper = new UniversalMapper((mappings as any)[entityType] || { fields: {} }, {
|
|
373
|
+
customResolvers: {
|
|
374
|
+
'custom.fullName': (v: any, s: any) =>
|
|
375
|
+
`${s?.customer?.firstName || ''} ${s?.customer?.lastName || ''}`.trim(),
|
|
376
|
+
},
|
|
377
|
+
});
|
|
378
|
+
|
|
379
|
+
const transformed: any[] = [];
|
|
380
|
+
for (const n of nodes) {
|
|
381
|
+
const res = await mapper.map(n);
|
|
382
|
+
if (res.success) transformed.push(res.data);
|
|
383
|
+
}
|
|
384
|
+
|
|
385
|
+
// Partitioning
|
|
386
|
+
const groups: any[][] = [];
|
|
387
|
+
if (partitionBy === 'none') {
|
|
388
|
+
for (let i = 0; i < transformed.length; i += partitionSize)
|
|
389
|
+
groups.push(transformed.slice(i, i + partitionSize));
|
|
390
|
+
} else if (partitionBy === 'entity') {
|
|
391
|
+
const byKey = new Map<string, any[]>();
|
|
392
|
+
for (const rec of transformed) {
|
|
393
|
+
const key = String(rec.status || 'default');
|
|
394
|
+
const arr = byKey.get(key) || [];
|
|
395
|
+
arr.push(rec);
|
|
396
|
+
byKey.set(key, arr);
|
|
397
|
+
}
|
|
398
|
+
for (const arr of byKey.values())
|
|
399
|
+
for (let i = 0; i < arr.length; i += partitionSize)
|
|
400
|
+
groups.push(arr.slice(i, i + partitionSize));
|
|
401
|
+
} else {
|
|
402
|
+
// date partition (YYYY/MM/DD)
|
|
403
|
+
const byDate = new Map<string, any[]>();
|
|
404
|
+
for (const rec of transformed) {
|
|
405
|
+
const iso = String(rec.updated_at || rec.created_at || new Date().toISOString());
|
|
406
|
+
const d = new Date(iso);
|
|
407
|
+
const key = `${d.getUTCFullYear()}/${String(d.getUTCMonth() + 1).padStart(2, '0')}/${String(d.getUTCDate()).padStart(2, '0')}`;
|
|
408
|
+
const arr = byDate.get(key) || [];
|
|
409
|
+
arr.push(rec);
|
|
410
|
+
byDate.set(key, arr);
|
|
411
|
+
}
|
|
412
|
+
for (const [dateKey, arr] of byDate.entries()) {
|
|
413
|
+
for (let i = 0; i < arr.length; i += partitionSize)
|
|
414
|
+
groups.push(Object.assign(arr.slice(i, i + partitionSize), { __dateKey: dateKey }));
|
|
415
|
+
}
|
|
416
|
+
}
|
|
417
|
+
|
|
418
|
+
// Upload Parquet parts
|
|
419
|
+
const uploaded: string[] = [];
|
|
420
|
+
const prefix = activation?.getVariable('targetPrefix') || 'exports/';
|
|
421
|
+
let idx = 0;
|
|
422
|
+
|
|
423
|
+
for (const group of groups) {
|
|
424
|
+
const dateKey = (group as any).__dateKey as string | undefined;
|
|
425
|
+
const ts = new Date().toISOString().replace(/[:.]/g, '-');
|
|
426
|
+
const folder =
|
|
427
|
+
partitionBy === 'date' && dateKey
|
|
428
|
+
? `${prefix}${entityType}/${dateKey}/`
|
|
429
|
+
: `${prefix}${entityType}/`;
|
|
430
|
+
const key = `${folder}${entityType}_${ts}_part-${idx++}.parquet`;
|
|
431
|
+
|
|
432
|
+
const buffer = await (s3 as any).writeParquetContent(group as any[], {
|
|
433
|
+
compression: 'SNAPPY',
|
|
434
|
+
rowGroupSize: 10000,
|
|
435
|
+
});
|
|
436
|
+
|
|
437
|
+
await s3.uploadFile(key, buffer, {
|
|
438
|
+
contentType: 'application/octet-stream',
|
|
439
|
+
metadata: {
|
|
440
|
+
'entity-type': entityType,
|
|
441
|
+
'record-count': String(group.length),
|
|
442
|
+
'exported-at': new Date().toISOString(),
|
|
443
|
+
},
|
|
444
|
+
});
|
|
445
|
+
|
|
446
|
+
uploaded.push(key);
|
|
447
|
+
}
|
|
448
|
+
|
|
449
|
+
// Save incremental state
|
|
450
|
+
if (incremental) {
|
|
451
|
+
await kv.set([stateKey], { lastExportIso: new Date().toISOString() });
|
|
452
|
+
}
|
|
453
|
+
|
|
454
|
+
return { uploaded, total: transformed.length };
|
|
455
|
+
}
|
|
456
|
+
|
|
457
|
+
// Scheduled export (e.g., daily 2 AM UTC)
|
|
458
|
+
export const scheduledGraphqlExport = schedule('graphql-export-daily', '0 2 * * *').then(
|
|
459
|
+
http('export-to-parquet', { connection: 'fluent_commerce' }, async ctx => {
|
|
460
|
+
const result = await runExport(ctx);
|
|
461
|
+
return { success: true, ...result };
|
|
462
|
+
})
|
|
463
|
+
);
|
|
464
|
+
|
|
465
|
+
// Manual trigger for testing
|
|
466
|
+
export const exportNow = webhook('export-now').then(
|
|
467
|
+
http('export-now-run', { connection: 'fluent_commerce' }, async ctx => {
|
|
468
|
+
const result = await runExport(ctx);
|
|
469
|
+
return { success: true, ...result };
|
|
470
|
+
})
|
|
471
|
+
);
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
---
|
|
475
|
+
|
|
476
|
+
## Deployment (Versori)
|
|
477
|
+
|
|
478
|
+
```bash
|
|
479
|
+
npm run deploy
|
|
480
|
+
npm run logs
|
|
481
|
+
|
|
482
|
+
# Test manual endpoint:
|
|
483
|
+
# curl -X POST https://{workspace}.versori.run/export-now -H "Content-Type: application/json" -d '{}'
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
## Testing
|
|
487
|
+
|
|
488
|
+
- Small run: set `pageSize=50`, `maxRecords=200`, `incremental=false` and trigger `export-now`.
|
|
489
|
+
- Verify S3 keys under `targetPrefix/entityType/YYYY/MM/DD/`.
|
|
490
|
+
- Check metadata on uploaded objects for `record-count` and `exported-at`.
|
|
491
|
+
|
|
492
|
+
## Common Issues & Solutions
|
|
493
|
+
|
|
494
|
+
- **Auto-pagination not running**
|
|
495
|
+
- Ensure GraphQL query uses `$first` and `$after` variables and you pass `first` in variables.
|
|
496
|
+
- **KV state not updating**
|
|
497
|
+
- Verify `stateKey` and that `:project:` scope is used with `openKv(':project:')`.
|
|
498
|
+
- **Large uploads slow**
|
|
499
|
+
- Use SNAPPY compression (default shown), reduce partition size, and consider parallel uploads.
|
|
500
|
+
- **Time gaps in incremental**
|
|
501
|
+
- Add an overlap (export last 5-10 minutes again) and dedupe in downstream consumers.
|
|
502
|
+
|
|
503
|
+
## Production Checklist
|
|
504
|
+
|
|
505
|
+
- Activation variables set and secured (no secrets in code)
|
|
506
|
+
- Connection configured for Fluent Commerce
|
|
507
|
+
- S3 permissions validated (PutObject, ListBucket)
|
|
508
|
+
- Partition strategy agreed with consumers (date/entity/size)
|
|
509
|
+
- Incremental strategy defined with overlap
|
|
510
|
+
- Observability (logs/metrics) and alerting on failures
|
|
511
|
+
- Cost/performance tuned (page size, row groups, partition size)
|
|
512
|
+
|
|
513
|
+
## Related Guides
|
|
514
|
+
|
|
515
|
+
- Auto-pagination: `fc-connect-sdk/docs/guides/pagination.md`
|
|
516
|
+
- Universal Mapping: `fc-connect-sdk/docs/guides/mapping/readme.md`
|
|
517
|
+
- Versori KV patterns: `docs/use-cases/versori/03-kv-state-management.md`
|
|
518
|
+
- Error handling: `docs/use-cases/patterns/error-handling-retry.md`
|