@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/README.md +11 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
- package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
|
@@ -1,990 +1,990 @@
|
|
|
1
|
-
# Module 6: KV Storage - State Management & File Tracking
|
|
2
|
-
|
|
3
|
-
[← Back to Versori Platform Guide](../platforms-versori-readme.md)
|
|
4
|
-
|
|
5
|
-
**Module 6 of 8** | **Level**: Intermediate | **Time**: 25 minutes
|
|
6
|
-
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
## Learning Objectives
|
|
10
|
-
|
|
11
|
-
By the end of this module, you will:
|
|
12
|
-
|
|
13
|
-
- ✅ Understand Versori KV storage architecture and capabilities
|
|
14
|
-
- ✅ Use VersoriKVAdapter for low-level KV operations
|
|
15
|
-
- ✅ Implement VersoriFileTracker for duplicate prevention
|
|
16
|
-
- ✅ Master KV scoping and namespace patterns
|
|
17
|
-
- ✅ Implement distributed locking for concurrent workflows
|
|
18
|
-
- ✅ Optimize KV storage for performance and cost
|
|
19
|
-
- ✅ Debug and troubleshoot KV storage issues
|
|
20
|
-
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
## KV Storage Overview
|
|
24
|
-
|
|
25
|
-
Versori provides a distributed key-value store accessible from all workflow types. It's designed for:
|
|
26
|
-
|
|
27
|
-
| Use Case | Pattern | SDK Component |
|
|
28
|
-
| ----------------------- | ---------------------------- | ------------------------------------------- |
|
|
29
|
-
| **File tracking** | Prevent duplicate processing | `VersoriFileTracker` |
|
|
30
|
-
| **State management** | Workflow state persistence | `StateService` + `VersoriKVAdapter` |
|
|
31
|
-
| **Distributed locking** | Prevent concurrent execution | `StateService.acquireLock()` |
|
|
32
|
-
| **Caching** | Cache expensive API calls | Native Versori KV with TTL |
|
|
33
|
-
| **Rate limiting** | Track request counts | Native Versori KV (custom implementation) |
|
|
34
|
-
| **Incremental sync** | Track last processed cursor | `VersoriFileTracker.setLastProcessedFile()` |
|
|
35
|
-
|
|
36
|
-
---
|
|
37
|
-
|
|
38
|
-
## KV Storage Basics
|
|
39
|
-
|
|
40
|
-
### Opening KV Store
|
|
41
|
-
|
|
42
|
-
```typescript
|
|
43
|
-
import { fn } from '@versori/run';
|
|
44
|
-
|
|
45
|
-
export const kvBasics = fn('kv-basics', async ctx => {
|
|
46
|
-
// Three scoping options:
|
|
47
|
-
|
|
48
|
-
// 1. Project scope - shared across ALL workflows
|
|
49
|
-
const projectKV = ctx.openKv(':project:');
|
|
50
|
-
|
|
51
|
-
// 2. Workflow scope - isolated to THIS workflow
|
|
52
|
-
const workflowKV = ctx.openKv(':workflow:my-workflow');
|
|
53
|
-
|
|
54
|
-
// 3. Custom scope - custom namespace
|
|
55
|
-
const customKV = ctx.openKv('custom:namespace:name');
|
|
56
|
-
|
|
57
|
-
return { message: 'KV stores opened' };
|
|
58
|
-
});
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
### Basic Operations
|
|
62
|
-
|
|
63
|
-
```typescript
|
|
64
|
-
export const kvOperations = fn('kv-operations', async ctx => {
|
|
65
|
-
const kv = ctx.openKv(':project:');
|
|
66
|
-
|
|
67
|
-
// Set value
|
|
68
|
-
await kv.set('key', { foo: 'bar' });
|
|
69
|
-
|
|
70
|
-
// Set with TTL (30 minutes = 1800000ms)
|
|
71
|
-
await kv.set('cache:key', { data: [] }, { ttl: 1800000 });
|
|
72
|
-
|
|
73
|
-
// Get value
|
|
74
|
-
const value = await kv.get('key');
|
|
75
|
-
console.log(value); // { foo: 'bar' }
|
|
76
|
-
|
|
77
|
-
// Delete value
|
|
78
|
-
await kv.delete('key');
|
|
79
|
-
|
|
80
|
-
// Check existence
|
|
81
|
-
const exists = await kv.has('key'); // false
|
|
82
|
-
|
|
83
|
-
// List keys with prefix
|
|
84
|
-
const keys = await kv.list({ prefix: 'cache:' });
|
|
85
|
-
|
|
86
|
-
return { operations: 'complete' };
|
|
87
|
-
});
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
---
|
|
91
|
-
|
|
92
|
-
## VersoriKVAdapter - Low-Level KV Operations
|
|
93
|
-
|
|
94
|
-
The SDK provides `VersoriKVAdapter` which wraps the native Versori KV with additional features.
|
|
95
|
-
|
|
96
|
-
### Creating Adapter
|
|
97
|
-
|
|
98
|
-
```typescript
|
|
99
|
-
import { fn } from '@versori/run';
|
|
100
|
-
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
101
|
-
|
|
102
|
-
export const useAdapter = fn('use-adapter', async ctx => {
|
|
103
|
-
const kv = ctx.openKv(':project:');
|
|
104
|
-
const adapter = new VersoriKVAdapter(kv);
|
|
105
|
-
|
|
106
|
-
// Now use adapter methods...
|
|
107
|
-
});
|
|
108
|
-
```
|
|
109
|
-
|
|
110
|
-
### VersoriKVAdapter API Reference
|
|
111
|
-
|
|
112
|
-
**IMPORTANT**: VersoriKVAdapter uses **array-based keys** `['key']` to match the SDK's `KVStore` interface.
|
|
113
|
-
|
|
114
|
-
```typescript
|
|
115
|
-
interface VersoriKVAdapter implements KVStore {
|
|
116
|
-
// Basic operations (use array keys: ['key'])
|
|
117
|
-
get(keys: string[]): Promise<{ value: unknown } | null>;
|
|
118
|
-
set(keys: string[], value: unknown): Promise<void>;
|
|
119
|
-
delete(keys: string[]): Promise<void>;
|
|
120
|
-
|
|
121
|
-
// Atomic operations (simplified - Versori doesn't support true atomicity)
|
|
122
|
-
atomic(): {
|
|
123
|
-
check(entries: Array<{ key: string[]; versionstamp: string | null }>): void;
|
|
124
|
-
set(key: string[], value: unknown): void;
|
|
125
|
-
commit(): Promise<boolean>; // Returns true if all operations succeed
|
|
126
|
-
};
|
|
127
|
-
}
|
|
128
|
-
```
|
|
129
|
-
|
|
130
|
-
**Key Limitations:**
|
|
131
|
-
- ❌ **No versionstamp support** - Versori KV doesn't expose version stamps
|
|
132
|
-
- ❌ **No expireIn/TTL option** in VersoriKVAdapter - use native Versori KV for TTL
|
|
133
|
-
- ❌ **No list() method** - use native Versori KV with prefix patterns
|
|
134
|
-
- ❌ **No atomic guarantees** - `atomic()` executes operations sequentially, not atomically
|
|
135
|
-
- ❌ **No increment()** - use read-modify-write pattern
|
|
136
|
-
|
|
137
|
-
**When to Use VersoriKVAdapter:**
|
|
138
|
-
- ✅ You need SDK `KVStore` interface compatibility (e.g., for `StateService`)
|
|
139
|
-
- ✅ You're using array-based key patterns `['namespace', 'key']`
|
|
140
|
-
- ✅ Simple get/set/delete operations are sufficient
|
|
141
|
-
|
|
142
|
-
**When to Use Native Versori KV:**
|
|
143
|
-
- ✅ You need TTL/expiration support
|
|
144
|
-
- ✅ You need to list keys with prefixes
|
|
145
|
-
- ✅ You want string-based keys `'simple:key:name'`
|
|
146
|
-
- ✅ You need increment operations
|
|
147
|
-
- ✅ Better performance (no adapter overhead)
|
|
148
|
-
|
|
149
|
-
### Practical Examples
|
|
150
|
-
|
|
151
|
-
#### Example 1: Caching Expensive API Calls
|
|
152
|
-
|
|
153
|
-
**Note**: For caching with TTL, use **native Versori KV** directly. VersoriKVAdapter doesn't support TTL.
|
|
154
|
-
|
|
155
|
-
```typescript
|
|
156
|
-
import { http } from '@versori/run';
|
|
157
|
-
import { createClient } from '@fluentcommerce/fc-connect-sdk';
|
|
158
|
-
|
|
159
|
-
export const cachedQuery = http(
|
|
160
|
-
'cached-query',
|
|
161
|
-
{
|
|
162
|
-
connection: 'fluent_commerce',
|
|
163
|
-
},
|
|
164
|
-
async ctx => {
|
|
165
|
-
const kv = ctx.openKv(':project:');
|
|
166
|
-
|
|
167
|
-
const cacheKey = 'cache:products:list';
|
|
168
|
-
const cacheTTL = 3600000; // 1 hour
|
|
169
|
-
|
|
170
|
-
// Try cache first (using native Versori KV)
|
|
171
|
-
let products = await kv.get(cacheKey);
|
|
172
|
-
|
|
173
|
-
if (products) {
|
|
174
|
-
ctx.log('info', 'Cache hit', { key: cacheKey });
|
|
175
|
-
return { source: 'cache', data: products };
|
|
176
|
-
}
|
|
177
|
-
|
|
178
|
-
// Cache miss - fetch from API
|
|
179
|
-
ctx.log('info', 'Cache miss - fetching from API');
|
|
180
|
-
const client = await createClient(ctx);
|
|
181
|
-
|
|
182
|
-
const result = await client.graphql({
|
|
183
|
-
query: `
|
|
184
|
-
query GetProducts($first: Int!) {
|
|
185
|
-
products(first: $first) {
|
|
186
|
-
edges {
|
|
187
|
-
node {
|
|
188
|
-
id
|
|
189
|
-
ref
|
|
190
|
-
name
|
|
191
|
-
status
|
|
192
|
-
}
|
|
193
|
-
}
|
|
194
|
-
}
|
|
195
|
-
}
|
|
196
|
-
`,
|
|
197
|
-
variables: { first: 100 },
|
|
198
|
-
});
|
|
199
|
-
|
|
200
|
-
products = result.data.products.edges.map(e => e.node);
|
|
201
|
-
|
|
202
|
-
// Store in cache with TTL (using native Versori KV)
|
|
203
|
-
await kv.set(cacheKey, products, { ttl: cacheTTL });
|
|
204
|
-
|
|
205
|
-
return { source: 'api', data: products };
|
|
206
|
-
}
|
|
207
|
-
);
|
|
208
|
-
```
|
|
209
|
-
|
|
210
|
-
#### Example 2: Rate Limiting with Native Versori KV
|
|
211
|
-
|
|
212
|
-
**Note**: VersoriKVAdapter doesn't have `increment()`, `expire()`, or `ttl()` methods. Use **native Versori KV** for rate limiting. StateService is for distributed locking, not rate limiting.
|
|
213
|
-
|
|
214
|
-
```typescript
|
|
215
|
-
import { fn } from '@versori/run';
|
|
216
|
-
|
|
217
|
-
export const rateLimited = fn('rate-limited', async ctx => {
|
|
218
|
-
const kv = ctx.openKv(':project:');
|
|
219
|
-
|
|
220
|
-
const userId = ctx.data?.userId || 'anonymous';
|
|
221
|
-
const rateLimitKey = `ratelimit:${userId}`;
|
|
222
|
-
const maxRequests = 100;
|
|
223
|
-
const windowMs = 3600000; // 1 hour
|
|
224
|
-
|
|
225
|
-
// Get current count using native Versori KV
|
|
226
|
-
const stored = await kv.get(rateLimitKey);
|
|
227
|
-
const data = stored as { count: number; resetAt: number } | undefined;
|
|
228
|
-
|
|
229
|
-
const now = Date.now();
|
|
230
|
-
let count = 0;
|
|
231
|
-
let resetAt = now + windowMs;
|
|
232
|
-
|
|
233
|
-
if (data && data.resetAt > now) {
|
|
234
|
-
// Within rate limit window
|
|
235
|
-
count = data.count + 1;
|
|
236
|
-
resetAt = data.resetAt;
|
|
237
|
-
} else {
|
|
238
|
-
// New window
|
|
239
|
-
count = 1;
|
|
240
|
-
resetAt = now + windowMs;
|
|
241
|
-
}
|
|
242
|
-
|
|
243
|
-
// Update count using native Versori KV
|
|
244
|
-
await kv.set(rateLimitKey, { count, resetAt });
|
|
245
|
-
|
|
246
|
-
// Check if rate limited
|
|
247
|
-
if (count > maxRequests) {
|
|
248
|
-
const retryAfter = Math.ceil((resetAt - now) / 1000);
|
|
249
|
-
return {
|
|
250
|
-
rateLimited: true,
|
|
251
|
-
retryAfter,
|
|
252
|
-
message: `Rate limit exceeded. Try again in ${retryAfter}s`,
|
|
253
|
-
};
|
|
254
|
-
}
|
|
255
|
-
|
|
256
|
-
// Process request
|
|
257
|
-
return {
|
|
258
|
-
rateLimited: false,
|
|
259
|
-
requestsRemaining: maxRequests - count,
|
|
260
|
-
};
|
|
261
|
-
});
|
|
262
|
-
```
|
|
263
|
-
|
|
264
|
-
#### Example 3: Using VersoriKVAdapter for Workflow State
|
|
265
|
-
|
|
266
|
-
**Note**: VersoriKVAdapter doesn't have `mset()` or `mget()` methods. For batch operations, use native Versori KV or loop through individual operations.
|
|
267
|
-
|
|
268
|
-
```typescript
|
|
269
|
-
import { fn } from '@versori/run';
|
|
270
|
-
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
271
|
-
|
|
272
|
-
export const workflowState = fn('workflow-state', async ctx => {
|
|
273
|
-
const kv = ctx.openKv(':project:');
|
|
274
|
-
const adapter = new VersoriKVAdapter(kv);
|
|
275
|
-
|
|
276
|
-
// Set workflow states (individual operations with array-based keys)
|
|
277
|
-
await adapter.set(['state', 'workflow1'], { status: 'running', startedAt: Date.now() });
|
|
278
|
-
await adapter.set(['state', 'workflow2'], { status: 'pending', queuedAt: Date.now() });
|
|
279
|
-
await adapter.set(['state', 'workflow3'], { status: 'completed', finishedAt: Date.now() });
|
|
280
|
-
|
|
281
|
-
// Get workflow states (individual operations)
|
|
282
|
-
const workflow1 = await adapter.get(['state', 'workflow1']);
|
|
283
|
-
const workflow2 = await adapter.get(['state', 'workflow2']);
|
|
284
|
-
const workflow3 = await adapter.get(['state', 'workflow3']);
|
|
285
|
-
|
|
286
|
-
const states = [workflow1?.value, workflow2?.value, workflow3?.value];
|
|
287
|
-
|
|
288
|
-
ctx.log('info', 'Workflow states retrieved', {
|
|
289
|
-
retrieved: states.filter(v => v !== null && v !== undefined).length,
|
|
290
|
-
});
|
|
291
|
-
|
|
292
|
-
return { states };
|
|
293
|
-
});
|
|
294
|
-
```
|
|
295
|
-
|
|
296
|
-
---
|
|
297
|
-
|
|
298
|
-
## VersoriFileTracker - Duplicate Prevention
|
|
299
|
-
|
|
300
|
-
The most common KV use case: tracking processed files to prevent duplicates.
|
|
301
|
-
|
|
302
|
-
### VersoriFileTracker API Reference
|
|
303
|
-
|
|
304
|
-
```typescript
|
|
305
|
-
interface VersoriFileTracker {
|
|
306
|
-
// Check if file was processed
|
|
307
|
-
wasFileProcessed(fileName: string): Promise<boolean>;
|
|
308
|
-
|
|
309
|
-
// Mark file as processed with metadata
|
|
310
|
-
markFileProcessed(fileName: string, metadata?: { recordCount?: number }): Promise<void>;
|
|
311
|
-
|
|
312
|
-
// Incremental sync helpers
|
|
313
|
-
setLastProcessedFile(fileName: string): Promise<void>;
|
|
314
|
-
getLastProcessedFile(): Promise<string | null>;
|
|
315
|
-
}
|
|
316
|
-
```
|
|
317
|
-
|
|
318
|
-
**Key Points**:
|
|
319
|
-
|
|
320
|
-
- **Array-based keys internally**: Uses `prefix:processed-files:${fileName}` pattern
|
|
321
|
-
- **Simple metadata**: Only stores `processedAt` timestamp and optional `recordCount`
|
|
322
|
-
- **No list/clear methods**: For advanced queries, use native Versori KV with prefix patterns
|
|
323
|
-
|
|
324
|
-
### Example 1: Basic File Tracking
|
|
325
|
-
|
|
326
|
-
```typescript
|
|
327
|
-
import { schedule } from '@versori/run';
|
|
328
|
-
import { createClient, VersoriFileTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
329
|
-
|
|
330
|
-
/**
|
|
331
|
-
* Daily inventory sync with duplicate prevention
|
|
332
|
-
*
|
|
333
|
-
* Cron: 0 2 * * * (daily at 2 AM)
|
|
334
|
-
*/
|
|
335
|
-
export const dailySyncWithTracking = schedule(
|
|
336
|
-
'daily-sync',
|
|
337
|
-
'0 2 * * *',
|
|
338
|
-
{
|
|
339
|
-
connection: 'fluent_commerce',
|
|
340
|
-
},
|
|
341
|
-
async ctx => {
|
|
342
|
-
const kv = ctx.openKv(':project:');
|
|
343
|
-
const tracker = new VersoriFileTracker(kv, 'daily-inventory-sync');
|
|
344
|
-
|
|
345
|
-
// Generate file name based on date
|
|
346
|
-
const today = new Date().toISOString().split('T')[0];
|
|
347
|
-
const fileName = `inventory-${today}.csv`;
|
|
348
|
-
|
|
349
|
-
// Check if already processed
|
|
350
|
-
if (await tracker.wasFileProcessed(fileName)) {
|
|
351
|
-
ctx.log('info', 'File already processed today', { fileName });
|
|
352
|
-
return {
|
|
353
|
-
skipped: true,
|
|
354
|
-
reason: 'Already processed',
|
|
355
|
-
fileName,
|
|
356
|
-
};
|
|
357
|
-
}
|
|
358
|
-
|
|
359
|
-
// Process file
|
|
360
|
-
ctx.log('info', 'Processing new file', { fileName });
|
|
361
|
-
const records = await fetchAndParseInventoryFile(fileName);
|
|
362
|
-
|
|
363
|
-
// Send to Fluent
|
|
364
|
-
const client = await createClient(ctx);
|
|
365
|
-
const job = await client.createJob({ name: `Daily Sync ${fileName}` });
|
|
366
|
-
await client.sendBatch(job.id, {
|
|
367
|
-
action: 'UPSERT',
|
|
368
|
-
entityType: 'INVENTORY',
|
|
369
|
-
entities: records,
|
|
370
|
-
});
|
|
371
|
-
|
|
372
|
-
// Mark as processed with metadata
|
|
373
|
-
await tracker.markFileProcessed(fileName, {
|
|
374
|
-
recordCount: records.length,
|
|
375
|
-
jobId: job.id,
|
|
376
|
-
timestamp: Date.now(),
|
|
377
|
-
});
|
|
378
|
-
|
|
379
|
-
ctx.log('info', 'File processed successfully', {
|
|
380
|
-
fileName,
|
|
381
|
-
recordCount: records.length,
|
|
382
|
-
jobId: job.id,
|
|
383
|
-
});
|
|
384
|
-
|
|
385
|
-
return {
|
|
386
|
-
success: true,
|
|
387
|
-
fileName,
|
|
388
|
-
recordCount: records.length,
|
|
389
|
-
jobId: job.id,
|
|
390
|
-
};
|
|
391
|
-
}
|
|
392
|
-
);
|
|
393
|
-
```
|
|
394
|
-
|
|
395
|
-
### Example 2: Incremental Processing
|
|
396
|
-
|
|
397
|
-
```typescript
|
|
398
|
-
export const incrementalSync = schedule(
|
|
399
|
-
'incremental-sync',
|
|
400
|
-
'0 * * * *',
|
|
401
|
-
{
|
|
402
|
-
connection: 'fluent_commerce',
|
|
403
|
-
},
|
|
404
|
-
async ctx => {
|
|
405
|
-
const kv = ctx.openKv(':project:');
|
|
406
|
-
const tracker = new VersoriFileTracker(kv, 'incremental-sync');
|
|
407
|
-
|
|
408
|
-
// Get last processed file
|
|
409
|
-
const lastFile = await tracker.getLastProcessedFile();
|
|
410
|
-
ctx.log('info', 'Last processed file', { lastFile });
|
|
411
|
-
|
|
412
|
-
// List new files since last run
|
|
413
|
-
const newFiles = await listFilesAfter(lastFile);
|
|
414
|
-
|
|
415
|
-
if (newFiles.length === 0) {
|
|
416
|
-
ctx.log('info', 'No new files to process');
|
|
417
|
-
return { skipped: true, reason: 'No new files' };
|
|
418
|
-
}
|
|
419
|
-
|
|
420
|
-
const client = await createClient(ctx);
|
|
421
|
-
let totalProcessed = 0;
|
|
422
|
-
|
|
423
|
-
for (const file of newFiles) {
|
|
424
|
-
// Skip if already processed (safety check)
|
|
425
|
-
if (await tracker.wasFileProcessed(file.name)) {
|
|
426
|
-
ctx.log('warn', 'File already processed (safety skip)', { file: file.name });
|
|
427
|
-
continue;
|
|
428
|
-
}
|
|
429
|
-
|
|
430
|
-
// Process file
|
|
431
|
-
const records = await processFile(file);
|
|
432
|
-
|
|
433
|
-
// Send to Fluent
|
|
434
|
-
const job = await client.createJob({ name: `Incremental ${file.name}` });
|
|
435
|
-
await client.sendBatch(job.id, {
|
|
436
|
-
action: 'UPSERT',
|
|
437
|
-
entityType: 'INVENTORY',
|
|
438
|
-
entities: records,
|
|
439
|
-
});
|
|
440
|
-
|
|
441
|
-
// Mark as processed
|
|
442
|
-
await tracker.markFileProcessed(file.name, {
|
|
443
|
-
recordCount: records.length,
|
|
444
|
-
jobId: job.id,
|
|
445
|
-
});
|
|
446
|
-
|
|
447
|
-
// Update last processed file
|
|
448
|
-
await tracker.setLastProcessedFile(file.name);
|
|
449
|
-
|
|
450
|
-
totalProcessed += records.length;
|
|
451
|
-
|
|
452
|
-
ctx.log('info', 'File processed', {
|
|
453
|
-
fileName: file.name,
|
|
454
|
-
recordCount: records.length,
|
|
455
|
-
});
|
|
456
|
-
}
|
|
457
|
-
|
|
458
|
-
return {
|
|
459
|
-
success: true,
|
|
460
|
-
filesProcessed: newFiles.length,
|
|
461
|
-
recordsProcessed: totalProcessed,
|
|
462
|
-
};
|
|
463
|
-
}
|
|
464
|
-
);
|
|
465
|
-
```
|
|
466
|
-
|
|
467
|
-
### Example 3: Checking Last Processed File
|
|
468
|
-
|
|
469
|
-
**Note**: VersoriFileTracker doesn't have `listProcessedFiles()` or `getFileProcessingInfo()` methods. Use `getLastProcessedFile()` to track incremental sync state.
|
|
470
|
-
|
|
471
|
-
```typescript
|
|
472
|
-
import { fn } from '@versori/run';
|
|
473
|
-
import { VersoriFileTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
474
|
-
|
|
475
|
-
export const checkProcessingState = fn('check-state', async ctx => {
|
|
476
|
-
const kv = ctx.openKv(':project:');
|
|
477
|
-
const tracker = new VersoriFileTracker(kv, 'daily-inventory-sync');
|
|
478
|
-
|
|
479
|
-
// Get last processed file
|
|
480
|
-
const lastFile = await tracker.getLastProcessedFile();
|
|
481
|
-
|
|
482
|
-
// Check if specific files were processed
|
|
483
|
-
const today = new Date().toISOString().split('T')[0];
|
|
484
|
-
const todayFile = `inventory-${today}.csv`;
|
|
485
|
-
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0];
|
|
486
|
-
const yesterdayFile = `inventory-${yesterday}.csv`;
|
|
487
|
-
|
|
488
|
-
const todayProcessed = await tracker.wasFileProcessed(todayFile);
|
|
489
|
-
const yesterdayProcessed = await tracker.wasFileProcessed(yesterdayFile);
|
|
490
|
-
|
|
491
|
-
ctx.log('info', 'Processing state', {
|
|
492
|
-
lastFile,
|
|
493
|
-
todayProcessed,
|
|
494
|
-
yesterdayProcessed,
|
|
495
|
-
});
|
|
496
|
-
|
|
497
|
-
return {
|
|
498
|
-
lastFile,
|
|
499
|
-
todayFile: {
|
|
500
|
-
name: todayFile,
|
|
501
|
-
processed: todayProcessed,
|
|
502
|
-
},
|
|
503
|
-
yesterdayFile: {
|
|
504
|
-
name: yesterdayFile,
|
|
505
|
-
processed: yesterdayProcessed,
|
|
506
|
-
},
|
|
507
|
-
};
|
|
508
|
-
});
|
|
509
|
-
```
|
|
510
|
-
|
|
511
|
-
**For viewing all processed files**, use native Versori KV with prefix listing:
|
|
512
|
-
|
|
513
|
-
```typescript
|
|
514
|
-
export const listAllProcessedFiles = fn('list-processed', async ctx => {
|
|
515
|
-
const kv = ctx.openKv(':project:');
|
|
516
|
-
const prefix = 'fc-sdk:processed-files:';
|
|
517
|
-
|
|
518
|
-
// List all keys with prefix (native Versori KV)
|
|
519
|
-
const keys = await kv.list({ prefix });
|
|
520
|
-
|
|
521
|
-
const files = [];
|
|
522
|
-
for (const key of keys) {
|
|
523
|
-
const fileName = key.replace(prefix, '');
|
|
524
|
-
const metadata = await kv.get(key);
|
|
525
|
-
files.push({
|
|
526
|
-
fileName,
|
|
527
|
-
processedAt: metadata?.processedAt,
|
|
528
|
-
recordCount: metadata?.recordCount,
|
|
529
|
-
});
|
|
530
|
-
}
|
|
531
|
-
|
|
532
|
-
// Sort by date (newest first)
|
|
533
|
-
files.sort(
|
|
534
|
-
(a, b) => new Date(b.processedAt || 0).getTime() - new Date(a.processedAt || 0).getTime()
|
|
535
|
-
);
|
|
536
|
-
|
|
537
|
-
return { totalFiles: files.length, files };
|
|
538
|
-
});
|
|
539
|
-
```
|
|
540
|
-
|
|
541
|
-
---
|
|
542
|
-
|
|
543
|
-
## Distributed Locking
|
|
544
|
-
|
|
545
|
-
Prevent concurrent execution of critical sections using distributed locks.
|
|
546
|
-
|
|
547
|
-
### StateService with Locking
|
|
548
|
-
|
|
549
|
-
```typescript
|
|
550
|
-
import { schedule } from '@versori/run';
|
|
551
|
-
import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
552
|
-
|
|
553
|
-
/**
|
|
554
|
-
* Scheduled workflow with distributed locking
|
|
555
|
-
*
|
|
556
|
-
* Ensures only ONE instance runs at a time
|
|
557
|
-
*/
|
|
558
|
-
export const criticalSection = schedule(
|
|
559
|
-
'critical',
|
|
560
|
-
'*/5 * * * *',
|
|
561
|
-
{
|
|
562
|
-
connection: 'fluent_commerce',
|
|
563
|
-
},
|
|
564
|
-
async ctx => {
|
|
565
|
-
const kv = ctx.openKv(':project:');
|
|
566
|
-
const kvAdapter = new VersoriKVAdapter(kv);
|
|
567
|
-
const stateService = new StateService(logger);
|
|
568
|
-
|
|
569
|
-
const lockName = 'critical-section';
|
|
570
|
-
const timeoutMinutes = 1; // 1 minute timeout
|
|
571
|
-
|
|
572
|
-
// Try to acquire lock (returns boolean)
|
|
573
|
-
const acquired = await stateService.acquireLock(lockName, kvAdapter, timeoutMinutes);
|
|
574
|
-
|
|
575
|
-
if (!acquired) {
|
|
576
|
-
ctx.log('warn', 'Lock acquisition failed - another instance is running', {
|
|
577
|
-
lockName,
|
|
578
|
-
});
|
|
579
|
-
return {
|
|
580
|
-
skipped: true,
|
|
581
|
-
reason: 'Another instance is running',
|
|
582
|
-
};
|
|
583
|
-
}
|
|
584
|
-
|
|
585
|
-
ctx.log('info', 'Lock acquired, starting critical section', {
|
|
586
|
-
lockName,
|
|
587
|
-
});
|
|
588
|
-
|
|
589
|
-
try {
|
|
590
|
-
// Critical section - only ONE workflow instance executes this
|
|
591
|
-
await performCriticalOperation(ctx);
|
|
592
|
-
|
|
593
|
-
ctx.log('info', 'Critical section completed successfully');
|
|
594
|
-
|
|
595
|
-
return { success: true };
|
|
596
|
-
} finally {
|
|
597
|
-
// Always release lock
|
|
598
|
-
await stateService.releaseLock(lockName, kvAdapter);
|
|
599
|
-
ctx.log('info', 'Lock released', { lockName });
|
|
600
|
-
}
|
|
601
|
-
}
|
|
602
|
-
);
|
|
603
|
-
```
|
|
604
|
-
|
|
605
|
-
### Advanced Locking Pattern
|
|
606
|
-
|
|
607
|
-
```typescript
|
|
608
|
-
export const advancedLocking = schedule(
|
|
609
|
-
'advanced-locking',
|
|
610
|
-
'*/10 * * * *',
|
|
611
|
-
{
|
|
612
|
-
connection: 'fluent_commerce',
|
|
613
|
-
},
|
|
614
|
-
async ctx => {
|
|
615
|
-
const kv = ctx.openKv(':project:');
|
|
616
|
-
const kvAdapter = new VersoriKVAdapter(kv);
|
|
617
|
-
const stateService = new StateService(logger);
|
|
618
|
-
|
|
619
|
-
const lockName = 'inventory-sync';
|
|
620
|
-
const timeoutMinutes = 5; // 5 minutes timeout
|
|
621
|
-
const maxRetries = 3;
|
|
622
|
-
const retryDelay = 5000; // 5 seconds
|
|
623
|
-
|
|
624
|
-
let acquired = false;
|
|
625
|
-
let attempts = 0;
|
|
626
|
-
|
|
627
|
-
// Retry lock acquisition
|
|
628
|
-
while (attempts < maxRetries) {
|
|
629
|
-
acquired = await stateService.acquireLock(lockName, kvAdapter, timeoutMinutes);
|
|
630
|
-
|
|
631
|
-
if (acquired) {
|
|
632
|
-
break;
|
|
633
|
-
}
|
|
634
|
-
|
|
635
|
-
attempts++;
|
|
636
|
-
ctx.log('info', `Lock acquisition failed, retrying... (${attempts}/${maxRetries})`);
|
|
637
|
-
await new Promise(resolve => setTimeout(resolve, retryDelay));
|
|
638
|
-
}
|
|
639
|
-
|
|
640
|
-
if (!acquired) {
|
|
641
|
-
ctx.log('error', 'Failed to acquire lock after retries', {
|
|
642
|
-
attempts,
|
|
643
|
-
});
|
|
644
|
-
return {
|
|
645
|
-
success: false,
|
|
646
|
-
error: 'Failed to acquire lock',
|
|
647
|
-
attempts,
|
|
648
|
-
};
|
|
649
|
-
}
|
|
650
|
-
|
|
651
|
-
try {
|
|
652
|
-
// Long-running operation
|
|
653
|
-
const result = await performLongOperation(ctx);
|
|
654
|
-
|
|
655
|
-
// Note: StateService doesn't have extendLock() - acquire a new lock if needed
|
|
656
|
-
// If operation needs more time, acquire a new lock before releasing
|
|
657
|
-
if (result.needsMoreTime) {
|
|
658
|
-
// Release current lock and acquire new one
|
|
659
|
-
await stateService.releaseLock(lockName, kvAdapter);
|
|
660
|
-
const reacquired = await stateService.acquireLock(lockName, kvAdapter, timeoutMinutes);
|
|
661
|
-
if (reacquired) {
|
|
662
|
-
ctx.log('info', 'Lock reacquired for extended operation');
|
|
663
|
-
}
|
|
664
|
-
}
|
|
665
|
-
|
|
666
|
-
return { success: true, result };
|
|
667
|
-
} catch (error) {
|
|
668
|
-
ctx.log('error', 'Operation failed', {
|
|
669
|
-
error: error instanceof Error ? error.message : String(error),
|
|
670
|
-
});
|
|
671
|
-
throw error;
|
|
672
|
-
} finally {
|
|
673
|
-
await stateService.releaseLock(lockName, kvAdapter);
|
|
674
|
-
ctx.log('info', 'Lock released');
|
|
675
|
-
}
|
|
676
|
-
}
|
|
677
|
-
);
|
|
678
|
-
```
|
|
679
|
-
|
|
680
|
-
---
|
|
681
|
-
|
|
682
|
-
## KV Namespace Patterns
|
|
683
|
-
|
|
684
|
-
### Scoping Best Practices
|
|
685
|
-
|
|
686
|
-
| Namespace | Use Case | Scope | Example Key |
|
|
687
|
-
| ---------------- | --------------------------- | ------------------- | ---------------------------- |
|
|
688
|
-
| `:project:` | Global config, shared state | All workflows | `config:retailerId` |
|
|
689
|
-
| `:workflow:name` | Workflow-specific state | Single workflow | `state:lastRun` |
|
|
690
|
-
| `cache:*` | Cached data | Project-wide | `cache:products:list` |
|
|
691
|
-
| `lock:*` | Distributed locks | Project-wide | `lock:inventory-sync` |
|
|
692
|
-
| `files:*` | File tracking | Workflow or project | `files:processed:2025-01-15` |
|
|
693
|
-
| `ratelimit:*` | Rate limiting | Per user/IP | `ratelimit:user:123` |
|
|
694
|
-
|
|
695
|
-
### Namespace Examples
|
|
696
|
-
|
|
697
|
-
```typescript
|
|
698
|
-
import { fn } from '@versori/run';
|
|
699
|
-
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
700
|
-
|
|
701
|
-
export const namespaceExamples = fn('namespace-examples', async ctx => {
|
|
702
|
-
// 1. Project-level configuration (native Versori KV)
|
|
703
|
-
const projectKV = ctx.openKv(':project:');
|
|
704
|
-
await projectKV.set('config:retailerId', '1');
|
|
705
|
-
await projectKV.set('config:environment', 'production');
|
|
706
|
-
|
|
707
|
-
// 2. Workflow-specific state (native Versori KV)
|
|
708
|
-
const workflowKV = ctx.openKv(':workflow:inventory-sync');
|
|
709
|
-
await workflowKV.set('state:lastRun', new Date().toISOString());
|
|
710
|
-
await workflowKV.set('state:lastCursor', 'cursor-value');
|
|
711
|
-
|
|
712
|
-
// 3. Custom namespace for caching (native Versori KV with TTL)
|
|
713
|
-
const cacheKV = ctx.openKv('cache:inventory');
|
|
714
|
-
await cacheKV.set('products', [], { ttl: 3600000 });
|
|
715
|
-
|
|
716
|
-
// 4. Custom namespace with VersoriKVAdapter (requires array-based keys)
|
|
717
|
-
const filesKV = ctx.openKv('files:daily-sync');
|
|
718
|
-
const adapter = new VersoriKVAdapter(filesKV);
|
|
719
|
-
// VersoriKVAdapter ALWAYS uses array syntax: ['processed', '2025-01-15']
|
|
720
|
-
await adapter.set(['processed', '2025-01-15'], { processedAt: Date.now() });
|
|
721
|
-
|
|
722
|
-
return { namespacesConfigured: 4 };
|
|
723
|
-
});
|
|
724
|
-
```
|
|
725
|
-
|
|
726
|
-
---
|
|
727
|
-
|
|
728
|
-
## Performance Optimization
|
|
729
|
-
|
|
730
|
-
### 1. Minimize Individual KV Operations
|
|
731
|
-
|
|
732
|
-
**Note**: VersoriKVAdapter doesn't have batch methods. Use native Versori KV for individual operations.
|
|
733
|
-
|
|
734
|
-
```typescript
|
|
735
|
-
import { fn } from '@versori/run';
|
|
736
|
-
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
737
|
-
|
|
738
|
-
// ❌ Inefficient - N separate KV operations
|
|
739
|
-
export const inefficientWrites = fn('inefficient', async ctx => {
|
|
740
|
-
const kv = ctx.openKv(':project:');
|
|
741
|
-
const adapter = new VersoriKVAdapter(kv);
|
|
742
|
-
|
|
743
|
-
for (const item of items) {
|
|
744
|
-
await adapter.set(['item', item.id], item); // N operations
|
|
745
|
-
}
|
|
746
|
-
});
|
|
747
|
-
|
|
748
|
-
// ✅ Better - Use native Versori KV for multiple writes
|
|
749
|
-
export const efficientWrites = fn('efficient', async ctx => {
|
|
750
|
-
const kv = ctx.openKv(':project:');
|
|
751
|
-
|
|
752
|
-
// Native KV operations are faster
|
|
753
|
-
for (const item of items) {
|
|
754
|
-
await kv.set(`item:${item.id}`, item);
|
|
755
|
-
}
|
|
756
|
-
});
|
|
757
|
-
|
|
758
|
-
// ✅ Best - Store as single aggregate if possible
|
|
759
|
-
export const aggregateWrite = fn('aggregate', async ctx => {
|
|
760
|
-
const kv = ctx.openKv(':project:');
|
|
761
|
-
|
|
762
|
-
// Store all items in one key (if items are related)
|
|
763
|
-
await kv.set('items:batch', items); // 1 operation
|
|
764
|
-
});
|
|
765
|
-
```
|
|
766
|
-
|
|
767
|
-
### 2. Set Appropriate TTLs
|
|
768
|
-
|
|
769
|
-
```typescript
|
|
770
|
-
// Cache duration guidelines:
|
|
771
|
-
const TTL = {
|
|
772
|
-
VERY_SHORT: 60000, // 1 minute - volatile data
|
|
773
|
-
SHORT: 300000, // 5 minutes - frequently changing
|
|
774
|
-
MEDIUM: 1800000, // 30 minutes - semi-static
|
|
775
|
-
LONG: 3600000, // 1 hour - stable data
|
|
776
|
-
VERY_LONG: 86400000, // 24 hours - rarely changing
|
|
777
|
-
PERMANENT: undefined, // No TTL - manual cleanup
|
|
778
|
-
};
|
|
779
|
-
|
|
780
|
-
// Use based on data characteristics
|
|
781
|
-
await adapter.set(['cache', 'products'], products, TTL.MEDIUM);
|
|
782
|
-
await adapter.set(['cache', 'config'], config, TTL.VERY_LONG);
|
|
783
|
-
await adapter.set(['session', 'user', '123'], session, TTL.SHORT);
|
|
784
|
-
```
|
|
785
|
-
|
|
786
|
-
### 3. Minimize KV Reads/Writes
|
|
787
|
-
|
|
788
|
-
**Note**: VersoriKVAdapter doesn't have `increment()` method. Use read-modify-write pattern.
|
|
789
|
-
|
|
790
|
-
```typescript
|
|
791
|
-
import { fn } from '@versori/run';
|
|
792
|
-
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
793
|
-
|
|
794
|
-
// ❌ Inefficient - read/write in loop
|
|
795
|
-
export const inefficientCounter = fn('inefficient', async ctx => {
|
|
796
|
-
const kv = ctx.openKv(':project:');
|
|
797
|
-
|
|
798
|
-
for (let i = 0; i < 100; i++) {
|
|
799
|
-
const count = (await kv.get('counter')) || 0;
|
|
800
|
-
await kv.set('counter', count + 1); // 100 read + 100 write operations
|
|
801
|
-
}
|
|
802
|
-
});
|
|
803
|
-
|
|
804
|
-
// ✅ Better - Single read-modify-write
|
|
805
|
-
export const efficientCounter = fn('efficient', async ctx => {
|
|
806
|
-
const kv = ctx.openKv(':project:');
|
|
807
|
-
const adapter = new VersoriKVAdapter(kv);
|
|
808
|
-
|
|
809
|
-
// Get current count
|
|
810
|
-
const stored = await adapter.get(['counter']);
|
|
811
|
-
const currentCount = (stored?.value as number) || 0;
|
|
812
|
-
|
|
813
|
-
// Update once
|
|
814
|
-
await adapter.set(['counter'], currentCount + 100); // 1 read + 1 write
|
|
815
|
-
|
|
816
|
-
return { count: currentCount + 100 };
|
|
817
|
-
});
|
|
818
|
-
|
|
819
|
-
// ✅ Best - Use atomic operations if needed
|
|
820
|
-
export const atomicCounter = fn('atomic', async ctx => {
|
|
821
|
-
const kv = ctx.openKv(':project:');
|
|
822
|
-
const adapter = new VersoriKVAdapter(kv);
|
|
823
|
-
|
|
824
|
-
const atomic = adapter.atomic();
|
|
825
|
-
atomic.set(['counter'], 100);
|
|
826
|
-
const success = await atomic.commit();
|
|
827
|
-
|
|
828
|
-
return { success };
|
|
829
|
-
});
|
|
830
|
-
```
|
|
831
|
-
|
|
832
|
-
### 4. Use Prefix-Based Cleanup
|
|
833
|
-
|
|
834
|
-
**Note**: VersoriKVAdapter doesn't have `clear()` or `keys()` methods. Use **native Versori KV** for prefix-based operations.
|
|
835
|
-
|
|
836
|
-
```typescript
|
|
837
|
-
import { fn } from '@versori/run';
|
|
838
|
-
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
839
|
-
|
|
840
|
-
export const cleanupCache = fn('cleanup-cache', async ctx => {
|
|
841
|
-
const kv = ctx.openKv(':project:');
|
|
842
|
-
|
|
843
|
-
// List all cache keys (native Versori KV)
|
|
844
|
-
const cacheKeys = await kv.list({ prefix: 'cache:' });
|
|
845
|
-
|
|
846
|
-
// Delete each cache entry
|
|
847
|
-
for (const key of cacheKeys) {
|
|
848
|
-
await kv.delete(key);
|
|
849
|
-
}
|
|
850
|
-
|
|
851
|
-
// Clear old sessions (older than 1 day)
|
|
852
|
-
const sessionKeys = await kv.list({ prefix: 'session:' });
|
|
853
|
-
const now = Date.now();
|
|
854
|
-
let deletedCount = 0;
|
|
855
|
-
|
|
856
|
-
for (const key of sessionKeys) {
|
|
857
|
-
const session = await kv.get(key);
|
|
858
|
-
if (session && now - session.createdAt > 86400000) {
|
|
859
|
-
await kv.delete(key);
|
|
860
|
-
deletedCount++;
|
|
861
|
-
}
|
|
862
|
-
}
|
|
863
|
-
|
|
864
|
-
ctx.log('info', 'Cleanup complete', {
|
|
865
|
-
cacheKeysDeleted: cacheKeys.length,
|
|
866
|
-
sessionsChecked: sessionKeys.length,
|
|
867
|
-
oldSessionsDeleted: deletedCount,
|
|
868
|
-
});
|
|
869
|
-
|
|
870
|
-
return {
|
|
871
|
-
cacheKeysDeleted: cacheKeys.length,
|
|
872
|
-
oldSessionsDeleted: deletedCount,
|
|
873
|
-
};
|
|
874
|
-
});
|
|
875
|
-
```
|
|
876
|
-
|
|
877
|
-
---
|
|
878
|
-
|
|
879
|
-
## Troubleshooting KV Storage
|
|
880
|
-
|
|
881
|
-
### Common Issues & Solutions
|
|
882
|
-
|
|
883
|
-
| Issue | Symptoms | Solution |
|
|
884
|
-
| --------------------------- | ------------------------------- | ---------------------------------------------------- |
|
|
885
|
-
| **Data not persisting** | Values disappear after workflow | Check TTL settings, ensure no accidental `clear()` |
|
|
886
|
-
| **Namespace conflicts** | Unexpected data in keys | Use unique namespaces per workflow/feature |
|
|
887
|
-
| **Performance degradation** | Slow KV operations | Use batch operations, optimize key structure |
|
|
888
|
-
| **Lock deadlocks** | Workflows stuck waiting | Ensure locks are always released in `finally` blocks |
|
|
889
|
-
| **Memory issues** | Large values causing errors | Split large objects, use pagination |
|
|
890
|
-
|
|
891
|
-
### Debug KV State
|
|
892
|
-
|
|
893
|
-
**Note**: Use **native Versori KV** for listing and debugging operations.
|
|
894
|
-
|
|
895
|
-
```typescript
|
|
896
|
-
import { fn } from '@versori/run';
|
|
897
|
-
|
|
898
|
-
export const debugKV = fn('debug-kv', async ctx => {
|
|
899
|
-
const kv = ctx.openKv(':project:');
|
|
900
|
-
|
|
901
|
-
// List all keys (native Versori KV)
|
|
902
|
-
const allKeys = await kv.list();
|
|
903
|
-
|
|
904
|
-
// Group by prefix
|
|
905
|
-
const grouped: Record<string, string[]> = {};
|
|
906
|
-
for (const key of allKeys) {
|
|
907
|
-
const prefix = key.split(':')[0];
|
|
908
|
-
if (!grouped[prefix]) {
|
|
909
|
-
grouped[prefix] = [];
|
|
910
|
-
}
|
|
911
|
-
grouped[prefix].push(key);
|
|
912
|
-
}
|
|
913
|
-
|
|
914
|
-
// Sample some values
|
|
915
|
-
const samples: Record<string, unknown> = {};
|
|
916
|
-
for (const [prefix, keys] of Object.entries(grouped)) {
|
|
917
|
-
const sampleKey = keys[0];
|
|
918
|
-
samples[sampleKey] = await kv.get(sampleKey);
|
|
919
|
-
}
|
|
920
|
-
|
|
921
|
-
ctx.log('info', 'KV Debug Info', {
|
|
922
|
-
totalKeys: allKeys.length,
|
|
923
|
-
namespaces: Object.keys(grouped),
|
|
924
|
-
keysByNamespace: Object.fromEntries(Object.entries(grouped).map(([k, v]) => [k, v.length])),
|
|
925
|
-
samples,
|
|
926
|
-
});
|
|
927
|
-
|
|
928
|
-
return {
|
|
929
|
-
totalKeys: allKeys.length,
|
|
930
|
-
namespaces: grouped,
|
|
931
|
-
keysByNamespace: Object.fromEntries(Object.entries(grouped).map(([k, v]) => [k, v.length])),
|
|
932
|
-
samples,
|
|
933
|
-
};
|
|
934
|
-
});
|
|
935
|
-
```
|
|
936
|
-
|
|
937
|
-
---
|
|
938
|
-
|
|
939
|
-
## Key Takeaways
|
|
940
|
-
|
|
941
|
-
- 🎯 **VersoriKVAdapter** provides array-based key interface matching SDK's KVStore (use `['key']` syntax)
|
|
942
|
-
- 🎯 **Native Versori KV** recommended for TTL, increment, list, and prefix operations
|
|
943
|
-
- 🎯 **VersoriFileTracker** prevents duplicate file processing with simple metadata (`wasFileProcessed`, `markFileProcessed`)
|
|
944
|
-
- 🎯 **Distributed locking** via StateService ensures concurrent workflow safety
|
|
945
|
-
- 🎯 **Namespace scoping** (`:project:`, `:workflow:`, custom) provides isolation
|
|
946
|
-
- 🎯 **TTL optimization** reduces storage costs - use native Versori KV for TTL support
|
|
947
|
-
- 🎯 **Minimize KV operations** - aggregate data when possible, avoid loops
|
|
948
|
-
- 🎯 **Always release locks** in `finally` blocks to prevent deadlocks
|
|
949
|
-
|
|
950
|
-
---
|
|
951
|
-
|
|
952
|
-
## Practice Exercise
|
|
953
|
-
|
|
954
|
-
Create a workflow that:
|
|
955
|
-
|
|
956
|
-
1. Uses VersoriFileTracker to prevent duplicate processing
|
|
957
|
-
2. Implements distributed locking for critical section
|
|
958
|
-
3. Caches expensive API results with 30-minute TTL
|
|
959
|
-
4. Tracks processing statistics using atomic counters
|
|
960
|
-
5. Provides debug endpoint to view KV state
|
|
961
|
-
|
|
962
|
-
**Hints**:
|
|
963
|
-
|
|
964
|
-
- Combine VersoriFileTracker + StateService + VersoriKVAdapter
|
|
965
|
-
- Use appropriate TTLs for cache vs state
|
|
966
|
-
- Implement cleanup for old cache entries
|
|
967
|
-
- Use atomic increment for statistics
|
|
968
|
-
|
|
969
|
-
**Solution** available in [Module 8: Best Practices](./platforms-versori-08-best-practices.md#practice-solutions)
|
|
970
|
-
|
|
971
|
-
---
|
|
972
|
-
|
|
973
|
-
## Next Steps
|
|
974
|
-
|
|
975
|
-
Now that you've mastered KV storage, let's explore deployment patterns and monitoring.
|
|
976
|
-
|
|
977
|
-
Continue to [Module 7: Deployment →](./platforms-versori-07-deployment.md) to learn about deploying connectors, CI/CD, monitoring, and production operations.
|
|
978
|
-
|
|
979
|
-
---
|
|
980
|
-
|
|
981
|
-
## Related Documentation
|
|
982
|
-
|
|
983
|
-
- [Module 4: Workflows](./platforms-versori-04-workflows.md) - Using KV in workflows
|
|
984
|
-
- [Module 7: Deployment](./platforms-versori-07-deployment.md) - Production deployment
|
|
985
|
-
- [Module 8: Best Practices](./platforms-versori-08-best-practices.md) - Production patterns
|
|
986
|
-
- [API Reference](../../../../02-CORE-GUIDES/api-reference/api-reference-readme.md) - Complete SDK API
|
|
987
|
-
|
|
988
|
-
---
|
|
989
|
-
|
|
990
|
-
[← Previous: Module 5](./platforms-versori-05-connections.md) | [Back to Guide](../platforms-versori-readme.md) | [Next: Module 7: Deployment →](./platforms-versori-07-deployment.md)
|
|
1
|
+
# Module 6: KV Storage - State Management & File Tracking
|
|
2
|
+
|
|
3
|
+
[← Back to Versori Platform Guide](../platforms-versori-readme.md)
|
|
4
|
+
|
|
5
|
+
**Module 6 of 8** | **Level**: Intermediate | **Time**: 25 minutes
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Learning Objectives
|
|
10
|
+
|
|
11
|
+
By the end of this module, you will:
|
|
12
|
+
|
|
13
|
+
- ✅ Understand Versori KV storage architecture and capabilities
|
|
14
|
+
- ✅ Use VersoriKVAdapter for low-level KV operations
|
|
15
|
+
- ✅ Implement VersoriFileTracker for duplicate prevention
|
|
16
|
+
- ✅ Master KV scoping and namespace patterns
|
|
17
|
+
- ✅ Implement distributed locking for concurrent workflows
|
|
18
|
+
- ✅ Optimize KV storage for performance and cost
|
|
19
|
+
- ✅ Debug and troubleshoot KV storage issues
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## KV Storage Overview
|
|
24
|
+
|
|
25
|
+
Versori provides a distributed key-value store accessible from all workflow types. It's designed for:
|
|
26
|
+
|
|
27
|
+
| Use Case | Pattern | SDK Component |
|
|
28
|
+
| ----------------------- | ---------------------------- | ------------------------------------------- |
|
|
29
|
+
| **File tracking** | Prevent duplicate processing | `VersoriFileTracker` |
|
|
30
|
+
| **State management** | Workflow state persistence | `StateService` + `VersoriKVAdapter` |
|
|
31
|
+
| **Distributed locking** | Prevent concurrent execution | `StateService.acquireLock()` |
|
|
32
|
+
| **Caching** | Cache expensive API calls | Native Versori KV with TTL |
|
|
33
|
+
| **Rate limiting** | Track request counts | Native Versori KV (custom implementation) |
|
|
34
|
+
| **Incremental sync** | Track last processed cursor | `VersoriFileTracker.setLastProcessedFile()` |
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## KV Storage Basics
|
|
39
|
+
|
|
40
|
+
### Opening KV Store
|
|
41
|
+
|
|
42
|
+
```typescript
|
|
43
|
+
import { fn } from '@versori/run';
|
|
44
|
+
|
|
45
|
+
export const kvBasics = fn('kv-basics', async ctx => {
|
|
46
|
+
// Three scoping options:
|
|
47
|
+
|
|
48
|
+
// 1. Project scope - shared across ALL workflows
|
|
49
|
+
const projectKV = ctx.openKv(':project:');
|
|
50
|
+
|
|
51
|
+
// 2. Workflow scope - isolated to THIS workflow
|
|
52
|
+
const workflowKV = ctx.openKv(':workflow:my-workflow');
|
|
53
|
+
|
|
54
|
+
// 3. Custom scope - custom namespace
|
|
55
|
+
const customKV = ctx.openKv('custom:namespace:name');
|
|
56
|
+
|
|
57
|
+
return { message: 'KV stores opened' };
|
|
58
|
+
});
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### Basic Operations
|
|
62
|
+
|
|
63
|
+
```typescript
|
|
64
|
+
export const kvOperations = fn('kv-operations', async ctx => {
|
|
65
|
+
const kv = ctx.openKv(':project:');
|
|
66
|
+
|
|
67
|
+
// Set value
|
|
68
|
+
await kv.set('key', { foo: 'bar' });
|
|
69
|
+
|
|
70
|
+
// Set with TTL (30 minutes = 1800000ms)
|
|
71
|
+
await kv.set('cache:key', { data: [] }, { ttl: 1800000 });
|
|
72
|
+
|
|
73
|
+
// Get value
|
|
74
|
+
const value = await kv.get('key');
|
|
75
|
+
console.log(value); // { foo: 'bar' }
|
|
76
|
+
|
|
77
|
+
// Delete value
|
|
78
|
+
await kv.delete('key');
|
|
79
|
+
|
|
80
|
+
// Check existence
|
|
81
|
+
const exists = await kv.has('key'); // false
|
|
82
|
+
|
|
83
|
+
// List keys with prefix
|
|
84
|
+
const keys = await kv.list({ prefix: 'cache:' });
|
|
85
|
+
|
|
86
|
+
return { operations: 'complete' };
|
|
87
|
+
});
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## VersoriKVAdapter - Low-Level KV Operations
|
|
93
|
+
|
|
94
|
+
The SDK provides `VersoriKVAdapter` which wraps the native Versori KV with additional features.
|
|
95
|
+
|
|
96
|
+
### Creating Adapter
|
|
97
|
+
|
|
98
|
+
```typescript
|
|
99
|
+
import { fn } from '@versori/run';
|
|
100
|
+
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
101
|
+
|
|
102
|
+
export const useAdapter = fn('use-adapter', async ctx => {
|
|
103
|
+
const kv = ctx.openKv(':project:');
|
|
104
|
+
const adapter = new VersoriKVAdapter(kv);
|
|
105
|
+
|
|
106
|
+
// Now use adapter methods...
|
|
107
|
+
});
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
### VersoriKVAdapter API Reference
|
|
111
|
+
|
|
112
|
+
**IMPORTANT**: VersoriKVAdapter uses **array-based keys** `['key']` to match the SDK's `KVStore` interface.
|
|
113
|
+
|
|
114
|
+
```typescript
|
|
115
|
+
interface VersoriKVAdapter implements KVStore {
|
|
116
|
+
// Basic operations (use array keys: ['key'])
|
|
117
|
+
get(keys: string[]): Promise<{ value: unknown } | null>;
|
|
118
|
+
set(keys: string[], value: unknown): Promise<void>;
|
|
119
|
+
delete(keys: string[]): Promise<void>;
|
|
120
|
+
|
|
121
|
+
// Atomic operations (simplified - Versori doesn't support true atomicity)
|
|
122
|
+
atomic(): {
|
|
123
|
+
check(entries: Array<{ key: string[]; versionstamp: string | null }>): void;
|
|
124
|
+
set(key: string[], value: unknown): void;
|
|
125
|
+
commit(): Promise<boolean>; // Returns true if all operations succeed
|
|
126
|
+
};
|
|
127
|
+
}
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
**Key Limitations:**
|
|
131
|
+
- ❌ **No versionstamp support** - Versori KV doesn't expose version stamps
|
|
132
|
+
- ❌ **No expireIn/TTL option** in VersoriKVAdapter - use native Versori KV for TTL
|
|
133
|
+
- ❌ **No list() method** - use native Versori KV with prefix patterns
|
|
134
|
+
- ❌ **No atomic guarantees** - `atomic()` executes operations sequentially, not atomically
|
|
135
|
+
- ❌ **No increment()** - use read-modify-write pattern
|
|
136
|
+
|
|
137
|
+
**When to Use VersoriKVAdapter:**
|
|
138
|
+
- ✅ You need SDK `KVStore` interface compatibility (e.g., for `StateService`)
|
|
139
|
+
- ✅ You're using array-based key patterns `['namespace', 'key']`
|
|
140
|
+
- ✅ Simple get/set/delete operations are sufficient
|
|
141
|
+
|
|
142
|
+
**When to Use Native Versori KV:**
|
|
143
|
+
- ✅ You need TTL/expiration support
|
|
144
|
+
- ✅ You need to list keys with prefixes
|
|
145
|
+
- ✅ You want string-based keys `'simple:key:name'`
|
|
146
|
+
- ✅ You need increment operations
|
|
147
|
+
- ✅ Better performance (no adapter overhead)
|
|
148
|
+
|
|
149
|
+
### Practical Examples
|
|
150
|
+
|
|
151
|
+
#### Example 1: Caching Expensive API Calls
|
|
152
|
+
|
|
153
|
+
**Note**: For caching with TTL, use **native Versori KV** directly. VersoriKVAdapter doesn't support TTL.
|
|
154
|
+
|
|
155
|
+
```typescript
|
|
156
|
+
import { http } from '@versori/run';
|
|
157
|
+
import { createClient } from '@fluentcommerce/fc-connect-sdk';
|
|
158
|
+
|
|
159
|
+
export const cachedQuery = http(
|
|
160
|
+
'cached-query',
|
|
161
|
+
{
|
|
162
|
+
connection: 'fluent_commerce',
|
|
163
|
+
},
|
|
164
|
+
async ctx => {
|
|
165
|
+
const kv = ctx.openKv(':project:');
|
|
166
|
+
|
|
167
|
+
const cacheKey = 'cache:products:list';
|
|
168
|
+
const cacheTTL = 3600000; // 1 hour
|
|
169
|
+
|
|
170
|
+
// Try cache first (using native Versori KV)
|
|
171
|
+
let products = await kv.get(cacheKey);
|
|
172
|
+
|
|
173
|
+
if (products) {
|
|
174
|
+
ctx.log('info', 'Cache hit', { key: cacheKey });
|
|
175
|
+
return { source: 'cache', data: products };
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
// Cache miss - fetch from API
|
|
179
|
+
ctx.log('info', 'Cache miss - fetching from API');
|
|
180
|
+
const client = await createClient(ctx);
|
|
181
|
+
|
|
182
|
+
const result = await client.graphql({
|
|
183
|
+
query: `
|
|
184
|
+
query GetProducts($first: Int!) {
|
|
185
|
+
products(first: $first) {
|
|
186
|
+
edges {
|
|
187
|
+
node {
|
|
188
|
+
id
|
|
189
|
+
ref
|
|
190
|
+
name
|
|
191
|
+
status
|
|
192
|
+
}
|
|
193
|
+
}
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
`,
|
|
197
|
+
variables: { first: 100 },
|
|
198
|
+
});
|
|
199
|
+
|
|
200
|
+
products = result.data.products.edges.map(e => e.node);
|
|
201
|
+
|
|
202
|
+
// Store in cache with TTL (using native Versori KV)
|
|
203
|
+
await kv.set(cacheKey, products, { ttl: cacheTTL });
|
|
204
|
+
|
|
205
|
+
return { source: 'api', data: products };
|
|
206
|
+
}
|
|
207
|
+
);
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
#### Example 2: Rate Limiting with Native Versori KV
|
|
211
|
+
|
|
212
|
+
**Note**: VersoriKVAdapter doesn't have `increment()`, `expire()`, or `ttl()` methods. Use **native Versori KV** for rate limiting. StateService is for distributed locking, not rate limiting.
|
|
213
|
+
|
|
214
|
+
```typescript
|
|
215
|
+
import { fn } from '@versori/run';
|
|
216
|
+
|
|
217
|
+
export const rateLimited = fn('rate-limited', async ctx => {
|
|
218
|
+
const kv = ctx.openKv(':project:');
|
|
219
|
+
|
|
220
|
+
const userId = ctx.data?.userId || 'anonymous';
|
|
221
|
+
const rateLimitKey = `ratelimit:${userId}`;
|
|
222
|
+
const maxRequests = 100;
|
|
223
|
+
const windowMs = 3600000; // 1 hour
|
|
224
|
+
|
|
225
|
+
// Get current count using native Versori KV
|
|
226
|
+
const stored = await kv.get(rateLimitKey);
|
|
227
|
+
const data = stored as { count: number; resetAt: number } | undefined;
|
|
228
|
+
|
|
229
|
+
const now = Date.now();
|
|
230
|
+
let count = 0;
|
|
231
|
+
let resetAt = now + windowMs;
|
|
232
|
+
|
|
233
|
+
if (data && data.resetAt > now) {
|
|
234
|
+
// Within rate limit window
|
|
235
|
+
count = data.count + 1;
|
|
236
|
+
resetAt = data.resetAt;
|
|
237
|
+
} else {
|
|
238
|
+
// New window
|
|
239
|
+
count = 1;
|
|
240
|
+
resetAt = now + windowMs;
|
|
241
|
+
}
|
|
242
|
+
|
|
243
|
+
// Update count using native Versori KV
|
|
244
|
+
await kv.set(rateLimitKey, { count, resetAt });
|
|
245
|
+
|
|
246
|
+
// Check if rate limited
|
|
247
|
+
if (count > maxRequests) {
|
|
248
|
+
const retryAfter = Math.ceil((resetAt - now) / 1000);
|
|
249
|
+
return {
|
|
250
|
+
rateLimited: true,
|
|
251
|
+
retryAfter,
|
|
252
|
+
message: `Rate limit exceeded. Try again in ${retryAfter}s`,
|
|
253
|
+
};
|
|
254
|
+
}
|
|
255
|
+
|
|
256
|
+
// Process request
|
|
257
|
+
return {
|
|
258
|
+
rateLimited: false,
|
|
259
|
+
requestsRemaining: maxRequests - count,
|
|
260
|
+
};
|
|
261
|
+
});
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
#### Example 3: Using VersoriKVAdapter for Workflow State
|
|
265
|
+
|
|
266
|
+
**Note**: VersoriKVAdapter doesn't have `mset()` or `mget()` methods. For batch operations, use native Versori KV or loop through individual operations.
|
|
267
|
+
|
|
268
|
+
```typescript
|
|
269
|
+
import { fn } from '@versori/run';
|
|
270
|
+
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
271
|
+
|
|
272
|
+
export const workflowState = fn('workflow-state', async ctx => {
|
|
273
|
+
const kv = ctx.openKv(':project:');
|
|
274
|
+
const adapter = new VersoriKVAdapter(kv);
|
|
275
|
+
|
|
276
|
+
// Set workflow states (individual operations with array-based keys)
|
|
277
|
+
await adapter.set(['state', 'workflow1'], { status: 'running', startedAt: Date.now() });
|
|
278
|
+
await adapter.set(['state', 'workflow2'], { status: 'pending', queuedAt: Date.now() });
|
|
279
|
+
await adapter.set(['state', 'workflow3'], { status: 'completed', finishedAt: Date.now() });
|
|
280
|
+
|
|
281
|
+
// Get workflow states (individual operations)
|
|
282
|
+
const workflow1 = await adapter.get(['state', 'workflow1']);
|
|
283
|
+
const workflow2 = await adapter.get(['state', 'workflow2']);
|
|
284
|
+
const workflow3 = await adapter.get(['state', 'workflow3']);
|
|
285
|
+
|
|
286
|
+
const states = [workflow1?.value, workflow2?.value, workflow3?.value];
|
|
287
|
+
|
|
288
|
+
ctx.log('info', 'Workflow states retrieved', {
|
|
289
|
+
retrieved: states.filter(v => v !== null && v !== undefined).length,
|
|
290
|
+
});
|
|
291
|
+
|
|
292
|
+
return { states };
|
|
293
|
+
});
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
---
|
|
297
|
+
|
|
298
|
+
## VersoriFileTracker - Duplicate Prevention
|
|
299
|
+
|
|
300
|
+
The most common KV use case: tracking processed files to prevent duplicates.
|
|
301
|
+
|
|
302
|
+
### VersoriFileTracker API Reference
|
|
303
|
+
|
|
304
|
+
```typescript
|
|
305
|
+
interface VersoriFileTracker {
|
|
306
|
+
// Check if file was processed
|
|
307
|
+
wasFileProcessed(fileName: string): Promise<boolean>;
|
|
308
|
+
|
|
309
|
+
// Mark file as processed with metadata
|
|
310
|
+
markFileProcessed(fileName: string, metadata?: { recordCount?: number }): Promise<void>;
|
|
311
|
+
|
|
312
|
+
// Incremental sync helpers
|
|
313
|
+
setLastProcessedFile(fileName: string): Promise<void>;
|
|
314
|
+
getLastProcessedFile(): Promise<string | null>;
|
|
315
|
+
}
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
**Key Points**:
|
|
319
|
+
|
|
320
|
+
- **Array-based keys internally**: Uses `prefix:processed-files:${fileName}` pattern
|
|
321
|
+
- **Simple metadata**: Only stores `processedAt` timestamp and optional `recordCount`
|
|
322
|
+
- **No list/clear methods**: For advanced queries, use native Versori KV with prefix patterns
|
|
323
|
+
|
|
324
|
+
### Example 1: Basic File Tracking
|
|
325
|
+
|
|
326
|
+
```typescript
|
|
327
|
+
import { schedule } from '@versori/run';
|
|
328
|
+
import { createClient, VersoriFileTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
329
|
+
|
|
330
|
+
/**
|
|
331
|
+
* Daily inventory sync with duplicate prevention
|
|
332
|
+
*
|
|
333
|
+
* Cron: 0 2 * * * (daily at 2 AM)
|
|
334
|
+
*/
|
|
335
|
+
export const dailySyncWithTracking = schedule(
|
|
336
|
+
'daily-sync',
|
|
337
|
+
'0 2 * * *',
|
|
338
|
+
{
|
|
339
|
+
connection: 'fluent_commerce',
|
|
340
|
+
},
|
|
341
|
+
async ctx => {
|
|
342
|
+
const kv = ctx.openKv(':project:');
|
|
343
|
+
const tracker = new VersoriFileTracker(kv, 'daily-inventory-sync');
|
|
344
|
+
|
|
345
|
+
// Generate file name based on date
|
|
346
|
+
const today = new Date().toISOString().split('T')[0];
|
|
347
|
+
const fileName = `inventory-${today}.csv`;
|
|
348
|
+
|
|
349
|
+
// Check if already processed
|
|
350
|
+
if (await tracker.wasFileProcessed(fileName)) {
|
|
351
|
+
ctx.log('info', 'File already processed today', { fileName });
|
|
352
|
+
return {
|
|
353
|
+
skipped: true,
|
|
354
|
+
reason: 'Already processed',
|
|
355
|
+
fileName,
|
|
356
|
+
};
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
// Process file
|
|
360
|
+
ctx.log('info', 'Processing new file', { fileName });
|
|
361
|
+
const records = await fetchAndParseInventoryFile(fileName);
|
|
362
|
+
|
|
363
|
+
// Send to Fluent
|
|
364
|
+
const client = await createClient(ctx);
|
|
365
|
+
const job = await client.createJob({ name: `Daily Sync ${fileName}` });
|
|
366
|
+
await client.sendBatch(job.id, {
|
|
367
|
+
action: 'UPSERT',
|
|
368
|
+
entityType: 'INVENTORY',
|
|
369
|
+
entities: records,
|
|
370
|
+
});
|
|
371
|
+
|
|
372
|
+
// Mark as processed with metadata
|
|
373
|
+
await tracker.markFileProcessed(fileName, {
|
|
374
|
+
recordCount: records.length,
|
|
375
|
+
jobId: job.id,
|
|
376
|
+
timestamp: Date.now(),
|
|
377
|
+
});
|
|
378
|
+
|
|
379
|
+
ctx.log('info', 'File processed successfully', {
|
|
380
|
+
fileName,
|
|
381
|
+
recordCount: records.length,
|
|
382
|
+
jobId: job.id,
|
|
383
|
+
});
|
|
384
|
+
|
|
385
|
+
return {
|
|
386
|
+
success: true,
|
|
387
|
+
fileName,
|
|
388
|
+
recordCount: records.length,
|
|
389
|
+
jobId: job.id,
|
|
390
|
+
};
|
|
391
|
+
}
|
|
392
|
+
);
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
### Example 2: Incremental Processing
|
|
396
|
+
|
|
397
|
+
```typescript
|
|
398
|
+
export const incrementalSync = schedule(
|
|
399
|
+
'incremental-sync',
|
|
400
|
+
'0 * * * *',
|
|
401
|
+
{
|
|
402
|
+
connection: 'fluent_commerce',
|
|
403
|
+
},
|
|
404
|
+
async ctx => {
|
|
405
|
+
const kv = ctx.openKv(':project:');
|
|
406
|
+
const tracker = new VersoriFileTracker(kv, 'incremental-sync');
|
|
407
|
+
|
|
408
|
+
// Get last processed file
|
|
409
|
+
const lastFile = await tracker.getLastProcessedFile();
|
|
410
|
+
ctx.log('info', 'Last processed file', { lastFile });
|
|
411
|
+
|
|
412
|
+
// List new files since last run
|
|
413
|
+
const newFiles = await listFilesAfter(lastFile);
|
|
414
|
+
|
|
415
|
+
if (newFiles.length === 0) {
|
|
416
|
+
ctx.log('info', 'No new files to process');
|
|
417
|
+
return { skipped: true, reason: 'No new files' };
|
|
418
|
+
}
|
|
419
|
+
|
|
420
|
+
const client = await createClient(ctx);
|
|
421
|
+
let totalProcessed = 0;
|
|
422
|
+
|
|
423
|
+
for (const file of newFiles) {
|
|
424
|
+
// Skip if already processed (safety check)
|
|
425
|
+
if (await tracker.wasFileProcessed(file.name)) {
|
|
426
|
+
ctx.log('warn', 'File already processed (safety skip)', { file: file.name });
|
|
427
|
+
continue;
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
// Process file
|
|
431
|
+
const records = await processFile(file);
|
|
432
|
+
|
|
433
|
+
// Send to Fluent
|
|
434
|
+
const job = await client.createJob({ name: `Incremental ${file.name}` });
|
|
435
|
+
await client.sendBatch(job.id, {
|
|
436
|
+
action: 'UPSERT',
|
|
437
|
+
entityType: 'INVENTORY',
|
|
438
|
+
entities: records,
|
|
439
|
+
});
|
|
440
|
+
|
|
441
|
+
// Mark as processed
|
|
442
|
+
await tracker.markFileProcessed(file.name, {
|
|
443
|
+
recordCount: records.length,
|
|
444
|
+
jobId: job.id,
|
|
445
|
+
});
|
|
446
|
+
|
|
447
|
+
// Update last processed file
|
|
448
|
+
await tracker.setLastProcessedFile(file.name);
|
|
449
|
+
|
|
450
|
+
totalProcessed += records.length;
|
|
451
|
+
|
|
452
|
+
ctx.log('info', 'File processed', {
|
|
453
|
+
fileName: file.name,
|
|
454
|
+
recordCount: records.length,
|
|
455
|
+
});
|
|
456
|
+
}
|
|
457
|
+
|
|
458
|
+
return {
|
|
459
|
+
success: true,
|
|
460
|
+
filesProcessed: newFiles.length,
|
|
461
|
+
recordsProcessed: totalProcessed,
|
|
462
|
+
};
|
|
463
|
+
}
|
|
464
|
+
);
|
|
465
|
+
```
|
|
466
|
+
|
|
467
|
+
### Example 3: Checking Last Processed File
|
|
468
|
+
|
|
469
|
+
**Note**: VersoriFileTracker doesn't have `listProcessedFiles()` or `getFileProcessingInfo()` methods. Use `getLastProcessedFile()` to track incremental sync state.
|
|
470
|
+
|
|
471
|
+
```typescript
|
|
472
|
+
import { fn } from '@versori/run';
|
|
473
|
+
import { VersoriFileTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
474
|
+
|
|
475
|
+
export const checkProcessingState = fn('check-state', async ctx => {
|
|
476
|
+
const kv = ctx.openKv(':project:');
|
|
477
|
+
const tracker = new VersoriFileTracker(kv, 'daily-inventory-sync');
|
|
478
|
+
|
|
479
|
+
// Get last processed file
|
|
480
|
+
const lastFile = await tracker.getLastProcessedFile();
|
|
481
|
+
|
|
482
|
+
// Check if specific files were processed
|
|
483
|
+
const today = new Date().toISOString().split('T')[0];
|
|
484
|
+
const todayFile = `inventory-${today}.csv`;
|
|
485
|
+
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0];
|
|
486
|
+
const yesterdayFile = `inventory-${yesterday}.csv`;
|
|
487
|
+
|
|
488
|
+
const todayProcessed = await tracker.wasFileProcessed(todayFile);
|
|
489
|
+
const yesterdayProcessed = await tracker.wasFileProcessed(yesterdayFile);
|
|
490
|
+
|
|
491
|
+
ctx.log('info', 'Processing state', {
|
|
492
|
+
lastFile,
|
|
493
|
+
todayProcessed,
|
|
494
|
+
yesterdayProcessed,
|
|
495
|
+
});
|
|
496
|
+
|
|
497
|
+
return {
|
|
498
|
+
lastFile,
|
|
499
|
+
todayFile: {
|
|
500
|
+
name: todayFile,
|
|
501
|
+
processed: todayProcessed,
|
|
502
|
+
},
|
|
503
|
+
yesterdayFile: {
|
|
504
|
+
name: yesterdayFile,
|
|
505
|
+
processed: yesterdayProcessed,
|
|
506
|
+
},
|
|
507
|
+
};
|
|
508
|
+
});
|
|
509
|
+
```
|
|
510
|
+
|
|
511
|
+
**For viewing all processed files**, use native Versori KV with prefix listing:
|
|
512
|
+
|
|
513
|
+
```typescript
|
|
514
|
+
export const listAllProcessedFiles = fn('list-processed', async ctx => {
|
|
515
|
+
const kv = ctx.openKv(':project:');
|
|
516
|
+
const prefix = 'fc-sdk:processed-files:';
|
|
517
|
+
|
|
518
|
+
// List all keys with prefix (native Versori KV)
|
|
519
|
+
const keys = await kv.list({ prefix });
|
|
520
|
+
|
|
521
|
+
const files = [];
|
|
522
|
+
for (const key of keys) {
|
|
523
|
+
const fileName = key.replace(prefix, '');
|
|
524
|
+
const metadata = await kv.get(key);
|
|
525
|
+
files.push({
|
|
526
|
+
fileName,
|
|
527
|
+
processedAt: metadata?.processedAt,
|
|
528
|
+
recordCount: metadata?.recordCount,
|
|
529
|
+
});
|
|
530
|
+
}
|
|
531
|
+
|
|
532
|
+
// Sort by date (newest first)
|
|
533
|
+
files.sort(
|
|
534
|
+
(a, b) => new Date(b.processedAt || 0).getTime() - new Date(a.processedAt || 0).getTime()
|
|
535
|
+
);
|
|
536
|
+
|
|
537
|
+
return { totalFiles: files.length, files };
|
|
538
|
+
});
|
|
539
|
+
```
|
|
540
|
+
|
|
541
|
+
---
|
|
542
|
+
|
|
543
|
+
## Distributed Locking
|
|
544
|
+
|
|
545
|
+
Prevent concurrent execution of critical sections using distributed locks.
|
|
546
|
+
|
|
547
|
+
### StateService with Locking
|
|
548
|
+
|
|
549
|
+
```typescript
|
|
550
|
+
import { schedule } from '@versori/run';
|
|
551
|
+
import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
552
|
+
|
|
553
|
+
/**
|
|
554
|
+
* Scheduled workflow with distributed locking
|
|
555
|
+
*
|
|
556
|
+
* Ensures only ONE instance runs at a time
|
|
557
|
+
*/
|
|
558
|
+
export const criticalSection = schedule(
|
|
559
|
+
'critical',
|
|
560
|
+
'*/5 * * * *',
|
|
561
|
+
{
|
|
562
|
+
connection: 'fluent_commerce',
|
|
563
|
+
},
|
|
564
|
+
async ctx => {
|
|
565
|
+
const kv = ctx.openKv(':project:');
|
|
566
|
+
const kvAdapter = new VersoriKVAdapter(kv);
|
|
567
|
+
const stateService = new StateService(logger);
|
|
568
|
+
|
|
569
|
+
const lockName = 'critical-section';
|
|
570
|
+
const timeoutMinutes = 1; // 1 minute timeout
|
|
571
|
+
|
|
572
|
+
// Try to acquire lock (returns boolean)
|
|
573
|
+
const acquired = await stateService.acquireLock(lockName, kvAdapter, timeoutMinutes);
|
|
574
|
+
|
|
575
|
+
if (!acquired) {
|
|
576
|
+
ctx.log('warn', 'Lock acquisition failed - another instance is running', {
|
|
577
|
+
lockName,
|
|
578
|
+
});
|
|
579
|
+
return {
|
|
580
|
+
skipped: true,
|
|
581
|
+
reason: 'Another instance is running',
|
|
582
|
+
};
|
|
583
|
+
}
|
|
584
|
+
|
|
585
|
+
ctx.log('info', 'Lock acquired, starting critical section', {
|
|
586
|
+
lockName,
|
|
587
|
+
});
|
|
588
|
+
|
|
589
|
+
try {
|
|
590
|
+
// Critical section - only ONE workflow instance executes this
|
|
591
|
+
await performCriticalOperation(ctx);
|
|
592
|
+
|
|
593
|
+
ctx.log('info', 'Critical section completed successfully');
|
|
594
|
+
|
|
595
|
+
return { success: true };
|
|
596
|
+
} finally {
|
|
597
|
+
// Always release lock
|
|
598
|
+
await stateService.releaseLock(lockName, kvAdapter);
|
|
599
|
+
ctx.log('info', 'Lock released', { lockName });
|
|
600
|
+
}
|
|
601
|
+
}
|
|
602
|
+
);
|
|
603
|
+
```
|
|
604
|
+
|
|
605
|
+
### Advanced Locking Pattern
|
|
606
|
+
|
|
607
|
+
```typescript
|
|
608
|
+
export const advancedLocking = schedule(
|
|
609
|
+
'advanced-locking',
|
|
610
|
+
'*/10 * * * *',
|
|
611
|
+
{
|
|
612
|
+
connection: 'fluent_commerce',
|
|
613
|
+
},
|
|
614
|
+
async ctx => {
|
|
615
|
+
const kv = ctx.openKv(':project:');
|
|
616
|
+
const kvAdapter = new VersoriKVAdapter(kv);
|
|
617
|
+
const stateService = new StateService(logger);
|
|
618
|
+
|
|
619
|
+
const lockName = 'inventory-sync';
|
|
620
|
+
const timeoutMinutes = 5; // 5 minutes timeout
|
|
621
|
+
const maxRetries = 3;
|
|
622
|
+
const retryDelay = 5000; // 5 seconds
|
|
623
|
+
|
|
624
|
+
let acquired = false;
|
|
625
|
+
let attempts = 0;
|
|
626
|
+
|
|
627
|
+
// Retry lock acquisition
|
|
628
|
+
while (attempts < maxRetries) {
|
|
629
|
+
acquired = await stateService.acquireLock(lockName, kvAdapter, timeoutMinutes);
|
|
630
|
+
|
|
631
|
+
if (acquired) {
|
|
632
|
+
break;
|
|
633
|
+
}
|
|
634
|
+
|
|
635
|
+
attempts++;
|
|
636
|
+
ctx.log('info', `Lock acquisition failed, retrying... (${attempts}/${maxRetries})`);
|
|
637
|
+
await new Promise(resolve => setTimeout(resolve, retryDelay));
|
|
638
|
+
}
|
|
639
|
+
|
|
640
|
+
if (!acquired) {
|
|
641
|
+
ctx.log('error', 'Failed to acquire lock after retries', {
|
|
642
|
+
attempts,
|
|
643
|
+
});
|
|
644
|
+
return {
|
|
645
|
+
success: false,
|
|
646
|
+
error: 'Failed to acquire lock',
|
|
647
|
+
attempts,
|
|
648
|
+
};
|
|
649
|
+
}
|
|
650
|
+
|
|
651
|
+
try {
|
|
652
|
+
// Long-running operation
|
|
653
|
+
const result = await performLongOperation(ctx);
|
|
654
|
+
|
|
655
|
+
// Note: StateService doesn't have extendLock() - acquire a new lock if needed
|
|
656
|
+
// If operation needs more time, acquire a new lock before releasing
|
|
657
|
+
if (result.needsMoreTime) {
|
|
658
|
+
// Release current lock and acquire new one
|
|
659
|
+
await stateService.releaseLock(lockName, kvAdapter);
|
|
660
|
+
const reacquired = await stateService.acquireLock(lockName, kvAdapter, timeoutMinutes);
|
|
661
|
+
if (reacquired) {
|
|
662
|
+
ctx.log('info', 'Lock reacquired for extended operation');
|
|
663
|
+
}
|
|
664
|
+
}
|
|
665
|
+
|
|
666
|
+
return { success: true, result };
|
|
667
|
+
} catch (error) {
|
|
668
|
+
ctx.log('error', 'Operation failed', {
|
|
669
|
+
error: error instanceof Error ? error.message : String(error),
|
|
670
|
+
});
|
|
671
|
+
throw error;
|
|
672
|
+
} finally {
|
|
673
|
+
await stateService.releaseLock(lockName, kvAdapter);
|
|
674
|
+
ctx.log('info', 'Lock released');
|
|
675
|
+
}
|
|
676
|
+
}
|
|
677
|
+
);
|
|
678
|
+
```
|
|
679
|
+
|
|
680
|
+
---
|
|
681
|
+
|
|
682
|
+
## KV Namespace Patterns
|
|
683
|
+
|
|
684
|
+
### Scoping Best Practices
|
|
685
|
+
|
|
686
|
+
| Namespace | Use Case | Scope | Example Key |
|
|
687
|
+
| ---------------- | --------------------------- | ------------------- | ---------------------------- |
|
|
688
|
+
| `:project:` | Global config, shared state | All workflows | `config:retailerId` |
|
|
689
|
+
| `:workflow:name` | Workflow-specific state | Single workflow | `state:lastRun` |
|
|
690
|
+
| `cache:*` | Cached data | Project-wide | `cache:products:list` |
|
|
691
|
+
| `lock:*` | Distributed locks | Project-wide | `lock:inventory-sync` |
|
|
692
|
+
| `files:*` | File tracking | Workflow or project | `files:processed:2025-01-15` |
|
|
693
|
+
| `ratelimit:*` | Rate limiting | Per user/IP | `ratelimit:user:123` |
|
|
694
|
+
|
|
695
|
+
### Namespace Examples
|
|
696
|
+
|
|
697
|
+
```typescript
|
|
698
|
+
import { fn } from '@versori/run';
|
|
699
|
+
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
700
|
+
|
|
701
|
+
export const namespaceExamples = fn('namespace-examples', async ctx => {
|
|
702
|
+
// 1. Project-level configuration (native Versori KV)
|
|
703
|
+
const projectKV = ctx.openKv(':project:');
|
|
704
|
+
await projectKV.set('config:retailerId', '1');
|
|
705
|
+
await projectKV.set('config:environment', 'production');
|
|
706
|
+
|
|
707
|
+
// 2. Workflow-specific state (native Versori KV)
|
|
708
|
+
const workflowKV = ctx.openKv(':workflow:inventory-sync');
|
|
709
|
+
await workflowKV.set('state:lastRun', new Date().toISOString());
|
|
710
|
+
await workflowKV.set('state:lastCursor', 'cursor-value');
|
|
711
|
+
|
|
712
|
+
// 3. Custom namespace for caching (native Versori KV with TTL)
|
|
713
|
+
const cacheKV = ctx.openKv('cache:inventory');
|
|
714
|
+
await cacheKV.set('products', [], { ttl: 3600000 });
|
|
715
|
+
|
|
716
|
+
// 4. Custom namespace with VersoriKVAdapter (requires array-based keys)
|
|
717
|
+
const filesKV = ctx.openKv('files:daily-sync');
|
|
718
|
+
const adapter = new VersoriKVAdapter(filesKV);
|
|
719
|
+
// VersoriKVAdapter ALWAYS uses array syntax: ['processed', '2025-01-15']
|
|
720
|
+
await adapter.set(['processed', '2025-01-15'], { processedAt: Date.now() });
|
|
721
|
+
|
|
722
|
+
return { namespacesConfigured: 4 };
|
|
723
|
+
});
|
|
724
|
+
```
|
|
725
|
+
|
|
726
|
+
---
|
|
727
|
+
|
|
728
|
+
## Performance Optimization
|
|
729
|
+
|
|
730
|
+
### 1. Minimize Individual KV Operations
|
|
731
|
+
|
|
732
|
+
**Note**: VersoriKVAdapter doesn't have batch methods. Use native Versori KV for individual operations.
|
|
733
|
+
|
|
734
|
+
```typescript
|
|
735
|
+
import { fn } from '@versori/run';
|
|
736
|
+
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
737
|
+
|
|
738
|
+
// ❌ Inefficient - N separate KV operations
|
|
739
|
+
export const inefficientWrites = fn('inefficient', async ctx => {
|
|
740
|
+
const kv = ctx.openKv(':project:');
|
|
741
|
+
const adapter = new VersoriKVAdapter(kv);
|
|
742
|
+
|
|
743
|
+
for (const item of items) {
|
|
744
|
+
await adapter.set(['item', item.id], item); // N operations
|
|
745
|
+
}
|
|
746
|
+
});
|
|
747
|
+
|
|
748
|
+
// ✅ Better - Use native Versori KV for multiple writes
|
|
749
|
+
export const efficientWrites = fn('efficient', async ctx => {
|
|
750
|
+
const kv = ctx.openKv(':project:');
|
|
751
|
+
|
|
752
|
+
// Native KV operations are faster
|
|
753
|
+
for (const item of items) {
|
|
754
|
+
await kv.set(`item:${item.id}`, item);
|
|
755
|
+
}
|
|
756
|
+
});
|
|
757
|
+
|
|
758
|
+
// ✅ Best - Store as single aggregate if possible
|
|
759
|
+
export const aggregateWrite = fn('aggregate', async ctx => {
|
|
760
|
+
const kv = ctx.openKv(':project:');
|
|
761
|
+
|
|
762
|
+
// Store all items in one key (if items are related)
|
|
763
|
+
await kv.set('items:batch', items); // 1 operation
|
|
764
|
+
});
|
|
765
|
+
```
|
|
766
|
+
|
|
767
|
+
### 2. Set Appropriate TTLs
|
|
768
|
+
|
|
769
|
+
```typescript
|
|
770
|
+
// Cache duration guidelines:
|
|
771
|
+
const TTL = {
|
|
772
|
+
VERY_SHORT: 60000, // 1 minute - volatile data
|
|
773
|
+
SHORT: 300000, // 5 minutes - frequently changing
|
|
774
|
+
MEDIUM: 1800000, // 30 minutes - semi-static
|
|
775
|
+
LONG: 3600000, // 1 hour - stable data
|
|
776
|
+
VERY_LONG: 86400000, // 24 hours - rarely changing
|
|
777
|
+
PERMANENT: undefined, // No TTL - manual cleanup
|
|
778
|
+
};
|
|
779
|
+
|
|
780
|
+
// Use based on data characteristics
|
|
781
|
+
await adapter.set(['cache', 'products'], products, TTL.MEDIUM);
|
|
782
|
+
await adapter.set(['cache', 'config'], config, TTL.VERY_LONG);
|
|
783
|
+
await adapter.set(['session', 'user', '123'], session, TTL.SHORT);
|
|
784
|
+
```
|
|
785
|
+
|
|
786
|
+
### 3. Minimize KV Reads/Writes
|
|
787
|
+
|
|
788
|
+
**Note**: VersoriKVAdapter doesn't have `increment()` method. Use read-modify-write pattern.
|
|
789
|
+
|
|
790
|
+
```typescript
|
|
791
|
+
import { fn } from '@versori/run';
|
|
792
|
+
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
793
|
+
|
|
794
|
+
// ❌ Inefficient - read/write in loop
|
|
795
|
+
export const inefficientCounter = fn('inefficient', async ctx => {
|
|
796
|
+
const kv = ctx.openKv(':project:');
|
|
797
|
+
|
|
798
|
+
for (let i = 0; i < 100; i++) {
|
|
799
|
+
const count = (await kv.get('counter')) || 0;
|
|
800
|
+
await kv.set('counter', count + 1); // 100 read + 100 write operations
|
|
801
|
+
}
|
|
802
|
+
});
|
|
803
|
+
|
|
804
|
+
// ✅ Better - Single read-modify-write
|
|
805
|
+
export const efficientCounter = fn('efficient', async ctx => {
|
|
806
|
+
const kv = ctx.openKv(':project:');
|
|
807
|
+
const adapter = new VersoriKVAdapter(kv);
|
|
808
|
+
|
|
809
|
+
// Get current count
|
|
810
|
+
const stored = await adapter.get(['counter']);
|
|
811
|
+
const currentCount = (stored?.value as number) || 0;
|
|
812
|
+
|
|
813
|
+
// Update once
|
|
814
|
+
await adapter.set(['counter'], currentCount + 100); // 1 read + 1 write
|
|
815
|
+
|
|
816
|
+
return { count: currentCount + 100 };
|
|
817
|
+
});
|
|
818
|
+
|
|
819
|
+
// ✅ Best - Use atomic operations if needed
|
|
820
|
+
export const atomicCounter = fn('atomic', async ctx => {
|
|
821
|
+
const kv = ctx.openKv(':project:');
|
|
822
|
+
const adapter = new VersoriKVAdapter(kv);
|
|
823
|
+
|
|
824
|
+
const atomic = adapter.atomic();
|
|
825
|
+
atomic.set(['counter'], 100);
|
|
826
|
+
const success = await atomic.commit();
|
|
827
|
+
|
|
828
|
+
return { success };
|
|
829
|
+
});
|
|
830
|
+
```
|
|
831
|
+
|
|
832
|
+
### 4. Use Prefix-Based Cleanup
|
|
833
|
+
|
|
834
|
+
**Note**: VersoriKVAdapter doesn't have `clear()` or `keys()` methods. Use **native Versori KV** for prefix-based operations.
|
|
835
|
+
|
|
836
|
+
```typescript
|
|
837
|
+
import { fn } from '@versori/run';
|
|
838
|
+
import { VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
|
|
839
|
+
|
|
840
|
+
export const cleanupCache = fn('cleanup-cache', async ctx => {
|
|
841
|
+
const kv = ctx.openKv(':project:');
|
|
842
|
+
|
|
843
|
+
// List all cache keys (native Versori KV)
|
|
844
|
+
const cacheKeys = await kv.list({ prefix: 'cache:' });
|
|
845
|
+
|
|
846
|
+
// Delete each cache entry
|
|
847
|
+
for (const key of cacheKeys) {
|
|
848
|
+
await kv.delete(key);
|
|
849
|
+
}
|
|
850
|
+
|
|
851
|
+
// Clear old sessions (older than 1 day)
|
|
852
|
+
const sessionKeys = await kv.list({ prefix: 'session:' });
|
|
853
|
+
const now = Date.now();
|
|
854
|
+
let deletedCount = 0;
|
|
855
|
+
|
|
856
|
+
for (const key of sessionKeys) {
|
|
857
|
+
const session = await kv.get(key);
|
|
858
|
+
if (session && now - session.createdAt > 86400000) {
|
|
859
|
+
await kv.delete(key);
|
|
860
|
+
deletedCount++;
|
|
861
|
+
}
|
|
862
|
+
}
|
|
863
|
+
|
|
864
|
+
ctx.log('info', 'Cleanup complete', {
|
|
865
|
+
cacheKeysDeleted: cacheKeys.length,
|
|
866
|
+
sessionsChecked: sessionKeys.length,
|
|
867
|
+
oldSessionsDeleted: deletedCount,
|
|
868
|
+
});
|
|
869
|
+
|
|
870
|
+
return {
|
|
871
|
+
cacheKeysDeleted: cacheKeys.length,
|
|
872
|
+
oldSessionsDeleted: deletedCount,
|
|
873
|
+
};
|
|
874
|
+
});
|
|
875
|
+
```
|
|
876
|
+
|
|
877
|
+
---
|
|
878
|
+
|
|
879
|
+
## Troubleshooting KV Storage
|
|
880
|
+
|
|
881
|
+
### Common Issues & Solutions
|
|
882
|
+
|
|
883
|
+
| Issue | Symptoms | Solution |
|
|
884
|
+
| --------------------------- | ------------------------------- | ---------------------------------------------------- |
|
|
885
|
+
| **Data not persisting** | Values disappear after workflow | Check TTL settings, ensure no accidental `clear()` |
|
|
886
|
+
| **Namespace conflicts** | Unexpected data in keys | Use unique namespaces per workflow/feature |
|
|
887
|
+
| **Performance degradation** | Slow KV operations | Use batch operations, optimize key structure |
|
|
888
|
+
| **Lock deadlocks** | Workflows stuck waiting | Ensure locks are always released in `finally` blocks |
|
|
889
|
+
| **Memory issues** | Large values causing errors | Split large objects, use pagination |
|
|
890
|
+
|
|
891
|
+
### Debug KV State
|
|
892
|
+
|
|
893
|
+
**Note**: Use **native Versori KV** for listing and debugging operations.
|
|
894
|
+
|
|
895
|
+
```typescript
|
|
896
|
+
import { fn } from '@versori/run';
|
|
897
|
+
|
|
898
|
+
export const debugKV = fn('debug-kv', async ctx => {
|
|
899
|
+
const kv = ctx.openKv(':project:');
|
|
900
|
+
|
|
901
|
+
// List all keys (native Versori KV)
|
|
902
|
+
const allKeys = await kv.list();
|
|
903
|
+
|
|
904
|
+
// Group by prefix
|
|
905
|
+
const grouped: Record<string, string[]> = {};
|
|
906
|
+
for (const key of allKeys) {
|
|
907
|
+
const prefix = key.split(':')[0];
|
|
908
|
+
if (!grouped[prefix]) {
|
|
909
|
+
grouped[prefix] = [];
|
|
910
|
+
}
|
|
911
|
+
grouped[prefix].push(key);
|
|
912
|
+
}
|
|
913
|
+
|
|
914
|
+
// Sample some values
|
|
915
|
+
const samples: Record<string, unknown> = {};
|
|
916
|
+
for (const [prefix, keys] of Object.entries(grouped)) {
|
|
917
|
+
const sampleKey = keys[0];
|
|
918
|
+
samples[sampleKey] = await kv.get(sampleKey);
|
|
919
|
+
}
|
|
920
|
+
|
|
921
|
+
ctx.log('info', 'KV Debug Info', {
|
|
922
|
+
totalKeys: allKeys.length,
|
|
923
|
+
namespaces: Object.keys(grouped),
|
|
924
|
+
keysByNamespace: Object.fromEntries(Object.entries(grouped).map(([k, v]) => [k, v.length])),
|
|
925
|
+
samples,
|
|
926
|
+
});
|
|
927
|
+
|
|
928
|
+
return {
|
|
929
|
+
totalKeys: allKeys.length,
|
|
930
|
+
namespaces: grouped,
|
|
931
|
+
keysByNamespace: Object.fromEntries(Object.entries(grouped).map(([k, v]) => [k, v.length])),
|
|
932
|
+
samples,
|
|
933
|
+
};
|
|
934
|
+
});
|
|
935
|
+
```
|
|
936
|
+
|
|
937
|
+
---
|
|
938
|
+
|
|
939
|
+
## Key Takeaways
|
|
940
|
+
|
|
941
|
+
- 🎯 **VersoriKVAdapter** provides array-based key interface matching SDK's KVStore (use `['key']` syntax)
|
|
942
|
+
- 🎯 **Native Versori KV** recommended for TTL, increment, list, and prefix operations
|
|
943
|
+
- 🎯 **VersoriFileTracker** prevents duplicate file processing with simple metadata (`wasFileProcessed`, `markFileProcessed`)
|
|
944
|
+
- 🎯 **Distributed locking** via StateService ensures concurrent workflow safety
|
|
945
|
+
- 🎯 **Namespace scoping** (`:project:`, `:workflow:`, custom) provides isolation
|
|
946
|
+
- 🎯 **TTL optimization** reduces storage costs - use native Versori KV for TTL support
|
|
947
|
+
- 🎯 **Minimize KV operations** - aggregate data when possible, avoid loops
|
|
948
|
+
- 🎯 **Always release locks** in `finally` blocks to prevent deadlocks
|
|
949
|
+
|
|
950
|
+
---
|
|
951
|
+
|
|
952
|
+
## Practice Exercise
|
|
953
|
+
|
|
954
|
+
Create a workflow that:
|
|
955
|
+
|
|
956
|
+
1. Uses VersoriFileTracker to prevent duplicate processing
|
|
957
|
+
2. Implements distributed locking for critical section
|
|
958
|
+
3. Caches expensive API results with 30-minute TTL
|
|
959
|
+
4. Tracks processing statistics using atomic counters
|
|
960
|
+
5. Provides debug endpoint to view KV state
|
|
961
|
+
|
|
962
|
+
**Hints**:
|
|
963
|
+
|
|
964
|
+
- Combine VersoriFileTracker + StateService + VersoriKVAdapter
|
|
965
|
+
- Use appropriate TTLs for cache vs state
|
|
966
|
+
- Implement cleanup for old cache entries
|
|
967
|
+
- Use atomic increment for statistics
|
|
968
|
+
|
|
969
|
+
**Solution** available in [Module 8: Best Practices](./platforms-versori-08-best-practices.md#practice-solutions)
|
|
970
|
+
|
|
971
|
+
---
|
|
972
|
+
|
|
973
|
+
## Next Steps
|
|
974
|
+
|
|
975
|
+
Now that you've mastered KV storage, let's explore deployment patterns and monitoring.
|
|
976
|
+
|
|
977
|
+
Continue to [Module 7: Deployment →](./platforms-versori-07-deployment.md) to learn about deploying connectors, CI/CD, monitoring, and production operations.
|
|
978
|
+
|
|
979
|
+
---
|
|
980
|
+
|
|
981
|
+
## Related Documentation
|
|
982
|
+
|
|
983
|
+
- [Module 4: Workflows](./platforms-versori-04-workflows.md) - Using KV in workflows
|
|
984
|
+
- [Module 7: Deployment](./platforms-versori-07-deployment.md) - Production deployment
|
|
985
|
+
- [Module 8: Best Practices](./platforms-versori-08-best-practices.md) - Production patterns
|
|
986
|
+
- [API Reference](../../../../02-CORE-GUIDES/api-reference/api-reference-readme.md) - Complete SDK API
|
|
987
|
+
|
|
988
|
+
---
|
|
989
|
+
|
|
990
|
+
[← Previous: Module 5](./platforms-versori-05-connections.md) | [Back to Guide](../platforms-versori-readme.md) | [Next: Module 7: Deployment →](./platforms-versori-07-deployment.md)
|