@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/README.md +11 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
- package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
|
@@ -1,1097 +1,1097 @@
|
|
|
1
|
-
# Module 07: Parsers
|
|
2
|
-
|
|
3
|
-
**Level:** Beginner to Intermediate
|
|
4
|
-
**Category:** Data Processing
|
|
5
|
-
|
|
6
|
-
## Overview
|
|
7
|
-
|
|
8
|
-
Parsers convert raw data formats (CSV, JSON, XML, Parquet) into JavaScript objects. The SDK provides robust parsers with streaming support, validation, and format-specific optimizations for processing data from various sources.
|
|
9
|
-
|
|
10
|
-
## Table of Contents
|
|
11
|
-
|
|
12
|
-
- [CSV Parser](#csv-parser)
|
|
13
|
-
- [JSON Parser Service](#json-parser-service)
|
|
14
|
-
- [parse Method](#parse)
|
|
15
|
-
- [parseStreaming Method](#parsestreaming)
|
|
16
|
-
- [parseMultiple Method](#parsemultiple)
|
|
17
|
-
- [validate Method](#validate)
|
|
18
|
-
- [query Method](#query)
|
|
19
|
-
- [stringify Method](#stringify)
|
|
20
|
-
- [XML Parser Service](#xml-parser-service)
|
|
21
|
-
- [parse Method](#parse-1)
|
|
22
|
-
- [parseSync Method](#parsesync)
|
|
23
|
-
- [parseStreaming Method](#parsestreaming-1)
|
|
24
|
-
- [parseWithSelector Method](#parsewithselector)
|
|
25
|
-
- [validate Method](#validate-1)
|
|
26
|
-
- [build Method](#build)
|
|
27
|
-
- [XMLBuilder](#xmlbuilder)
|
|
28
|
-
- [Understanding @ and #text Pattern](#understanding-the-and-text-pattern)
|
|
29
|
-
- [Constructor](#constructor)
|
|
30
|
-
- [build Method](#build-method)
|
|
31
|
-
- [Pattern Examples](#pattern-examples)
|
|
32
|
-
- [Parquet Parser Service](#parquet-parser-service)
|
|
33
|
-
- [parse Method](#parse-2)
|
|
34
|
-
- [parseStreaming Method](#parsestreaming-2)
|
|
35
|
-
- [validateFile Method](#validatefile)
|
|
36
|
-
- [getPerformanceMetrics Method](#getperformancemetrics)
|
|
37
|
-
- [resetMetrics Method](#resetmetrics)
|
|
38
|
-
- [Parser Comparison](#parser-comparison)
|
|
39
|
-
- [See Also](#see-also)
|
|
40
|
-
|
|
41
|
-
---
|
|
42
|
-
|
|
43
|
-
## CSV Parser
|
|
44
|
-
|
|
45
|
-
Parse CSV files with eager and streaming APIs. Defaults: `columns=true`, `skip_empty_lines=true`, `trim=true`.
|
|
46
|
-
|
|
47
|
-
```typescript
|
|
48
|
-
import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
49
|
-
|
|
50
|
-
const parser = new CSVParserService();
|
|
51
|
-
|
|
52
|
-
// Eager parse from string
|
|
53
|
-
const records = await parser.parse(csvString);
|
|
54
|
-
|
|
55
|
-
// Streaming from string (per record)
|
|
56
|
-
for await (const rec of parser.parseStreaming(csvString)) {
|
|
57
|
-
// rec is a single record
|
|
58
|
-
}
|
|
59
|
-
|
|
60
|
-
// Streaming from string (batches)
|
|
61
|
-
for await (const batch of parser.parseStreaming(csvString, {}, 100)) {
|
|
62
|
-
// batch is an array of up to 100 records
|
|
63
|
-
}
|
|
64
|
-
|
|
65
|
-
// Eager parse from Node stream
|
|
66
|
-
const stream = fs.createReadStream('file.csv');
|
|
67
|
-
const fromStream = await parser.parseStream(stream);
|
|
68
|
-
|
|
69
|
-
// Streaming from Node stream
|
|
70
|
-
for await (const rec of parser.parseStreamStreaming(stream)) {
|
|
71
|
-
// consume records one by one
|
|
72
|
-
}
|
|
73
|
-
|
|
74
|
-
// Parse with custom headers
|
|
75
|
-
const withHeaders = await parser.parseWithHeaders(csvString, ['sku', 'qty', 'location']);
|
|
76
|
-
|
|
77
|
-
// Parse and transform
|
|
78
|
-
const transformed = await parser.parseAndTransform(csvString, r => ({ ...r, qty: Number(r.qty) }));
|
|
79
|
-
|
|
80
|
-
// Stringify back to CSV
|
|
81
|
-
const csvOut = parser.stringify(transformed, { headers: true, delimiter: ',' });
|
|
82
|
-
|
|
83
|
-
// Validate structure
|
|
84
|
-
const validation = await parser.validate(csvString, {
|
|
85
|
-
requiredColumns: ['sku', 'qty'],
|
|
86
|
-
columnTypes: { qty: 'number' },
|
|
87
|
-
minRows: 1,
|
|
88
|
-
});
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
Key options (internal defaults via csv-parse):
|
|
92
|
-
|
|
93
|
-
- `columns: true` (previously shown as headers)
|
|
94
|
-
- `skip_empty_lines: true` (previously shown as skipEmptyLines)
|
|
95
|
-
- `trim: true`
|
|
96
|
-
|
|
97
|
-
## JSON Parser Service
|
|
98
|
-
|
|
99
|
-
Parse JSON, JSONL, and stream large JSON files with validation support.
|
|
100
|
-
|
|
101
|
-
```typescript
|
|
102
|
-
import { JSONParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
103
|
-
|
|
104
|
-
const parser = new JSONParserService();
|
|
105
|
-
|
|
106
|
-
// Parse standard JSON
|
|
107
|
-
const data = await parser.parse(jsonString);
|
|
108
|
-
|
|
109
|
-
// Parse with type safety
|
|
110
|
-
interface Product {
|
|
111
|
-
sku: string;
|
|
112
|
-
name: string;
|
|
113
|
-
price: number;
|
|
114
|
-
}
|
|
115
|
-
const products = await parser.parse<Product[]>(jsonString);
|
|
116
|
-
|
|
117
|
-
// Parse with options
|
|
118
|
-
const parsed = await parser.parse(jsonString, {
|
|
119
|
-
parseDates: true,
|
|
120
|
-
dataPath: 'response.data.items',
|
|
121
|
-
validateSchema: true,
|
|
122
|
-
schema: productSchema,
|
|
123
|
-
});
|
|
124
|
-
|
|
125
|
-
// Stream JSONL (JSON Lines)
|
|
126
|
-
for await (const item of parser.parseStreaming(jsonlContent, {
|
|
127
|
-
jsonLines: true,
|
|
128
|
-
batchSize: 100,
|
|
129
|
-
})) {
|
|
130
|
-
console.log('Processing batch:', item);
|
|
131
|
-
}
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
**Methods:**
|
|
135
|
-
|
|
136
|
-
### parse
|
|
137
|
-
|
|
138
|
-
Parse JSON string with optional validation and transformation.
|
|
139
|
-
|
|
140
|
-
```typescript
|
|
141
|
-
async parse<T = Record<string, unknown>>(
|
|
142
|
-
jsonContent: string,
|
|
143
|
-
options?: JSONParseOptions
|
|
144
|
-
): Promise<T>
|
|
145
|
-
```
|
|
146
|
-
|
|
147
|
-
**Example:**
|
|
148
|
-
|
|
149
|
-
```typescript
|
|
150
|
-
// Parse simple JSON
|
|
151
|
-
const data = await parser.parse('{"sku":"PROD-001","qty":100}');
|
|
152
|
-
|
|
153
|
-
// Parse with validation (enable via options)
|
|
154
|
-
const validated = await parser.parse(jsonString, {
|
|
155
|
-
validateSchema: true,
|
|
156
|
-
schema: {
|
|
157
|
-
type: 'object',
|
|
158
|
-
properties: {
|
|
159
|
-
sku: { type: 'string' },
|
|
160
|
-
qty: { type: 'number' },
|
|
161
|
-
},
|
|
162
|
-
required: ['sku', 'qty'],
|
|
163
|
-
},
|
|
164
|
-
});
|
|
165
|
-
```
|
|
166
|
-
|
|
167
|
-
### parseStreaming
|
|
168
|
-
|
|
169
|
-
Stream parse JSON or JSONL for memory efficiency.
|
|
170
|
-
|
|
171
|
-
```typescript
|
|
172
|
-
async *parseStreaming<T = Record<string, unknown>>(
|
|
173
|
-
jsonContent: string | Readable,
|
|
174
|
-
options?: JSONStreamOptions
|
|
175
|
-
): AsyncGenerator<T | T[], void, unknown>
|
|
176
|
-
```
|
|
177
|
-
|
|
178
|
-
**Example - JSONL (JSON Lines):**
|
|
179
|
-
|
|
180
|
-
```typescript
|
|
181
|
-
// Parse JSONL content using parseStreaming with jsonLines option
|
|
182
|
-
const jsonlContent = `{"sku":"PROD-001","qty":100}\n{"sku":"PROD-002","qty":200}`;
|
|
183
|
-
|
|
184
|
-
for await (const item of parser.parseStreaming(jsonlContent, {
|
|
185
|
-
jsonLines: true,
|
|
186
|
-
batchSize: 1,
|
|
187
|
-
})) {
|
|
188
|
-
console.log('Parsed item:', item);
|
|
189
|
-
}
|
|
190
|
-
```
|
|
191
|
-
|
|
192
|
-
### validate
|
|
193
|
-
|
|
194
|
-
Validate JSON structure and syntax.
|
|
195
|
-
|
|
196
|
-
```typescript
|
|
197
|
-
validate(
|
|
198
|
-
jsonContent: string,
|
|
199
|
-
options?: JSONParseOptions
|
|
200
|
-
): JSONValidationResult
|
|
201
|
-
```
|
|
202
|
-
|
|
203
|
-
### parseMultiple
|
|
204
|
-
|
|
205
|
-
Parse an array of JSON objects.
|
|
206
|
-
|
|
207
|
-
```typescript
|
|
208
|
-
async parseMultiple<T = Record<string, unknown>>(
|
|
209
|
-
jsonArray: unknown[],
|
|
210
|
-
options?: JSONParseOptions
|
|
211
|
-
): Promise<T[]>
|
|
212
|
-
```
|
|
213
|
-
|
|
214
|
-
### query
|
|
215
|
-
|
|
216
|
-
Query JSON data using JSONPath-like syntax.
|
|
217
|
-
|
|
218
|
-
```typescript
|
|
219
|
-
query(data: unknown, path: string): unknown
|
|
220
|
-
```
|
|
221
|
-
|
|
222
|
-
### stringify
|
|
223
|
-
|
|
224
|
-
Convert JavaScript object to JSON string.
|
|
225
|
-
|
|
226
|
-
```typescript
|
|
227
|
-
stringify(
|
|
228
|
-
data: unknown,
|
|
229
|
-
options?: {
|
|
230
|
-
pretty?: boolean;
|
|
231
|
-
indent?: number | string;
|
|
232
|
-
replacer?: (key: string, value: unknown) => unknown;
|
|
233
|
-
sortKeys?: boolean;
|
|
234
|
-
}
|
|
235
|
-
): string
|
|
236
|
-
```
|
|
237
|
-
|
|
238
|
-
**Options (JSONParseOptions):**
|
|
239
|
-
|
|
240
|
-
- `maxDepth?: number` - Maximum nesting depth
|
|
241
|
-
- `parseDates?: boolean` - Auto-parse date strings
|
|
242
|
-
- `dateFormats?: string[]` - Custom date formats
|
|
243
|
-
- `validateSchema?: boolean` - Enable schema validation
|
|
244
|
-
- `schema?: Record<string, unknown>` - JSON schema
|
|
245
|
-
- `dataPath?: string` - Path to extract data from
|
|
246
|
-
- `jsonLines?: boolean` - Handle JSONL format
|
|
247
|
-
- `reviver?: (key: string, value: unknown) => unknown` - Custom reviver
|
|
248
|
-
- `allowComments?: boolean` - Allow // and /\* \*/ comments
|
|
249
|
-
- `allowTrailingCommas?: boolean` - Allow trailing commas
|
|
250
|
-
|
|
251
|
-
## XML Parser Service
|
|
252
|
-
|
|
253
|
-
Parse XML files with streaming support, validation, and XPath-like selectors.
|
|
254
|
-
|
|
255
|
-
```typescript
|
|
256
|
-
import { XMLParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
257
|
-
|
|
258
|
-
const parser = new XMLParserService();
|
|
259
|
-
|
|
260
|
-
// Parse XML string
|
|
261
|
-
const data = await parser.parse(xmlString);
|
|
262
|
-
|
|
263
|
-
// Parse with options
|
|
264
|
-
const parsed = await parser.parse(xmlString, {
|
|
265
|
-
lowercaseKeys: true,
|
|
266
|
-
parseNumbers: true,
|
|
267
|
-
includeAttributes: true,
|
|
268
|
-
attributePrefix: '@',
|
|
269
|
-
});
|
|
270
|
-
|
|
271
|
-
// Stream large XML files
|
|
272
|
-
for await (const item of parser.parseStreaming(xmlContent, {
|
|
273
|
-
itemPath: '//product',
|
|
274
|
-
})) {
|
|
275
|
-
console.log('Processing product:', item);
|
|
276
|
-
}
|
|
277
|
-
|
|
278
|
-
// Use XPath-like selectors
|
|
279
|
-
const products = await parser.parseWithSelector(xmlContent, '//catalog/product');
|
|
280
|
-
|
|
281
|
-
// Convert object to XML
|
|
282
|
-
const xmlString = parser.stringify(data, {
|
|
283
|
-
rootElement: 'catalog',
|
|
284
|
-
indent: ' ',
|
|
285
|
-
declaration: true,
|
|
286
|
-
});
|
|
287
|
-
```
|
|
288
|
-
|
|
289
|
-
**Methods:**
|
|
290
|
-
|
|
291
|
-
### parse
|
|
292
|
-
|
|
293
|
-
Parse XML string to JavaScript object (async).
|
|
294
|
-
|
|
295
|
-
```typescript
|
|
296
|
-
async parse(
|
|
297
|
-
xmlContent: string,
|
|
298
|
-
options?: XMLParseOptions
|
|
299
|
-
): Promise<Record<string, unknown>>
|
|
300
|
-
```
|
|
301
|
-
|
|
302
|
-
**Example:**
|
|
303
|
-
|
|
304
|
-
```typescript
|
|
305
|
-
// Parse simple XML
|
|
306
|
-
const xmlContent = `
|
|
307
|
-
<inventory>
|
|
308
|
-
<item>
|
|
309
|
-
<sku>PROD-001</sku>
|
|
310
|
-
<qty>100</qty>
|
|
311
|
-
</item>
|
|
312
|
-
</inventory>
|
|
313
|
-
`;
|
|
314
|
-
|
|
315
|
-
const data = await parser.parse(xmlContent);
|
|
316
|
-
console.log(data.inventory.item.sku); // 'PROD-001'
|
|
317
|
-
|
|
318
|
-
// Parse with options
|
|
319
|
-
const parsed = await parser.parse(xmlContent, {
|
|
320
|
-
parseNumbers: true, // qty becomes number 100
|
|
321
|
-
lowercaseKeys: true, // SKU becomes sku
|
|
322
|
-
includeAttributes: true, // Include XML attributes
|
|
323
|
-
attributePrefix: '@', // Prefix for attributes
|
|
324
|
-
});
|
|
325
|
-
```
|
|
326
|
-
|
|
327
|
-
### parseSync
|
|
328
|
-
|
|
329
|
-
Parse XML string to JavaScript object synchronously.
|
|
330
|
-
|
|
331
|
-
```typescript
|
|
332
|
-
parseSync(
|
|
333
|
-
xmlContent: string,
|
|
334
|
-
options?: XMLParseOptions
|
|
335
|
-
): Record<string, unknown>
|
|
336
|
-
```
|
|
337
|
-
|
|
338
|
-
**Description:**
|
|
339
|
-
Identical to `parse()` but returns synchronously. Used internally by `XMLPathResolver` when auto-parsing raw XML strings. Useful when working in non-async contexts or when integrating with synchronous code paths.
|
|
340
|
-
|
|
341
|
-
**Attributes Format:**
|
|
342
|
-
|
|
343
|
-
- Returns @-prefixed attributes by default: `{ order: { "@id": "123" } }`
|
|
344
|
-
- When `includeAttributes: true`, attributes are accessible via `@` prefix
|
|
345
|
-
- XMLPathResolver automatically handles both @-prefix and $-object attribute formats
|
|
346
|
-
|
|
347
|
-
**Example:**
|
|
348
|
-
|
|
349
|
-
```typescript
|
|
350
|
-
// Basic synchronous parsing
|
|
351
|
-
const xml = '<order id="123"><customer>John</customer></order>';
|
|
352
|
-
const data = parser.parseSync(xml, { includeAttributes: true });
|
|
353
|
-
console.log(data.order['@id']); // '123'
|
|
354
|
-
console.log(data.order.customer); // 'John'
|
|
355
|
-
|
|
356
|
-
// With XMLPathResolver (handles both formats)
|
|
357
|
-
import { XMLPathResolver } from '@fluentcommerce/fc-connect-sdk';
|
|
358
|
-
const resolver = new XMLPathResolver();
|
|
359
|
-
|
|
360
|
-
// Works with raw XML (auto-parsed via parseSync)
|
|
361
|
-
const orderId = resolver.resolve(xml, 'order@id'); // '123'
|
|
362
|
-
|
|
363
|
-
// Also works with pre-parsed objects
|
|
364
|
-
const customerId = resolver.resolve(data, 'order.customer'); // 'John'
|
|
365
|
-
```
|
|
366
|
-
|
|
367
|
-
**Use Cases:**
|
|
368
|
-
|
|
369
|
-
- Internal SDK usage (XMLPathResolver auto-parsing)
|
|
370
|
-
- Synchronous workflows where async is not available
|
|
371
|
-
- Testing and debugging with immediate results
|
|
372
|
-
- Integration with older synchronous code
|
|
373
|
-
|
|
374
|
-
### parseStreaming
|
|
375
|
-
|
|
376
|
-
Stream parse XML for large files.
|
|
377
|
-
|
|
378
|
-
```typescript
|
|
379
|
-
async *parseStreaming(
|
|
380
|
-
xmlContent: string,
|
|
381
|
-
options?: XMLParseOptions & { itemPath?: string }
|
|
382
|
-
): AsyncGenerator<Record<string, unknown>, void, unknown>
|
|
383
|
-
```
|
|
384
|
-
|
|
385
|
-
### parseWithSelector
|
|
386
|
-
|
|
387
|
-
Parse specific elements using XPath-like selectors.
|
|
388
|
-
|
|
389
|
-
```typescript
|
|
390
|
-
async parseWithSelector(
|
|
391
|
-
xmlContent: string,
|
|
392
|
-
selector: string,
|
|
393
|
-
options?: XMLParseOptions
|
|
394
|
-
): Promise<unknown[]>
|
|
395
|
-
```
|
|
396
|
-
|
|
397
|
-
### validate
|
|
398
|
-
|
|
399
|
-
Validate XML structure.
|
|
400
|
-
|
|
401
|
-
```typescript
|
|
402
|
-
validate(xmlContent: string): {
|
|
403
|
-
valid: boolean;
|
|
404
|
-
errors: string[];
|
|
405
|
-
}
|
|
406
|
-
```
|
|
407
|
-
|
|
408
|
-
### build
|
|
409
|
-
|
|
410
|
-
Convert JavaScript object to XML string.
|
|
411
|
-
|
|
412
|
-
```typescript
|
|
413
|
-
build(
|
|
414
|
-
data: Record<string, unknown>,
|
|
415
|
-
options?: {
|
|
416
|
-
format?: boolean;
|
|
417
|
-
indentBy?: string;
|
|
418
|
-
declaration?: boolean;
|
|
419
|
-
rootElement?: string;
|
|
420
|
-
}
|
|
421
|
-
): string
|
|
422
|
-
```
|
|
423
|
-
|
|
424
|
-
**Example:**
|
|
425
|
-
|
|
426
|
-
```typescript
|
|
427
|
-
// Build XML from object
|
|
428
|
-
const xmlString = parser.build(
|
|
429
|
-
{
|
|
430
|
-
inventory: {
|
|
431
|
-
item: [
|
|
432
|
-
{ sku: 'PROD-001', qty: 100 },
|
|
433
|
-
{ sku: 'PROD-002', qty: 200 },
|
|
434
|
-
],
|
|
435
|
-
},
|
|
436
|
-
},
|
|
437
|
-
{
|
|
438
|
-
format: true,
|
|
439
|
-
indentBy: ' ',
|
|
440
|
-
declaration: true,
|
|
441
|
-
}
|
|
442
|
-
);
|
|
443
|
-
|
|
444
|
-
console.log(xmlString);
|
|
445
|
-
// <?xml version="1.0" encoding="UTF-8"?>
|
|
446
|
-
// <inventory>
|
|
447
|
-
// <item>
|
|
448
|
-
// <sku>PROD-001</sku>
|
|
449
|
-
// <qty>100</qty>
|
|
450
|
-
// </item>
|
|
451
|
-
// <item>
|
|
452
|
-
// <sku>PROD-002</sku>
|
|
453
|
-
// <qty>200</qty>
|
|
454
|
-
// </item>
|
|
455
|
-
// </inventory>
|
|
456
|
-
```
|
|
457
|
-
|
|
458
|
-
**Options (XMLParseOptions):**
|
|
459
|
-
|
|
460
|
-
- `lowercaseKeys?: boolean` - Convert element names to lowercase
|
|
461
|
-
- `normalizeWhitespace?: boolean` - Normalize whitespace in text
|
|
462
|
-
- `parseNumbers?: boolean` - Auto-parse numeric values
|
|
463
|
-
- `parseBooleans?: boolean` - Auto-parse boolean values
|
|
464
|
-
- `rootElement?: string` - Root element to start from
|
|
465
|
-
- `arrayElements?: string[]` - Elements to treat as arrays
|
|
466
|
-
- `includeAttributes?: boolean` - Include XML attributes
|
|
467
|
-
- `attributePrefix?: string` - Prefix for attributes (default: '@')
|
|
468
|
-
- `textKey?: string` - Key for text content (default: '#text')
|
|
469
|
-
- `maxDepth?: number` - Maximum parsing depth (default: 100)
|
|
470
|
-
|
|
471
|
-
## XMLBuilder
|
|
472
|
-
|
|
473
|
-
Generate XML from JavaScript objects with support for attributes, text content, and complex nested structures.
|
|
474
|
-
|
|
475
|
-
```typescript
|
|
476
|
-
import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
|
|
477
|
-
|
|
478
|
-
const builder = new XMLBuilder({
|
|
479
|
-
xmlDeclaration: true,
|
|
480
|
-
prettyPrint: true,
|
|
481
|
-
encoding: 'UTF-8',
|
|
482
|
-
});
|
|
483
|
-
|
|
484
|
-
// Generate XML
|
|
485
|
-
const xml = builder.build(data, 'OrderDetailResponse');
|
|
486
|
-
```
|
|
487
|
-
|
|
488
|
-
### Understanding the `@` and `#text` Pattern
|
|
489
|
-
|
|
490
|
-
XMLBuilder uses special keys to represent XML structures in JSON:
|
|
491
|
-
|
|
492
|
-
| JSON Key | XML Output | Example |
|
|
493
|
-
| ----------- | ------------- | ----------------------------------- |
|
|
494
|
-
| `@prefix` | Attribute | `"@id": "123"` → `<Order id="123">` |
|
|
495
|
-
| `#text` | Text content | `"#text": "Value"` → Element text |
|
|
496
|
-
| `fieldName` | Child element | `"Customer": {...}` → `<Customer>` |
|
|
497
|
-
|
|
498
|
-
**When to use `#text`:** Use it when an XML element has **BOTH attributes AND text content** (mixed content).
|
|
499
|
-
|
|
500
|
-
### Constructor
|
|
501
|
-
|
|
502
|
-
```typescript
|
|
503
|
-
new XMLBuilder(options?: XMLBuildOptions)
|
|
504
|
-
```
|
|
505
|
-
|
|
506
|
-
**Options:**
|
|
507
|
-
|
|
508
|
-
```typescript
|
|
509
|
-
interface XMLBuildOptions {
|
|
510
|
-
rootElement?: string; // Root element name (default: 'root')
|
|
511
|
-
xmlDeclaration?: boolean; // Add <?xml version="1.0"?> (default: true)
|
|
512
|
-
prettyPrint?: boolean; // Format with indentation (default: true)
|
|
513
|
-
indent?: string; // Indentation string (default: ' ')
|
|
514
|
-
attributePrefix?: string; // Attribute prefix (default: '@')
|
|
515
|
-
textKey?: string; // Text content key (default: '#text')
|
|
516
|
-
cdataKeys?: string[]; // Fields to wrap in CDATA
|
|
517
|
-
encoding?: string; // Character encoding (default: 'UTF-8')
|
|
518
|
-
}
|
|
519
|
-
```
|
|
520
|
-
|
|
521
|
-
### build Method
|
|
522
|
-
|
|
523
|
-
Generate XML from JavaScript object.
|
|
524
|
-
|
|
525
|
-
```typescript
|
|
526
|
-
build(
|
|
527
|
-
data: Record<string, unknown>,
|
|
528
|
-
rootElement?: string
|
|
529
|
-
): string
|
|
530
|
-
```
|
|
531
|
-
|
|
532
|
-
### Pattern Examples
|
|
533
|
-
|
|
534
|
-
#### Pattern 1: Simple Text Element
|
|
535
|
-
|
|
536
|
-
```typescript
|
|
537
|
-
const data = { ItemId: 'SKU001' };
|
|
538
|
-
builder.build(data, 'Order');
|
|
539
|
-
|
|
540
|
-
// Output:
|
|
541
|
-
// <Order>
|
|
542
|
-
// <ItemId>SKU001</ItemId>
|
|
543
|
-
// </Order>
|
|
544
|
-
```
|
|
545
|
-
|
|
546
|
-
#### Pattern 2: Attributes Only
|
|
547
|
-
|
|
548
|
-
```typescript
|
|
549
|
-
const data = {
|
|
550
|
-
Order: {
|
|
551
|
-
'@id': '12345',
|
|
552
|
-
'@status': 'confirmed',
|
|
553
|
-
},
|
|
554
|
-
};
|
|
555
|
-
builder.build(data);
|
|
556
|
-
|
|
557
|
-
// Output:
|
|
558
|
-
// <Order id="12345" status="confirmed"/>
|
|
559
|
-
```
|
|
560
|
-
|
|
561
|
-
#### Pattern 3: Attributes + Text Content (Using `#text`)
|
|
562
|
-
|
|
563
|
-
```typescript
|
|
564
|
-
const data = {
|
|
565
|
-
Color: {
|
|
566
|
-
'@id': '9101',
|
|
567
|
-
'#text': 'MULTI-COLOR',
|
|
568
|
-
},
|
|
569
|
-
};
|
|
570
|
-
builder.build(data, 'Product');
|
|
571
|
-
|
|
572
|
-
// Output:
|
|
573
|
-
// <Product>
|
|
574
|
-
// <Color id="9101">MULTI-COLOR</Color>
|
|
575
|
-
// </Product>
|
|
576
|
-
```
|
|
577
|
-
|
|
578
|
-
**Why `#text`?** This is the standard convention used by XML parsers (like `fast-xml-parser`) to represent text content in JSON. The SDK follows the same convention for consistency.
|
|
579
|
-
|
|
580
|
-
#### Pattern 4: Complex Nested Structure
|
|
581
|
-
|
|
582
|
-
```typescript
|
|
583
|
-
const data = {
|
|
584
|
-
Order: {
|
|
585
|
-
'@customerOrderId': '12345',
|
|
586
|
-
'@levelOfService': 'REGULAR',
|
|
587
|
-
Customer: {
|
|
588
|
-
'@customerId': '232',
|
|
589
|
-
Name: {
|
|
590
|
-
FirstName: 'John',
|
|
591
|
-
LastName: 'Doe',
|
|
592
|
-
},
|
|
593
|
-
EmailAddress: 'john@example.com',
|
|
594
|
-
},
|
|
595
|
-
OrderSource: {
|
|
596
|
-
'@type': 'source',
|
|
597
|
-
'#text': 'WEBSTORE',
|
|
598
|
-
},
|
|
599
|
-
OrderItems: {
|
|
600
|
-
OrderItem: [
|
|
601
|
-
{
|
|
602
|
-
'@id': 'item1',
|
|
603
|
-
ItemId: 'SKU001',
|
|
604
|
-
Quantity: 2,
|
|
605
|
-
Description: {
|
|
606
|
-
Color: {
|
|
607
|
-
'@id': '9101',
|
|
608
|
-
'#text': 'Black',
|
|
609
|
-
},
|
|
610
|
-
Size: {
|
|
611
|
-
'@id': 'M',
|
|
612
|
-
'#text': 'Medium',
|
|
613
|
-
},
|
|
614
|
-
},
|
|
615
|
-
},
|
|
616
|
-
],
|
|
617
|
-
},
|
|
618
|
-
},
|
|
619
|
-
};
|
|
620
|
-
|
|
621
|
-
const xml = builder.build(data, 'OrderDetailResponse');
|
|
622
|
-
|
|
623
|
-
// Output:
|
|
624
|
-
// <?xml version="1.0" encoding="UTF-8"?>
|
|
625
|
-
// <OrderDetailResponse>
|
|
626
|
-
// <Order customerOrderId="12345" levelOfService="REGULAR">
|
|
627
|
-
// <Customer customerId="232">
|
|
628
|
-
// <Name>
|
|
629
|
-
// <FirstName>John</FirstName>
|
|
630
|
-
// <LastName>Doe</LastName>
|
|
631
|
-
// </Name>
|
|
632
|
-
// <EmailAddress>john@example.com</EmailAddress>
|
|
633
|
-
// </Customer>
|
|
634
|
-
// <OrderSource type="source">WEBSTORE</OrderSource>
|
|
635
|
-
// <OrderItems>
|
|
636
|
-
// <OrderItem id="item1">
|
|
637
|
-
// <ItemId>SKU001</ItemId>
|
|
638
|
-
// <Quantity>2</Quantity>
|
|
639
|
-
// <Description>
|
|
640
|
-
// <Color id="9101">Black</Color>
|
|
641
|
-
// <Size id="M">Medium</Size>
|
|
642
|
-
// </Description>
|
|
643
|
-
// </OrderItem>
|
|
644
|
-
// </OrderItems>
|
|
645
|
-
// </Order>
|
|
646
|
-
// </OrderDetailResponse>
|
|
647
|
-
```
|
|
648
|
-
|
|
649
|
-
#### Pattern 5: Arrays (Repeated Elements)
|
|
650
|
-
|
|
651
|
-
```typescript
|
|
652
|
-
const data = {
|
|
653
|
-
OrderItems: {
|
|
654
|
-
OrderItem: [
|
|
655
|
-
{ ItemId: 'SKU001', Quantity: 2 },
|
|
656
|
-
{ ItemId: 'SKU002', Quantity: 1 },
|
|
657
|
-
],
|
|
658
|
-
},
|
|
659
|
-
};
|
|
660
|
-
|
|
661
|
-
// Output:
|
|
662
|
-
// <OrderItems>
|
|
663
|
-
// <OrderItem>
|
|
664
|
-
// <ItemId>SKU001</ItemId>
|
|
665
|
-
// <Quantity>2</Quantity>
|
|
666
|
-
// </OrderItem>
|
|
667
|
-
// <OrderItem>
|
|
668
|
-
// <ItemId>SKU002</ItemId>
|
|
669
|
-
// <Quantity>1</Quantity>
|
|
670
|
-
// </OrderItem>
|
|
671
|
-
// </OrderItems>
|
|
672
|
-
```
|
|
673
|
-
|
|
674
|
-
#### Decision Tree: When to Use `#text`
|
|
675
|
-
|
|
676
|
-
```
|
|
677
|
-
Does the element have attributes?
|
|
678
|
-
├─ NO → Just use field name
|
|
679
|
-
│ "ItemId": "SKU001"
|
|
680
|
-
│ Result: <ItemId>SKU001</ItemId>
|
|
681
|
-
│
|
|
682
|
-
└─ YES → Does it also have text content?
|
|
683
|
-
├─ NO → Just use @prefix
|
|
684
|
-
│ "@id": "123"
|
|
685
|
-
│ Result: <Order id="123"/>
|
|
686
|
-
│
|
|
687
|
-
└─ YES → Use BOTH @prefix AND #text
|
|
688
|
-
"@id": "9101", "#text": "MULTI-COLOR"
|
|
689
|
-
Result: <Color id="9101">MULTI-COLOR</Color>
|
|
690
|
-
```
|
|
691
|
-
|
|
692
|
-
#### Common Issues and Solutions
|
|
693
|
-
|
|
694
|
-
**Issue: Text appears as child element**
|
|
695
|
-
|
|
696
|
-
```typescript
|
|
697
|
-
// ❌ Wrong - creates <text> element
|
|
698
|
-
"Color": { "@id": "9101", "text": "MULTI-COLOR" }
|
|
699
|
-
|
|
700
|
-
// ✅ Correct - creates text content
|
|
701
|
-
"Color": { "@id": "9101", "#text": "MULTI-COLOR" }
|
|
702
|
-
```
|
|
703
|
-
|
|
704
|
-
**Issue: Attributes not showing**
|
|
705
|
-
|
|
706
|
-
```typescript
|
|
707
|
-
// ❌ Wrong - 'id' becomes child element
|
|
708
|
-
"Color": { "id": "9101", "#text": "MULTI-COLOR" }
|
|
709
|
-
|
|
710
|
-
// ✅ Correct - @ prefix makes it an attribute
|
|
711
|
-
"Color": { "@id": "9101", "#text": "MULTI-COLOR" }
|
|
712
|
-
```
|
|
713
|
-
|
|
714
|
-
**Issue: Self-closing tag when content expected**
|
|
715
|
-
|
|
716
|
-
```typescript
|
|
717
|
-
// ❌ Wrong - empty or undefined value
|
|
718
|
-
"Description": { "#text": undefined } // → <Description/>
|
|
719
|
-
|
|
720
|
-
// ✅ Correct - provide actual value
|
|
721
|
-
"Description": { "#text": product?.description || '' }
|
|
722
|
-
```
|
|
723
|
-
|
|
724
|
-
#### Real-World Example: Fluent → SFCC Order Export
|
|
725
|
-
|
|
726
|
-
```typescript
|
|
727
|
-
// Fluent Commerce order data
|
|
728
|
-
const fluentOrder = await fluentClient.graphql({
|
|
729
|
-
query: orderQuery,
|
|
730
|
-
variables: { ref: orderId },
|
|
731
|
-
});
|
|
732
|
-
|
|
733
|
-
// Map to SFCC XML structure using UniversalMapper
|
|
734
|
-
const mapper = new UniversalMapper(mappingConfig, {
|
|
735
|
-
customResolvers: extractionResolvers,
|
|
736
|
-
});
|
|
737
|
-
|
|
738
|
-
const mapped = await mapper.map(fluentOrder);
|
|
739
|
-
|
|
740
|
-
// Generate XML
|
|
741
|
-
const builder = new XMLBuilder({
|
|
742
|
-
xmlDeclaration: true,
|
|
743
|
-
prettyPrint: true,
|
|
744
|
-
encoding: 'UTF-8',
|
|
745
|
-
});
|
|
746
|
-
|
|
747
|
-
const sfccXml = builder.build(mapped.data, 'OrderDetailResponse');
|
|
748
|
-
|
|
749
|
-
// Returns valid SFCC XML ready to send
|
|
750
|
-
console.log(sfccXml);
|
|
751
|
-
```
|
|
752
|
-
|
|
753
|
-
## Parquet Parser Service
|
|
754
|
-
|
|
755
|
-
**FULLY FUNCTIONAL**: The ParquetParserService is now **FULLY IMPLEMENTED** and ready for use, powered by the high-performance `hyparquet` library. It provides complete Parquet file parsing with streaming support, validation, and detailed statistics.
|
|
756
|
-
|
|
757
|
-
**Large File Testing Support**: The SDK includes complete test files for performance validation:
|
|
758
|
-
|
|
759
|
-
- **Small**: 3KB files for unit testing
|
|
760
|
-
- **Medium**: 170KB files for integration testing
|
|
761
|
-
- **Large**: 1.7MB files for standard processing
|
|
762
|
-
- **50K Records**: 8.5MB files with 50,000 records
|
|
763
|
-
- **100K Records**: 17MB files with 100,000 records
|
|
764
|
-
- **250K Records**: 43MB files with 250,000 records
|
|
765
|
-
- **500K Records**: 85MB files with 500,000 records
|
|
766
|
-
- **1M Records**: 169MB files with 1,000,000 records
|
|
767
|
-
|
|
768
|
-
```typescript
|
|
769
|
-
import {
|
|
770
|
-
ParquetParserService,
|
|
771
|
-
createConsoleLogger,
|
|
772
|
-
toStructuredLogger
|
|
773
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
774
|
-
|
|
775
|
-
const logger = toStructuredLogger(createConsoleLogger(), {
|
|
776
|
-
logLevel: 'info'
|
|
777
|
-
});
|
|
778
|
-
const parser = new ParquetParserService(logger, {
|
|
779
|
-
batchSize: 1000,
|
|
780
|
-
maxFileSize: 100 * 1024 * 1024, // 100MB
|
|
781
|
-
enableStreaming: true,
|
|
782
|
-
validation: {
|
|
783
|
-
enabled: true,
|
|
784
|
-
requiredColumns: ['productRef', 'qty'],
|
|
785
|
-
columnTypes: {
|
|
786
|
-
productRef: 'string',
|
|
787
|
-
qty: 'int32',
|
|
788
|
-
},
|
|
789
|
-
},
|
|
790
|
-
});
|
|
791
|
-
|
|
792
|
-
// Parse Parquet file - FULLY WORKING
|
|
793
|
-
const { records, stats } = await parser.parse(arrayBuffer, 'inventory.parquet');
|
|
794
|
-
|
|
795
|
-
console.log(`Parsed ${stats.totalRecords} records`);
|
|
796
|
-
console.log(`Processing time: ${stats.processingTimeMs}ms`);
|
|
797
|
-
console.log(`Records per second: ${stats.recordsPerSecond}`);
|
|
798
|
-
|
|
799
|
-
// Parse large files (e.g., 500K records)
|
|
800
|
-
const largeBuffer = await fs.readFile('inventory-position-500k.parquet');
|
|
801
|
-
const largeArrayBuffer = largeBuffer.buffer.slice(
|
|
802
|
-
largeBuffer.byteOffset,
|
|
803
|
-
largeBuffer.byteOffset + largeBuffer.byteLength
|
|
804
|
-
);
|
|
805
|
-
|
|
806
|
-
const { records: largeRecords, stats: largeStats } = await parser.parse(
|
|
807
|
-
largeArrayBuffer,
|
|
808
|
-
'inventory-position-500k.parquet',
|
|
809
|
-
{
|
|
810
|
-
batchSize: 10000, // Larger batch size for performance
|
|
811
|
-
validation: {
|
|
812
|
-
enabled: false, // Disable validation for speed
|
|
813
|
-
},
|
|
814
|
-
}
|
|
815
|
-
);
|
|
816
|
-
|
|
817
|
-
console.log(`Processed ${largeStats.totalRecords} records in ${largeStats.processingTimeMs}ms`);
|
|
818
|
-
console.log(`Throughput: ${largeStats.recordsPerSecond} records/sec`);
|
|
819
|
-
|
|
820
|
-
// Stream massive files (1M+ records)
|
|
821
|
-
const streamResult = await parser.parseStreaming(massiveArrayBuffer, 'mixed-positions-1m.parquet');
|
|
822
|
-
|
|
823
|
-
for await (const batch of streamResult.records) {
|
|
824
|
-
console.log(`Processing batch of ${batch.length} records`);
|
|
825
|
-
}
|
|
826
|
-
```
|
|
827
|
-
|
|
828
|
-
**Constructor:**
|
|
829
|
-
|
|
830
|
-
```typescript
|
|
831
|
-
constructor(
|
|
832
|
-
logger: StructuredLogger,
|
|
833
|
-
options?: ParquetParsingOptions
|
|
834
|
-
)
|
|
835
|
-
```
|
|
836
|
-
|
|
837
|
-
**Methods:**
|
|
838
|
-
|
|
839
|
-
### parse
|
|
840
|
-
|
|
841
|
-
Parse a complete Parquet file into memory. Fully functional with hyparquet.
|
|
842
|
-
|
|
843
|
-
```typescript
|
|
844
|
-
async parse(
|
|
845
|
-
arrayBuffer: ArrayBuffer,
|
|
846
|
-
fileName: string,
|
|
847
|
-
options?: Partial<ParquetParsingOptions>
|
|
848
|
-
): Promise<{
|
|
849
|
-
records: ParquetDataRecord[];
|
|
850
|
-
stats: ParquetParsingStats;
|
|
851
|
-
}>
|
|
852
|
-
```
|
|
853
|
-
|
|
854
|
-
**Parameters:**
|
|
855
|
-
|
|
856
|
-
- `arrayBuffer: ArrayBuffer` - The Parquet file data as ArrayBuffer
|
|
857
|
-
- `fileName: string` - Name of the file being parsed (for logging)
|
|
858
|
-
- `options?: Partial<ParquetParsingOptions>` - Optional parsing configuration
|
|
859
|
-
|
|
860
|
-
**Returns:**
|
|
861
|
-
|
|
862
|
-
- `records: ParquetDataRecord[]` - Array of parsed records
|
|
863
|
-
- `stats: ParquetParsingStats` - Detailed parsing statistics
|
|
864
|
-
|
|
865
|
-
**Example:**
|
|
866
|
-
|
|
867
|
-
```typescript
|
|
868
|
-
const buffer = await fs.readFile('data.parquet');
|
|
869
|
-
const arrayBuffer = buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength);
|
|
870
|
-
|
|
871
|
-
const { records, stats } = await parser.parse(arrayBuffer, 'data.parquet', {
|
|
872
|
-
batchSize: 500,
|
|
873
|
-
validation: {
|
|
874
|
-
enabled: true,
|
|
875
|
-
requiredColumns: ['id', 'name', 'qty'],
|
|
876
|
-
},
|
|
877
|
-
});
|
|
878
|
-
|
|
879
|
-
// Check validation results
|
|
880
|
-
if (stats.validation && !stats.validation.isValid) {
|
|
881
|
-
console.error('Validation errors:', stats.validation.errors);
|
|
882
|
-
}
|
|
883
|
-
```
|
|
884
|
-
|
|
885
|
-
### parseStreaming
|
|
886
|
-
|
|
887
|
-
Parse Parquet file with streaming support for memory-efficient processing of large files.
|
|
888
|
-
|
|
889
|
-
```typescript
|
|
890
|
-
async parseStreaming(
|
|
891
|
-
arrayBuffer: ArrayBuffer,
|
|
892
|
-
fileName: string,
|
|
893
|
-
options?: Partial<ParquetParsingOptions>
|
|
894
|
-
): Promise<ParquetStreamingResult>
|
|
895
|
-
```
|
|
896
|
-
|
|
897
|
-
**Parameters:**
|
|
898
|
-
|
|
899
|
-
- `arrayBuffer: ArrayBuffer` - The Parquet file data
|
|
900
|
-
- `fileName: string` - Name of the file being parsed
|
|
901
|
-
- `options?: Partial<ParquetParsingOptions>` - Optional configuration
|
|
902
|
-
|
|
903
|
-
**Returns:**
|
|
904
|
-
|
|
905
|
-
- `records: AsyncIterable<ParquetDataRecord[]>` - Async iterator of record batches
|
|
906
|
-
- `stats: Promise<ParquetParsingStats>` - Promise resolving to final statistics
|
|
907
|
-
- `cancel: () => void` - Function to cancel streaming
|
|
908
|
-
|
|
909
|
-
**Example:**
|
|
910
|
-
|
|
911
|
-
```typescript
|
|
912
|
-
const streamResult = await parser.parseStreaming(arrayBuffer, 'large_dataset.parquet', {
|
|
913
|
-
batchSize: 1000,
|
|
914
|
-
});
|
|
915
|
-
|
|
916
|
-
// Process batches as they stream
|
|
917
|
-
for await (const batch of streamResult.records) {
|
|
918
|
-
for (const record of batch) {
|
|
919
|
-
await processRecord(record);
|
|
920
|
-
}
|
|
921
|
-
}
|
|
922
|
-
|
|
923
|
-
// Get final statistics
|
|
924
|
-
const stats = await streamResult.stats;
|
|
925
|
-
console.log(`Total records: ${stats.totalRecords}`);
|
|
926
|
-
|
|
927
|
-
// Cancel if needed
|
|
928
|
-
setTimeout(() => streamResult.cancel(), 5000);
|
|
929
|
-
```
|
|
930
|
-
|
|
931
|
-
### validateFile
|
|
932
|
-
|
|
933
|
-
Validate a Parquet file before parsing.
|
|
934
|
-
|
|
935
|
-
```typescript
|
|
936
|
-
async validateFile(
|
|
937
|
-
arrayBuffer: ArrayBuffer,
|
|
938
|
-
fileName: string
|
|
939
|
-
): Promise<ValidationResult>
|
|
940
|
-
```
|
|
941
|
-
|
|
942
|
-
**Parameters:**
|
|
943
|
-
|
|
944
|
-
- `arrayBuffer: ArrayBuffer` - The Parquet file data
|
|
945
|
-
- `fileName: string` - Name of the file
|
|
946
|
-
|
|
947
|
-
**Returns:**
|
|
948
|
-
|
|
949
|
-
- `ValidationResult` - Validation results with errors and warnings
|
|
950
|
-
|
|
951
|
-
**Example:**
|
|
952
|
-
|
|
953
|
-
```typescript
|
|
954
|
-
const validation = await parser.validateFile(arrayBuffer, 'data.parquet');
|
|
955
|
-
|
|
956
|
-
if (!validation.isValid) {
|
|
957
|
-
console.error('File validation failed:');
|
|
958
|
-
validation.errors.forEach(error => {
|
|
959
|
-
console.error(`- ${error.field}: ${error.message}`);
|
|
960
|
-
});
|
|
961
|
-
}
|
|
962
|
-
```
|
|
963
|
-
|
|
964
|
-
### getPerformanceMetrics
|
|
965
|
-
|
|
966
|
-
Get current performance metrics for the parser.
|
|
967
|
-
|
|
968
|
-
```typescript
|
|
969
|
-
getPerformanceMetrics(): ParquetPerformanceMetrics
|
|
970
|
-
```
|
|
971
|
-
|
|
972
|
-
**Returns:**
|
|
973
|
-
|
|
974
|
-
- `totalRequests: number` - Total parsing requests
|
|
975
|
-
- `successfulRequests: number` - Successful parses
|
|
976
|
-
- `failedRequests: number` - Failed parses
|
|
977
|
-
- `averageResponseTime: number` - Average parsing time (ms)
|
|
978
|
-
- `lastRequestTime: Date` - Timestamp of last request
|
|
979
|
-
|
|
980
|
-
**Example:**
|
|
981
|
-
|
|
982
|
-
```typescript
|
|
983
|
-
const metrics = parser.getPerformanceMetrics();
|
|
984
|
-
console.log(
|
|
985
|
-
`Success rate: ${((metrics.successfulRequests / metrics.totalRequests) * 100).toFixed(2)}%`
|
|
986
|
-
);
|
|
987
|
-
console.log(`Average time: ${metrics.averageResponseTime.toFixed(2)}ms`);
|
|
988
|
-
```
|
|
989
|
-
|
|
990
|
-
### resetMetrics
|
|
991
|
-
|
|
992
|
-
Reset performance metrics to initial state.
|
|
993
|
-
|
|
994
|
-
```typescript
|
|
995
|
-
resetMetrics(): void
|
|
996
|
-
```
|
|
997
|
-
|
|
998
|
-
**Example:**
|
|
999
|
-
|
|
1000
|
-
```typescript
|
|
1001
|
-
// Reset metrics after batch processing
|
|
1002
|
-
parser.resetMetrics();
|
|
1003
|
-
console.log('Performance metrics reset');
|
|
1004
|
-
```
|
|
1005
|
-
|
|
1006
|
-
**Interfaces:**
|
|
1007
|
-
|
|
1008
|
-
#### ParquetParsingOptions
|
|
1009
|
-
|
|
1010
|
-
```typescript
|
|
1011
|
-
interface ParquetParsingOptions {
|
|
1012
|
-
batchSize?: number; // Records per batch (default: 1000)
|
|
1013
|
-
maxFileSize?: number; // Max file size in bytes (default: 100MB)
|
|
1014
|
-
enableStreaming?: boolean; // Enable streaming mode (default: true)
|
|
1015
|
-
validation?: ParquetValidationOptions;
|
|
1016
|
-
}
|
|
1017
|
-
```
|
|
1018
|
-
|
|
1019
|
-
#### ParquetValidationOptions
|
|
1020
|
-
|
|
1021
|
-
```typescript
|
|
1022
|
-
interface ParquetValidationOptions {
|
|
1023
|
-
enabled?: boolean;
|
|
1024
|
-
requiredColumns?: string[];
|
|
1025
|
-
columnTypes?: Record<string, ParquetDataType>;
|
|
1026
|
-
allowAdditionalColumns?: boolean;
|
|
1027
|
-
}
|
|
1028
|
-
```
|
|
1029
|
-
|
|
1030
|
-
#### ParquetParsingStats
|
|
1031
|
-
|
|
1032
|
-
```typescript
|
|
1033
|
-
interface ParquetParsingStats {
|
|
1034
|
-
totalRecords: number;
|
|
1035
|
-
processingTimeMs: number;
|
|
1036
|
-
fileSizeBytes: number;
|
|
1037
|
-
recordsPerSecond: number;
|
|
1038
|
-
peakMemoryUsage: number;
|
|
1039
|
-
batchesProcessed: number;
|
|
1040
|
-
compressionRatio?: number;
|
|
1041
|
-
columnStats: Record<string, ParquetColumnStatistics>;
|
|
1042
|
-
validation?: ValidationResult;
|
|
1043
|
-
}
|
|
1044
|
-
```
|
|
1045
|
-
|
|
1046
|
-
#### ParquetStreamingResult
|
|
1047
|
-
|
|
1048
|
-
```typescript
|
|
1049
|
-
interface ParquetStreamingResult {
|
|
1050
|
-
records: AsyncIterable<ParquetDataRecord[]>;
|
|
1051
|
-
stats: Promise<ParquetParsingStats>;
|
|
1052
|
-
cancel: () => void;
|
|
1053
|
-
}
|
|
1054
|
-
```
|
|
1055
|
-
|
|
1056
|
-
#### ParquetDataType
|
|
1057
|
-
|
|
1058
|
-
```typescript
|
|
1059
|
-
type ParquetDataType =
|
|
1060
|
-
| 'string'
|
|
1061
|
-
| 'number'
|
|
1062
|
-
| 'boolean'
|
|
1063
|
-
| 'date'
|
|
1064
|
-
| 'int32'
|
|
1065
|
-
| 'int64'
|
|
1066
|
-
| 'float'
|
|
1067
|
-
| 'double'
|
|
1068
|
-
| 'binary';
|
|
1069
|
-
```
|
|
1070
|
-
|
|
1071
|
-
## Parser Comparison
|
|
1072
|
-
|
|
1073
|
-
| Parser | Streaming | Validation | Best For |
|
|
1074
|
-
| ------------------------ | --------- | ---------- | --------------------------------- |
|
|
1075
|
-
| **CSVParserService** | ✅ | ✅ | Text-based tabular data |
|
|
1076
|
-
| **JSONParserService** | ✅ | ✅ | JSON/JSONL formats, API responses |
|
|
1077
|
-
| **XMLParserService** | ✅ | ✅ | XML documents, SOAP APIs, SFCC |
|
|
1078
|
-
| **ParquetParserService** | ✅ | ✅ | Columnar data, large datasets |
|
|
1079
|
-
|
|
1080
|
-
**Choosing a Parser:**
|
|
1081
|
-
|
|
1082
|
-
- **CSV**: Simple imports, spreadsheets, EDI and B2B systems
|
|
1083
|
-
- **JSON**: Modern APIs, flexible schemas, nested data
|
|
1084
|
-
- **XML**: Enterprise systems, SOAP, SFCC orders
|
|
1085
|
-
- **Parquet**: Big data, analytics, data lakes
|
|
1086
|
-
|
|
1087
|
-
## See Also
|
|
1088
|
-
|
|
1089
|
-
- [Module 06: Data Sources](./api-reference-06-data-sources.md) - File operations
|
|
1090
|
-
- [Module 05: Services](./api-reference-05-services.md#universal-mapper) - Field transformations
|
|
1091
|
-
- [Module 10: Utilities](./api-reference-10-utilities.md) - Helper functions
|
|
1092
|
-
- [Parquet Guide](../../../03-PATTERN-GUIDES/parquet/parquet-readme.md) - Detailed Parquet examples
|
|
1093
|
-
- [XML Parser Guide](../../parsers/modules/02-core-guides-parsers-04-xml-parser.md) - XML patterns
|
|
1094
|
-
|
|
1095
|
-
---
|
|
1096
|
-
|
|
1097
|
-
**[← Previous: Module 06 - Data Sources](./api-reference-06-data-sources.md)** | **[API Reference Home](../api-reference-readme.md)** | **[Next: Module 08 - Types →](./api-reference-08-types.md)**
|
|
1
|
+
# Module 07: Parsers
|
|
2
|
+
|
|
3
|
+
**Level:** Beginner to Intermediate
|
|
4
|
+
**Category:** Data Processing
|
|
5
|
+
|
|
6
|
+
## Overview
|
|
7
|
+
|
|
8
|
+
Parsers convert raw data formats (CSV, JSON, XML, Parquet) into JavaScript objects. The SDK provides robust parsers with streaming support, validation, and format-specific optimizations for processing data from various sources.
|
|
9
|
+
|
|
10
|
+
## Table of Contents
|
|
11
|
+
|
|
12
|
+
- [CSV Parser](#csv-parser)
|
|
13
|
+
- [JSON Parser Service](#json-parser-service)
|
|
14
|
+
- [parse Method](#parse)
|
|
15
|
+
- [parseStreaming Method](#parsestreaming)
|
|
16
|
+
- [parseMultiple Method](#parsemultiple)
|
|
17
|
+
- [validate Method](#validate)
|
|
18
|
+
- [query Method](#query)
|
|
19
|
+
- [stringify Method](#stringify)
|
|
20
|
+
- [XML Parser Service](#xml-parser-service)
|
|
21
|
+
- [parse Method](#parse-1)
|
|
22
|
+
- [parseSync Method](#parsesync)
|
|
23
|
+
- [parseStreaming Method](#parsestreaming-1)
|
|
24
|
+
- [parseWithSelector Method](#parsewithselector)
|
|
25
|
+
- [validate Method](#validate-1)
|
|
26
|
+
- [build Method](#build)
|
|
27
|
+
- [XMLBuilder](#xmlbuilder)
|
|
28
|
+
- [Understanding @ and #text Pattern](#understanding-the-and-text-pattern)
|
|
29
|
+
- [Constructor](#constructor)
|
|
30
|
+
- [build Method](#build-method)
|
|
31
|
+
- [Pattern Examples](#pattern-examples)
|
|
32
|
+
- [Parquet Parser Service](#parquet-parser-service)
|
|
33
|
+
- [parse Method](#parse-2)
|
|
34
|
+
- [parseStreaming Method](#parsestreaming-2)
|
|
35
|
+
- [validateFile Method](#validatefile)
|
|
36
|
+
- [getPerformanceMetrics Method](#getperformancemetrics)
|
|
37
|
+
- [resetMetrics Method](#resetmetrics)
|
|
38
|
+
- [Parser Comparison](#parser-comparison)
|
|
39
|
+
- [See Also](#see-also)
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## CSV Parser
|
|
44
|
+
|
|
45
|
+
Parse CSV files with eager and streaming APIs. Defaults: `columns=true`, `skip_empty_lines=true`, `trim=true`.
|
|
46
|
+
|
|
47
|
+
```typescript
|
|
48
|
+
import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
49
|
+
|
|
50
|
+
const parser = new CSVParserService();
|
|
51
|
+
|
|
52
|
+
// Eager parse from string
|
|
53
|
+
const records = await parser.parse(csvString);
|
|
54
|
+
|
|
55
|
+
// Streaming from string (per record)
|
|
56
|
+
for await (const rec of parser.parseStreaming(csvString)) {
|
|
57
|
+
// rec is a single record
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
// Streaming from string (batches)
|
|
61
|
+
for await (const batch of parser.parseStreaming(csvString, {}, 100)) {
|
|
62
|
+
// batch is an array of up to 100 records
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
// Eager parse from Node stream
|
|
66
|
+
const stream = fs.createReadStream('file.csv');
|
|
67
|
+
const fromStream = await parser.parseStream(stream);
|
|
68
|
+
|
|
69
|
+
// Streaming from Node stream
|
|
70
|
+
for await (const rec of parser.parseStreamStreaming(stream)) {
|
|
71
|
+
// consume records one by one
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
// Parse with custom headers
|
|
75
|
+
const withHeaders = await parser.parseWithHeaders(csvString, ['sku', 'qty', 'location']);
|
|
76
|
+
|
|
77
|
+
// Parse and transform
|
|
78
|
+
const transformed = await parser.parseAndTransform(csvString, r => ({ ...r, qty: Number(r.qty) }));
|
|
79
|
+
|
|
80
|
+
// Stringify back to CSV
|
|
81
|
+
const csvOut = parser.stringify(transformed, { headers: true, delimiter: ',' });
|
|
82
|
+
|
|
83
|
+
// Validate structure
|
|
84
|
+
const validation = await parser.validate(csvString, {
|
|
85
|
+
requiredColumns: ['sku', 'qty'],
|
|
86
|
+
columnTypes: { qty: 'number' },
|
|
87
|
+
minRows: 1,
|
|
88
|
+
});
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Key options (internal defaults via csv-parse):
|
|
92
|
+
|
|
93
|
+
- `columns: true` (previously shown as headers)
|
|
94
|
+
- `skip_empty_lines: true` (previously shown as skipEmptyLines)
|
|
95
|
+
- `trim: true`
|
|
96
|
+
|
|
97
|
+
## JSON Parser Service
|
|
98
|
+
|
|
99
|
+
Parse JSON, JSONL, and stream large JSON files with validation support.
|
|
100
|
+
|
|
101
|
+
```typescript
|
|
102
|
+
import { JSONParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
103
|
+
|
|
104
|
+
const parser = new JSONParserService();
|
|
105
|
+
|
|
106
|
+
// Parse standard JSON
|
|
107
|
+
const data = await parser.parse(jsonString);
|
|
108
|
+
|
|
109
|
+
// Parse with type safety
|
|
110
|
+
interface Product {
|
|
111
|
+
sku: string;
|
|
112
|
+
name: string;
|
|
113
|
+
price: number;
|
|
114
|
+
}
|
|
115
|
+
const products = await parser.parse<Product[]>(jsonString);
|
|
116
|
+
|
|
117
|
+
// Parse with options
|
|
118
|
+
const parsed = await parser.parse(jsonString, {
|
|
119
|
+
parseDates: true,
|
|
120
|
+
dataPath: 'response.data.items',
|
|
121
|
+
validateSchema: true,
|
|
122
|
+
schema: productSchema,
|
|
123
|
+
});
|
|
124
|
+
|
|
125
|
+
// Stream JSONL (JSON Lines)
|
|
126
|
+
for await (const item of parser.parseStreaming(jsonlContent, {
|
|
127
|
+
jsonLines: true,
|
|
128
|
+
batchSize: 100,
|
|
129
|
+
})) {
|
|
130
|
+
console.log('Processing batch:', item);
|
|
131
|
+
}
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
**Methods:**
|
|
135
|
+
|
|
136
|
+
### parse
|
|
137
|
+
|
|
138
|
+
Parse JSON string with optional validation and transformation.
|
|
139
|
+
|
|
140
|
+
```typescript
|
|
141
|
+
async parse<T = Record<string, unknown>>(
|
|
142
|
+
jsonContent: string,
|
|
143
|
+
options?: JSONParseOptions
|
|
144
|
+
): Promise<T>
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
**Example:**
|
|
148
|
+
|
|
149
|
+
```typescript
|
|
150
|
+
// Parse simple JSON
|
|
151
|
+
const data = await parser.parse('{"sku":"PROD-001","qty":100}');
|
|
152
|
+
|
|
153
|
+
// Parse with validation (enable via options)
|
|
154
|
+
const validated = await parser.parse(jsonString, {
|
|
155
|
+
validateSchema: true,
|
|
156
|
+
schema: {
|
|
157
|
+
type: 'object',
|
|
158
|
+
properties: {
|
|
159
|
+
sku: { type: 'string' },
|
|
160
|
+
qty: { type: 'number' },
|
|
161
|
+
},
|
|
162
|
+
required: ['sku', 'qty'],
|
|
163
|
+
},
|
|
164
|
+
});
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
### parseStreaming
|
|
168
|
+
|
|
169
|
+
Stream parse JSON or JSONL for memory efficiency.
|
|
170
|
+
|
|
171
|
+
```typescript
|
|
172
|
+
async *parseStreaming<T = Record<string, unknown>>(
|
|
173
|
+
jsonContent: string | Readable,
|
|
174
|
+
options?: JSONStreamOptions
|
|
175
|
+
): AsyncGenerator<T | T[], void, unknown>
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
**Example - JSONL (JSON Lines):**
|
|
179
|
+
|
|
180
|
+
```typescript
|
|
181
|
+
// Parse JSONL content using parseStreaming with jsonLines option
|
|
182
|
+
const jsonlContent = `{"sku":"PROD-001","qty":100}\n{"sku":"PROD-002","qty":200}`;
|
|
183
|
+
|
|
184
|
+
for await (const item of parser.parseStreaming(jsonlContent, {
|
|
185
|
+
jsonLines: true,
|
|
186
|
+
batchSize: 1,
|
|
187
|
+
})) {
|
|
188
|
+
console.log('Parsed item:', item);
|
|
189
|
+
}
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
### validate
|
|
193
|
+
|
|
194
|
+
Validate JSON structure and syntax.
|
|
195
|
+
|
|
196
|
+
```typescript
|
|
197
|
+
validate(
|
|
198
|
+
jsonContent: string,
|
|
199
|
+
options?: JSONParseOptions
|
|
200
|
+
): JSONValidationResult
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
### parseMultiple
|
|
204
|
+
|
|
205
|
+
Parse an array of JSON objects.
|
|
206
|
+
|
|
207
|
+
```typescript
|
|
208
|
+
async parseMultiple<T = Record<string, unknown>>(
|
|
209
|
+
jsonArray: unknown[],
|
|
210
|
+
options?: JSONParseOptions
|
|
211
|
+
): Promise<T[]>
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
### query
|
|
215
|
+
|
|
216
|
+
Query JSON data using JSONPath-like syntax.
|
|
217
|
+
|
|
218
|
+
```typescript
|
|
219
|
+
query(data: unknown, path: string): unknown
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### stringify
|
|
223
|
+
|
|
224
|
+
Convert JavaScript object to JSON string.
|
|
225
|
+
|
|
226
|
+
```typescript
|
|
227
|
+
stringify(
|
|
228
|
+
data: unknown,
|
|
229
|
+
options?: {
|
|
230
|
+
pretty?: boolean;
|
|
231
|
+
indent?: number | string;
|
|
232
|
+
replacer?: (key: string, value: unknown) => unknown;
|
|
233
|
+
sortKeys?: boolean;
|
|
234
|
+
}
|
|
235
|
+
): string
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
**Options (JSONParseOptions):**
|
|
239
|
+
|
|
240
|
+
- `maxDepth?: number` - Maximum nesting depth
|
|
241
|
+
- `parseDates?: boolean` - Auto-parse date strings
|
|
242
|
+
- `dateFormats?: string[]` - Custom date formats
|
|
243
|
+
- `validateSchema?: boolean` - Enable schema validation
|
|
244
|
+
- `schema?: Record<string, unknown>` - JSON schema
|
|
245
|
+
- `dataPath?: string` - Path to extract data from
|
|
246
|
+
- `jsonLines?: boolean` - Handle JSONL format
|
|
247
|
+
- `reviver?: (key: string, value: unknown) => unknown` - Custom reviver
|
|
248
|
+
- `allowComments?: boolean` - Allow // and /\* \*/ comments
|
|
249
|
+
- `allowTrailingCommas?: boolean` - Allow trailing commas
|
|
250
|
+
|
|
251
|
+
## XML Parser Service
|
|
252
|
+
|
|
253
|
+
Parse XML files with streaming support, validation, and XPath-like selectors.
|
|
254
|
+
|
|
255
|
+
```typescript
|
|
256
|
+
import { XMLParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
257
|
+
|
|
258
|
+
const parser = new XMLParserService();
|
|
259
|
+
|
|
260
|
+
// Parse XML string
|
|
261
|
+
const data = await parser.parse(xmlString);
|
|
262
|
+
|
|
263
|
+
// Parse with options
|
|
264
|
+
const parsed = await parser.parse(xmlString, {
|
|
265
|
+
lowercaseKeys: true,
|
|
266
|
+
parseNumbers: true,
|
|
267
|
+
includeAttributes: true,
|
|
268
|
+
attributePrefix: '@',
|
|
269
|
+
});
|
|
270
|
+
|
|
271
|
+
// Stream large XML files
|
|
272
|
+
for await (const item of parser.parseStreaming(xmlContent, {
|
|
273
|
+
itemPath: '//product',
|
|
274
|
+
})) {
|
|
275
|
+
console.log('Processing product:', item);
|
|
276
|
+
}
|
|
277
|
+
|
|
278
|
+
// Use XPath-like selectors
|
|
279
|
+
const products = await parser.parseWithSelector(xmlContent, '//catalog/product');
|
|
280
|
+
|
|
281
|
+
// Convert object to XML
|
|
282
|
+
const xmlString = parser.stringify(data, {
|
|
283
|
+
rootElement: 'catalog',
|
|
284
|
+
indent: ' ',
|
|
285
|
+
declaration: true,
|
|
286
|
+
});
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
**Methods:**
|
|
290
|
+
|
|
291
|
+
### parse
|
|
292
|
+
|
|
293
|
+
Parse XML string to JavaScript object (async).
|
|
294
|
+
|
|
295
|
+
```typescript
|
|
296
|
+
async parse(
|
|
297
|
+
xmlContent: string,
|
|
298
|
+
options?: XMLParseOptions
|
|
299
|
+
): Promise<Record<string, unknown>>
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
**Example:**
|
|
303
|
+
|
|
304
|
+
```typescript
|
|
305
|
+
// Parse simple XML
|
|
306
|
+
const xmlContent = `
|
|
307
|
+
<inventory>
|
|
308
|
+
<item>
|
|
309
|
+
<sku>PROD-001</sku>
|
|
310
|
+
<qty>100</qty>
|
|
311
|
+
</item>
|
|
312
|
+
</inventory>
|
|
313
|
+
`;
|
|
314
|
+
|
|
315
|
+
const data = await parser.parse(xmlContent);
|
|
316
|
+
console.log(data.inventory.item.sku); // 'PROD-001'
|
|
317
|
+
|
|
318
|
+
// Parse with options
|
|
319
|
+
const parsed = await parser.parse(xmlContent, {
|
|
320
|
+
parseNumbers: true, // qty becomes number 100
|
|
321
|
+
lowercaseKeys: true, // SKU becomes sku
|
|
322
|
+
includeAttributes: true, // Include XML attributes
|
|
323
|
+
attributePrefix: '@', // Prefix for attributes
|
|
324
|
+
});
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
### parseSync
|
|
328
|
+
|
|
329
|
+
Parse XML string to JavaScript object synchronously.
|
|
330
|
+
|
|
331
|
+
```typescript
|
|
332
|
+
parseSync(
|
|
333
|
+
xmlContent: string,
|
|
334
|
+
options?: XMLParseOptions
|
|
335
|
+
): Record<string, unknown>
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
**Description:**
|
|
339
|
+
Identical to `parse()` but returns synchronously. Used internally by `XMLPathResolver` when auto-parsing raw XML strings. Useful when working in non-async contexts or when integrating with synchronous code paths.
|
|
340
|
+
|
|
341
|
+
**Attributes Format:**
|
|
342
|
+
|
|
343
|
+
- Returns @-prefixed attributes by default: `{ order: { "@id": "123" } }`
|
|
344
|
+
- When `includeAttributes: true`, attributes are accessible via `@` prefix
|
|
345
|
+
- XMLPathResolver automatically handles both @-prefix and $-object attribute formats
|
|
346
|
+
|
|
347
|
+
**Example:**
|
|
348
|
+
|
|
349
|
+
```typescript
|
|
350
|
+
// Basic synchronous parsing
|
|
351
|
+
const xml = '<order id="123"><customer>John</customer></order>';
|
|
352
|
+
const data = parser.parseSync(xml, { includeAttributes: true });
|
|
353
|
+
console.log(data.order['@id']); // '123'
|
|
354
|
+
console.log(data.order.customer); // 'John'
|
|
355
|
+
|
|
356
|
+
// With XMLPathResolver (handles both formats)
|
|
357
|
+
import { XMLPathResolver } from '@fluentcommerce/fc-connect-sdk';
|
|
358
|
+
const resolver = new XMLPathResolver();
|
|
359
|
+
|
|
360
|
+
// Works with raw XML (auto-parsed via parseSync)
|
|
361
|
+
const orderId = resolver.resolve(xml, 'order@id'); // '123'
|
|
362
|
+
|
|
363
|
+
// Also works with pre-parsed objects
|
|
364
|
+
const customerId = resolver.resolve(data, 'order.customer'); // 'John'
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
**Use Cases:**
|
|
368
|
+
|
|
369
|
+
- Internal SDK usage (XMLPathResolver auto-parsing)
|
|
370
|
+
- Synchronous workflows where async is not available
|
|
371
|
+
- Testing and debugging with immediate results
|
|
372
|
+
- Integration with older synchronous code
|
|
373
|
+
|
|
374
|
+
### parseStreaming
|
|
375
|
+
|
|
376
|
+
Stream parse XML for large files.
|
|
377
|
+
|
|
378
|
+
```typescript
|
|
379
|
+
async *parseStreaming(
|
|
380
|
+
xmlContent: string,
|
|
381
|
+
options?: XMLParseOptions & { itemPath?: string }
|
|
382
|
+
): AsyncGenerator<Record<string, unknown>, void, unknown>
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
### parseWithSelector
|
|
386
|
+
|
|
387
|
+
Parse specific elements using XPath-like selectors.
|
|
388
|
+
|
|
389
|
+
```typescript
|
|
390
|
+
async parseWithSelector(
|
|
391
|
+
xmlContent: string,
|
|
392
|
+
selector: string,
|
|
393
|
+
options?: XMLParseOptions
|
|
394
|
+
): Promise<unknown[]>
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
### validate
|
|
398
|
+
|
|
399
|
+
Validate XML structure.
|
|
400
|
+
|
|
401
|
+
```typescript
|
|
402
|
+
validate(xmlContent: string): {
|
|
403
|
+
valid: boolean;
|
|
404
|
+
errors: string[];
|
|
405
|
+
}
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
### build
|
|
409
|
+
|
|
410
|
+
Convert JavaScript object to XML string.
|
|
411
|
+
|
|
412
|
+
```typescript
|
|
413
|
+
build(
|
|
414
|
+
data: Record<string, unknown>,
|
|
415
|
+
options?: {
|
|
416
|
+
format?: boolean;
|
|
417
|
+
indentBy?: string;
|
|
418
|
+
declaration?: boolean;
|
|
419
|
+
rootElement?: string;
|
|
420
|
+
}
|
|
421
|
+
): string
|
|
422
|
+
```
|
|
423
|
+
|
|
424
|
+
**Example:**
|
|
425
|
+
|
|
426
|
+
```typescript
|
|
427
|
+
// Build XML from object
|
|
428
|
+
const xmlString = parser.build(
|
|
429
|
+
{
|
|
430
|
+
inventory: {
|
|
431
|
+
item: [
|
|
432
|
+
{ sku: 'PROD-001', qty: 100 },
|
|
433
|
+
{ sku: 'PROD-002', qty: 200 },
|
|
434
|
+
],
|
|
435
|
+
},
|
|
436
|
+
},
|
|
437
|
+
{
|
|
438
|
+
format: true,
|
|
439
|
+
indentBy: ' ',
|
|
440
|
+
declaration: true,
|
|
441
|
+
}
|
|
442
|
+
);
|
|
443
|
+
|
|
444
|
+
console.log(xmlString);
|
|
445
|
+
// <?xml version="1.0" encoding="UTF-8"?>
|
|
446
|
+
// <inventory>
|
|
447
|
+
// <item>
|
|
448
|
+
// <sku>PROD-001</sku>
|
|
449
|
+
// <qty>100</qty>
|
|
450
|
+
// </item>
|
|
451
|
+
// <item>
|
|
452
|
+
// <sku>PROD-002</sku>
|
|
453
|
+
// <qty>200</qty>
|
|
454
|
+
// </item>
|
|
455
|
+
// </inventory>
|
|
456
|
+
```
|
|
457
|
+
|
|
458
|
+
**Options (XMLParseOptions):**
|
|
459
|
+
|
|
460
|
+
- `lowercaseKeys?: boolean` - Convert element names to lowercase
|
|
461
|
+
- `normalizeWhitespace?: boolean` - Normalize whitespace in text
|
|
462
|
+
- `parseNumbers?: boolean` - Auto-parse numeric values
|
|
463
|
+
- `parseBooleans?: boolean` - Auto-parse boolean values
|
|
464
|
+
- `rootElement?: string` - Root element to start from
|
|
465
|
+
- `arrayElements?: string[]` - Elements to treat as arrays
|
|
466
|
+
- `includeAttributes?: boolean` - Include XML attributes
|
|
467
|
+
- `attributePrefix?: string` - Prefix for attributes (default: '@')
|
|
468
|
+
- `textKey?: string` - Key for text content (default: '#text')
|
|
469
|
+
- `maxDepth?: number` - Maximum parsing depth (default: 100)
|
|
470
|
+
|
|
471
|
+
## XMLBuilder
|
|
472
|
+
|
|
473
|
+
Generate XML from JavaScript objects with support for attributes, text content, and complex nested structures.
|
|
474
|
+
|
|
475
|
+
```typescript
|
|
476
|
+
import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
|
|
477
|
+
|
|
478
|
+
const builder = new XMLBuilder({
|
|
479
|
+
xmlDeclaration: true,
|
|
480
|
+
prettyPrint: true,
|
|
481
|
+
encoding: 'UTF-8',
|
|
482
|
+
});
|
|
483
|
+
|
|
484
|
+
// Generate XML
|
|
485
|
+
const xml = builder.build(data, 'OrderDetailResponse');
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
### Understanding the `@` and `#text` Pattern
|
|
489
|
+
|
|
490
|
+
XMLBuilder uses special keys to represent XML structures in JSON:
|
|
491
|
+
|
|
492
|
+
| JSON Key | XML Output | Example |
|
|
493
|
+
| ----------- | ------------- | ----------------------------------- |
|
|
494
|
+
| `@prefix` | Attribute | `"@id": "123"` → `<Order id="123">` |
|
|
495
|
+
| `#text` | Text content | `"#text": "Value"` → Element text |
|
|
496
|
+
| `fieldName` | Child element | `"Customer": {...}` → `<Customer>` |
|
|
497
|
+
|
|
498
|
+
**When to use `#text`:** Use it when an XML element has **BOTH attributes AND text content** (mixed content).
|
|
499
|
+
|
|
500
|
+
### Constructor
|
|
501
|
+
|
|
502
|
+
```typescript
|
|
503
|
+
new XMLBuilder(options?: XMLBuildOptions)
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
**Options:**
|
|
507
|
+
|
|
508
|
+
```typescript
|
|
509
|
+
interface XMLBuildOptions {
|
|
510
|
+
rootElement?: string; // Root element name (default: 'root')
|
|
511
|
+
xmlDeclaration?: boolean; // Add <?xml version="1.0"?> (default: true)
|
|
512
|
+
prettyPrint?: boolean; // Format with indentation (default: true)
|
|
513
|
+
indent?: string; // Indentation string (default: ' ')
|
|
514
|
+
attributePrefix?: string; // Attribute prefix (default: '@')
|
|
515
|
+
textKey?: string; // Text content key (default: '#text')
|
|
516
|
+
cdataKeys?: string[]; // Fields to wrap in CDATA
|
|
517
|
+
encoding?: string; // Character encoding (default: 'UTF-8')
|
|
518
|
+
}
|
|
519
|
+
```
|
|
520
|
+
|
|
521
|
+
### build Method
|
|
522
|
+
|
|
523
|
+
Generate XML from JavaScript object.
|
|
524
|
+
|
|
525
|
+
```typescript
|
|
526
|
+
build(
|
|
527
|
+
data: Record<string, unknown>,
|
|
528
|
+
rootElement?: string
|
|
529
|
+
): string
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
### Pattern Examples
|
|
533
|
+
|
|
534
|
+
#### Pattern 1: Simple Text Element
|
|
535
|
+
|
|
536
|
+
```typescript
|
|
537
|
+
const data = { ItemId: 'SKU001' };
|
|
538
|
+
builder.build(data, 'Order');
|
|
539
|
+
|
|
540
|
+
// Output:
|
|
541
|
+
// <Order>
|
|
542
|
+
// <ItemId>SKU001</ItemId>
|
|
543
|
+
// </Order>
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
#### Pattern 2: Attributes Only
|
|
547
|
+
|
|
548
|
+
```typescript
|
|
549
|
+
const data = {
|
|
550
|
+
Order: {
|
|
551
|
+
'@id': '12345',
|
|
552
|
+
'@status': 'confirmed',
|
|
553
|
+
},
|
|
554
|
+
};
|
|
555
|
+
builder.build(data);
|
|
556
|
+
|
|
557
|
+
// Output:
|
|
558
|
+
// <Order id="12345" status="confirmed"/>
|
|
559
|
+
```
|
|
560
|
+
|
|
561
|
+
#### Pattern 3: Attributes + Text Content (Using `#text`)
|
|
562
|
+
|
|
563
|
+
```typescript
|
|
564
|
+
const data = {
|
|
565
|
+
Color: {
|
|
566
|
+
'@id': '9101',
|
|
567
|
+
'#text': 'MULTI-COLOR',
|
|
568
|
+
},
|
|
569
|
+
};
|
|
570
|
+
builder.build(data, 'Product');
|
|
571
|
+
|
|
572
|
+
// Output:
|
|
573
|
+
// <Product>
|
|
574
|
+
// <Color id="9101">MULTI-COLOR</Color>
|
|
575
|
+
// </Product>
|
|
576
|
+
```
|
|
577
|
+
|
|
578
|
+
**Why `#text`?** This is the standard convention used by XML parsers (like `fast-xml-parser`) to represent text content in JSON. The SDK follows the same convention for consistency.
|
|
579
|
+
|
|
580
|
+
#### Pattern 4: Complex Nested Structure
|
|
581
|
+
|
|
582
|
+
```typescript
|
|
583
|
+
const data = {
|
|
584
|
+
Order: {
|
|
585
|
+
'@customerOrderId': '12345',
|
|
586
|
+
'@levelOfService': 'REGULAR',
|
|
587
|
+
Customer: {
|
|
588
|
+
'@customerId': '232',
|
|
589
|
+
Name: {
|
|
590
|
+
FirstName: 'John',
|
|
591
|
+
LastName: 'Doe',
|
|
592
|
+
},
|
|
593
|
+
EmailAddress: 'john@example.com',
|
|
594
|
+
},
|
|
595
|
+
OrderSource: {
|
|
596
|
+
'@type': 'source',
|
|
597
|
+
'#text': 'WEBSTORE',
|
|
598
|
+
},
|
|
599
|
+
OrderItems: {
|
|
600
|
+
OrderItem: [
|
|
601
|
+
{
|
|
602
|
+
'@id': 'item1',
|
|
603
|
+
ItemId: 'SKU001',
|
|
604
|
+
Quantity: 2,
|
|
605
|
+
Description: {
|
|
606
|
+
Color: {
|
|
607
|
+
'@id': '9101',
|
|
608
|
+
'#text': 'Black',
|
|
609
|
+
},
|
|
610
|
+
Size: {
|
|
611
|
+
'@id': 'M',
|
|
612
|
+
'#text': 'Medium',
|
|
613
|
+
},
|
|
614
|
+
},
|
|
615
|
+
},
|
|
616
|
+
],
|
|
617
|
+
},
|
|
618
|
+
},
|
|
619
|
+
};
|
|
620
|
+
|
|
621
|
+
const xml = builder.build(data, 'OrderDetailResponse');
|
|
622
|
+
|
|
623
|
+
// Output:
|
|
624
|
+
// <?xml version="1.0" encoding="UTF-8"?>
|
|
625
|
+
// <OrderDetailResponse>
|
|
626
|
+
// <Order customerOrderId="12345" levelOfService="REGULAR">
|
|
627
|
+
// <Customer customerId="232">
|
|
628
|
+
// <Name>
|
|
629
|
+
// <FirstName>John</FirstName>
|
|
630
|
+
// <LastName>Doe</LastName>
|
|
631
|
+
// </Name>
|
|
632
|
+
// <EmailAddress>john@example.com</EmailAddress>
|
|
633
|
+
// </Customer>
|
|
634
|
+
// <OrderSource type="source">WEBSTORE</OrderSource>
|
|
635
|
+
// <OrderItems>
|
|
636
|
+
// <OrderItem id="item1">
|
|
637
|
+
// <ItemId>SKU001</ItemId>
|
|
638
|
+
// <Quantity>2</Quantity>
|
|
639
|
+
// <Description>
|
|
640
|
+
// <Color id="9101">Black</Color>
|
|
641
|
+
// <Size id="M">Medium</Size>
|
|
642
|
+
// </Description>
|
|
643
|
+
// </OrderItem>
|
|
644
|
+
// </OrderItems>
|
|
645
|
+
// </Order>
|
|
646
|
+
// </OrderDetailResponse>
|
|
647
|
+
```
|
|
648
|
+
|
|
649
|
+
#### Pattern 5: Arrays (Repeated Elements)
|
|
650
|
+
|
|
651
|
+
```typescript
|
|
652
|
+
const data = {
|
|
653
|
+
OrderItems: {
|
|
654
|
+
OrderItem: [
|
|
655
|
+
{ ItemId: 'SKU001', Quantity: 2 },
|
|
656
|
+
{ ItemId: 'SKU002', Quantity: 1 },
|
|
657
|
+
],
|
|
658
|
+
},
|
|
659
|
+
};
|
|
660
|
+
|
|
661
|
+
// Output:
|
|
662
|
+
// <OrderItems>
|
|
663
|
+
// <OrderItem>
|
|
664
|
+
// <ItemId>SKU001</ItemId>
|
|
665
|
+
// <Quantity>2</Quantity>
|
|
666
|
+
// </OrderItem>
|
|
667
|
+
// <OrderItem>
|
|
668
|
+
// <ItemId>SKU002</ItemId>
|
|
669
|
+
// <Quantity>1</Quantity>
|
|
670
|
+
// </OrderItem>
|
|
671
|
+
// </OrderItems>
|
|
672
|
+
```
|
|
673
|
+
|
|
674
|
+
#### Decision Tree: When to Use `#text`
|
|
675
|
+
|
|
676
|
+
```
|
|
677
|
+
Does the element have attributes?
|
|
678
|
+
├─ NO → Just use field name
|
|
679
|
+
│ "ItemId": "SKU001"
|
|
680
|
+
│ Result: <ItemId>SKU001</ItemId>
|
|
681
|
+
│
|
|
682
|
+
└─ YES → Does it also have text content?
|
|
683
|
+
├─ NO → Just use @prefix
|
|
684
|
+
│ "@id": "123"
|
|
685
|
+
│ Result: <Order id="123"/>
|
|
686
|
+
│
|
|
687
|
+
└─ YES → Use BOTH @prefix AND #text
|
|
688
|
+
"@id": "9101", "#text": "MULTI-COLOR"
|
|
689
|
+
Result: <Color id="9101">MULTI-COLOR</Color>
|
|
690
|
+
```
|
|
691
|
+
|
|
692
|
+
#### Common Issues and Solutions
|
|
693
|
+
|
|
694
|
+
**Issue: Text appears as child element**
|
|
695
|
+
|
|
696
|
+
```typescript
|
|
697
|
+
// ❌ Wrong - creates <text> element
|
|
698
|
+
"Color": { "@id": "9101", "text": "MULTI-COLOR" }
|
|
699
|
+
|
|
700
|
+
// ✅ Correct - creates text content
|
|
701
|
+
"Color": { "@id": "9101", "#text": "MULTI-COLOR" }
|
|
702
|
+
```
|
|
703
|
+
|
|
704
|
+
**Issue: Attributes not showing**
|
|
705
|
+
|
|
706
|
+
```typescript
|
|
707
|
+
// ❌ Wrong - 'id' becomes child element
|
|
708
|
+
"Color": { "id": "9101", "#text": "MULTI-COLOR" }
|
|
709
|
+
|
|
710
|
+
// ✅ Correct - @ prefix makes it an attribute
|
|
711
|
+
"Color": { "@id": "9101", "#text": "MULTI-COLOR" }
|
|
712
|
+
```
|
|
713
|
+
|
|
714
|
+
**Issue: Self-closing tag when content expected**
|
|
715
|
+
|
|
716
|
+
```typescript
|
|
717
|
+
// ❌ Wrong - empty or undefined value
|
|
718
|
+
"Description": { "#text": undefined } // → <Description/>
|
|
719
|
+
|
|
720
|
+
// ✅ Correct - provide actual value
|
|
721
|
+
"Description": { "#text": product?.description || '' }
|
|
722
|
+
```
|
|
723
|
+
|
|
724
|
+
#### Real-World Example: Fluent → SFCC Order Export
|
|
725
|
+
|
|
726
|
+
```typescript
|
|
727
|
+
// Fluent Commerce order data
|
|
728
|
+
const fluentOrder = await fluentClient.graphql({
|
|
729
|
+
query: orderQuery,
|
|
730
|
+
variables: { ref: orderId },
|
|
731
|
+
});
|
|
732
|
+
|
|
733
|
+
// Map to SFCC XML structure using UniversalMapper
|
|
734
|
+
const mapper = new UniversalMapper(mappingConfig, {
|
|
735
|
+
customResolvers: extractionResolvers,
|
|
736
|
+
});
|
|
737
|
+
|
|
738
|
+
const mapped = await mapper.map(fluentOrder);
|
|
739
|
+
|
|
740
|
+
// Generate XML
|
|
741
|
+
const builder = new XMLBuilder({
|
|
742
|
+
xmlDeclaration: true,
|
|
743
|
+
prettyPrint: true,
|
|
744
|
+
encoding: 'UTF-8',
|
|
745
|
+
});
|
|
746
|
+
|
|
747
|
+
const sfccXml = builder.build(mapped.data, 'OrderDetailResponse');
|
|
748
|
+
|
|
749
|
+
// Returns valid SFCC XML ready to send
|
|
750
|
+
console.log(sfccXml);
|
|
751
|
+
```
|
|
752
|
+
|
|
753
|
+
## Parquet Parser Service
|
|
754
|
+
|
|
755
|
+
**FULLY FUNCTIONAL**: The ParquetParserService is now **FULLY IMPLEMENTED** and ready for use, powered by the high-performance `hyparquet` library. It provides complete Parquet file parsing with streaming support, validation, and detailed statistics.
|
|
756
|
+
|
|
757
|
+
**Large File Testing Support**: The SDK includes complete test files for performance validation:
|
|
758
|
+
|
|
759
|
+
- **Small**: 3KB files for unit testing
|
|
760
|
+
- **Medium**: 170KB files for integration testing
|
|
761
|
+
- **Large**: 1.7MB files for standard processing
|
|
762
|
+
- **50K Records**: 8.5MB files with 50,000 records
|
|
763
|
+
- **100K Records**: 17MB files with 100,000 records
|
|
764
|
+
- **250K Records**: 43MB files with 250,000 records
|
|
765
|
+
- **500K Records**: 85MB files with 500,000 records
|
|
766
|
+
- **1M Records**: 169MB files with 1,000,000 records
|
|
767
|
+
|
|
768
|
+
```typescript
|
|
769
|
+
import {
|
|
770
|
+
ParquetParserService,
|
|
771
|
+
createConsoleLogger,
|
|
772
|
+
toStructuredLogger
|
|
773
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
774
|
+
|
|
775
|
+
const logger = toStructuredLogger(createConsoleLogger(), {
|
|
776
|
+
logLevel: 'info'
|
|
777
|
+
});
|
|
778
|
+
const parser = new ParquetParserService(logger, {
|
|
779
|
+
batchSize: 1000,
|
|
780
|
+
maxFileSize: 100 * 1024 * 1024, // 100MB
|
|
781
|
+
enableStreaming: true,
|
|
782
|
+
validation: {
|
|
783
|
+
enabled: true,
|
|
784
|
+
requiredColumns: ['productRef', 'qty'],
|
|
785
|
+
columnTypes: {
|
|
786
|
+
productRef: 'string',
|
|
787
|
+
qty: 'int32',
|
|
788
|
+
},
|
|
789
|
+
},
|
|
790
|
+
});
|
|
791
|
+
|
|
792
|
+
// Parse Parquet file - FULLY WORKING
|
|
793
|
+
const { records, stats } = await parser.parse(arrayBuffer, 'inventory.parquet');
|
|
794
|
+
|
|
795
|
+
console.log(`Parsed ${stats.totalRecords} records`);
|
|
796
|
+
console.log(`Processing time: ${stats.processingTimeMs}ms`);
|
|
797
|
+
console.log(`Records per second: ${stats.recordsPerSecond}`);
|
|
798
|
+
|
|
799
|
+
// Parse large files (e.g., 500K records)
|
|
800
|
+
const largeBuffer = await fs.readFile('inventory-position-500k.parquet');
|
|
801
|
+
const largeArrayBuffer = largeBuffer.buffer.slice(
|
|
802
|
+
largeBuffer.byteOffset,
|
|
803
|
+
largeBuffer.byteOffset + largeBuffer.byteLength
|
|
804
|
+
);
|
|
805
|
+
|
|
806
|
+
const { records: largeRecords, stats: largeStats } = await parser.parse(
|
|
807
|
+
largeArrayBuffer,
|
|
808
|
+
'inventory-position-500k.parquet',
|
|
809
|
+
{
|
|
810
|
+
batchSize: 10000, // Larger batch size for performance
|
|
811
|
+
validation: {
|
|
812
|
+
enabled: false, // Disable validation for speed
|
|
813
|
+
},
|
|
814
|
+
}
|
|
815
|
+
);
|
|
816
|
+
|
|
817
|
+
console.log(`Processed ${largeStats.totalRecords} records in ${largeStats.processingTimeMs}ms`);
|
|
818
|
+
console.log(`Throughput: ${largeStats.recordsPerSecond} records/sec`);
|
|
819
|
+
|
|
820
|
+
// Stream massive files (1M+ records)
|
|
821
|
+
const streamResult = await parser.parseStreaming(massiveArrayBuffer, 'mixed-positions-1m.parquet');
|
|
822
|
+
|
|
823
|
+
for await (const batch of streamResult.records) {
|
|
824
|
+
console.log(`Processing batch of ${batch.length} records`);
|
|
825
|
+
}
|
|
826
|
+
```
|
|
827
|
+
|
|
828
|
+
**Constructor:**
|
|
829
|
+
|
|
830
|
+
```typescript
|
|
831
|
+
constructor(
|
|
832
|
+
logger: StructuredLogger,
|
|
833
|
+
options?: ParquetParsingOptions
|
|
834
|
+
)
|
|
835
|
+
```
|
|
836
|
+
|
|
837
|
+
**Methods:**
|
|
838
|
+
|
|
839
|
+
### parse
|
|
840
|
+
|
|
841
|
+
Parse a complete Parquet file into memory. Fully functional with hyparquet.
|
|
842
|
+
|
|
843
|
+
```typescript
|
|
844
|
+
async parse(
|
|
845
|
+
arrayBuffer: ArrayBuffer,
|
|
846
|
+
fileName: string,
|
|
847
|
+
options?: Partial<ParquetParsingOptions>
|
|
848
|
+
): Promise<{
|
|
849
|
+
records: ParquetDataRecord[];
|
|
850
|
+
stats: ParquetParsingStats;
|
|
851
|
+
}>
|
|
852
|
+
```
|
|
853
|
+
|
|
854
|
+
**Parameters:**
|
|
855
|
+
|
|
856
|
+
- `arrayBuffer: ArrayBuffer` - The Parquet file data as ArrayBuffer
|
|
857
|
+
- `fileName: string` - Name of the file being parsed (for logging)
|
|
858
|
+
- `options?: Partial<ParquetParsingOptions>` - Optional parsing configuration
|
|
859
|
+
|
|
860
|
+
**Returns:**
|
|
861
|
+
|
|
862
|
+
- `records: ParquetDataRecord[]` - Array of parsed records
|
|
863
|
+
- `stats: ParquetParsingStats` - Detailed parsing statistics
|
|
864
|
+
|
|
865
|
+
**Example:**
|
|
866
|
+
|
|
867
|
+
```typescript
|
|
868
|
+
const buffer = await fs.readFile('data.parquet');
|
|
869
|
+
const arrayBuffer = buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength);
|
|
870
|
+
|
|
871
|
+
const { records, stats } = await parser.parse(arrayBuffer, 'data.parquet', {
|
|
872
|
+
batchSize: 500,
|
|
873
|
+
validation: {
|
|
874
|
+
enabled: true,
|
|
875
|
+
requiredColumns: ['id', 'name', 'qty'],
|
|
876
|
+
},
|
|
877
|
+
});
|
|
878
|
+
|
|
879
|
+
// Check validation results
|
|
880
|
+
if (stats.validation && !stats.validation.isValid) {
|
|
881
|
+
console.error('Validation errors:', stats.validation.errors);
|
|
882
|
+
}
|
|
883
|
+
```
|
|
884
|
+
|
|
885
|
+
### parseStreaming
|
|
886
|
+
|
|
887
|
+
Parse Parquet file with streaming support for memory-efficient processing of large files.
|
|
888
|
+
|
|
889
|
+
```typescript
|
|
890
|
+
async parseStreaming(
|
|
891
|
+
arrayBuffer: ArrayBuffer,
|
|
892
|
+
fileName: string,
|
|
893
|
+
options?: Partial<ParquetParsingOptions>
|
|
894
|
+
): Promise<ParquetStreamingResult>
|
|
895
|
+
```
|
|
896
|
+
|
|
897
|
+
**Parameters:**
|
|
898
|
+
|
|
899
|
+
- `arrayBuffer: ArrayBuffer` - The Parquet file data
|
|
900
|
+
- `fileName: string` - Name of the file being parsed
|
|
901
|
+
- `options?: Partial<ParquetParsingOptions>` - Optional configuration
|
|
902
|
+
|
|
903
|
+
**Returns:**
|
|
904
|
+
|
|
905
|
+
- `records: AsyncIterable<ParquetDataRecord[]>` - Async iterator of record batches
|
|
906
|
+
- `stats: Promise<ParquetParsingStats>` - Promise resolving to final statistics
|
|
907
|
+
- `cancel: () => void` - Function to cancel streaming
|
|
908
|
+
|
|
909
|
+
**Example:**
|
|
910
|
+
|
|
911
|
+
```typescript
|
|
912
|
+
const streamResult = await parser.parseStreaming(arrayBuffer, 'large_dataset.parquet', {
|
|
913
|
+
batchSize: 1000,
|
|
914
|
+
});
|
|
915
|
+
|
|
916
|
+
// Process batches as they stream
|
|
917
|
+
for await (const batch of streamResult.records) {
|
|
918
|
+
for (const record of batch) {
|
|
919
|
+
await processRecord(record);
|
|
920
|
+
}
|
|
921
|
+
}
|
|
922
|
+
|
|
923
|
+
// Get final statistics
|
|
924
|
+
const stats = await streamResult.stats;
|
|
925
|
+
console.log(`Total records: ${stats.totalRecords}`);
|
|
926
|
+
|
|
927
|
+
// Cancel if needed
|
|
928
|
+
setTimeout(() => streamResult.cancel(), 5000);
|
|
929
|
+
```
|
|
930
|
+
|
|
931
|
+
### validateFile
|
|
932
|
+
|
|
933
|
+
Validate a Parquet file before parsing.
|
|
934
|
+
|
|
935
|
+
```typescript
|
|
936
|
+
async validateFile(
|
|
937
|
+
arrayBuffer: ArrayBuffer,
|
|
938
|
+
fileName: string
|
|
939
|
+
): Promise<ValidationResult>
|
|
940
|
+
```
|
|
941
|
+
|
|
942
|
+
**Parameters:**
|
|
943
|
+
|
|
944
|
+
- `arrayBuffer: ArrayBuffer` - The Parquet file data
|
|
945
|
+
- `fileName: string` - Name of the file
|
|
946
|
+
|
|
947
|
+
**Returns:**
|
|
948
|
+
|
|
949
|
+
- `ValidationResult` - Validation results with errors and warnings
|
|
950
|
+
|
|
951
|
+
**Example:**
|
|
952
|
+
|
|
953
|
+
```typescript
|
|
954
|
+
const validation = await parser.validateFile(arrayBuffer, 'data.parquet');
|
|
955
|
+
|
|
956
|
+
if (!validation.isValid) {
|
|
957
|
+
console.error('File validation failed:');
|
|
958
|
+
validation.errors.forEach(error => {
|
|
959
|
+
console.error(`- ${error.field}: ${error.message}`);
|
|
960
|
+
});
|
|
961
|
+
}
|
|
962
|
+
```
|
|
963
|
+
|
|
964
|
+
### getPerformanceMetrics
|
|
965
|
+
|
|
966
|
+
Get current performance metrics for the parser.
|
|
967
|
+
|
|
968
|
+
```typescript
|
|
969
|
+
getPerformanceMetrics(): ParquetPerformanceMetrics
|
|
970
|
+
```
|
|
971
|
+
|
|
972
|
+
**Returns:**
|
|
973
|
+
|
|
974
|
+
- `totalRequests: number` - Total parsing requests
|
|
975
|
+
- `successfulRequests: number` - Successful parses
|
|
976
|
+
- `failedRequests: number` - Failed parses
|
|
977
|
+
- `averageResponseTime: number` - Average parsing time (ms)
|
|
978
|
+
- `lastRequestTime: Date` - Timestamp of last request
|
|
979
|
+
|
|
980
|
+
**Example:**
|
|
981
|
+
|
|
982
|
+
```typescript
|
|
983
|
+
const metrics = parser.getPerformanceMetrics();
|
|
984
|
+
console.log(
|
|
985
|
+
`Success rate: ${((metrics.successfulRequests / metrics.totalRequests) * 100).toFixed(2)}%`
|
|
986
|
+
);
|
|
987
|
+
console.log(`Average time: ${metrics.averageResponseTime.toFixed(2)}ms`);
|
|
988
|
+
```
|
|
989
|
+
|
|
990
|
+
### resetMetrics
|
|
991
|
+
|
|
992
|
+
Reset performance metrics to initial state.
|
|
993
|
+
|
|
994
|
+
```typescript
|
|
995
|
+
resetMetrics(): void
|
|
996
|
+
```
|
|
997
|
+
|
|
998
|
+
**Example:**
|
|
999
|
+
|
|
1000
|
+
```typescript
|
|
1001
|
+
// Reset metrics after batch processing
|
|
1002
|
+
parser.resetMetrics();
|
|
1003
|
+
console.log('Performance metrics reset');
|
|
1004
|
+
```
|
|
1005
|
+
|
|
1006
|
+
**Interfaces:**
|
|
1007
|
+
|
|
1008
|
+
#### ParquetParsingOptions
|
|
1009
|
+
|
|
1010
|
+
```typescript
|
|
1011
|
+
interface ParquetParsingOptions {
|
|
1012
|
+
batchSize?: number; // Records per batch (default: 1000)
|
|
1013
|
+
maxFileSize?: number; // Max file size in bytes (default: 100MB)
|
|
1014
|
+
enableStreaming?: boolean; // Enable streaming mode (default: true)
|
|
1015
|
+
validation?: ParquetValidationOptions;
|
|
1016
|
+
}
|
|
1017
|
+
```
|
|
1018
|
+
|
|
1019
|
+
#### ParquetValidationOptions
|
|
1020
|
+
|
|
1021
|
+
```typescript
|
|
1022
|
+
interface ParquetValidationOptions {
|
|
1023
|
+
enabled?: boolean;
|
|
1024
|
+
requiredColumns?: string[];
|
|
1025
|
+
columnTypes?: Record<string, ParquetDataType>;
|
|
1026
|
+
allowAdditionalColumns?: boolean;
|
|
1027
|
+
}
|
|
1028
|
+
```
|
|
1029
|
+
|
|
1030
|
+
#### ParquetParsingStats
|
|
1031
|
+
|
|
1032
|
+
```typescript
|
|
1033
|
+
interface ParquetParsingStats {
|
|
1034
|
+
totalRecords: number;
|
|
1035
|
+
processingTimeMs: number;
|
|
1036
|
+
fileSizeBytes: number;
|
|
1037
|
+
recordsPerSecond: number;
|
|
1038
|
+
peakMemoryUsage: number;
|
|
1039
|
+
batchesProcessed: number;
|
|
1040
|
+
compressionRatio?: number;
|
|
1041
|
+
columnStats: Record<string, ParquetColumnStatistics>;
|
|
1042
|
+
validation?: ValidationResult;
|
|
1043
|
+
}
|
|
1044
|
+
```
|
|
1045
|
+
|
|
1046
|
+
#### ParquetStreamingResult
|
|
1047
|
+
|
|
1048
|
+
```typescript
|
|
1049
|
+
interface ParquetStreamingResult {
|
|
1050
|
+
records: AsyncIterable<ParquetDataRecord[]>;
|
|
1051
|
+
stats: Promise<ParquetParsingStats>;
|
|
1052
|
+
cancel: () => void;
|
|
1053
|
+
}
|
|
1054
|
+
```
|
|
1055
|
+
|
|
1056
|
+
#### ParquetDataType
|
|
1057
|
+
|
|
1058
|
+
```typescript
|
|
1059
|
+
type ParquetDataType =
|
|
1060
|
+
| 'string'
|
|
1061
|
+
| 'number'
|
|
1062
|
+
| 'boolean'
|
|
1063
|
+
| 'date'
|
|
1064
|
+
| 'int32'
|
|
1065
|
+
| 'int64'
|
|
1066
|
+
| 'float'
|
|
1067
|
+
| 'double'
|
|
1068
|
+
| 'binary';
|
|
1069
|
+
```
|
|
1070
|
+
|
|
1071
|
+
## Parser Comparison
|
|
1072
|
+
|
|
1073
|
+
| Parser | Streaming | Validation | Best For |
|
|
1074
|
+
| ------------------------ | --------- | ---------- | --------------------------------- |
|
|
1075
|
+
| **CSVParserService** | ✅ | ✅ | Text-based tabular data |
|
|
1076
|
+
| **JSONParserService** | ✅ | ✅ | JSON/JSONL formats, API responses |
|
|
1077
|
+
| **XMLParserService** | ✅ | ✅ | XML documents, SOAP APIs, SFCC |
|
|
1078
|
+
| **ParquetParserService** | ✅ | ✅ | Columnar data, large datasets |
|
|
1079
|
+
|
|
1080
|
+
**Choosing a Parser:**
|
|
1081
|
+
|
|
1082
|
+
- **CSV**: Simple imports, spreadsheets, EDI and B2B systems
|
|
1083
|
+
- **JSON**: Modern APIs, flexible schemas, nested data
|
|
1084
|
+
- **XML**: Enterprise systems, SOAP, SFCC orders
|
|
1085
|
+
- **Parquet**: Big data, analytics, data lakes
|
|
1086
|
+
|
|
1087
|
+
## See Also
|
|
1088
|
+
|
|
1089
|
+
- [Module 06: Data Sources](./api-reference-06-data-sources.md) - File operations
|
|
1090
|
+
- [Module 05: Services](./api-reference-05-services.md#universal-mapper) - Field transformations
|
|
1091
|
+
- [Module 10: Utilities](./api-reference-10-utilities.md) - Helper functions
|
|
1092
|
+
- [Parquet Guide](../../../03-PATTERN-GUIDES/parquet/parquet-readme.md) - Detailed Parquet examples
|
|
1093
|
+
- [XML Parser Guide](../../parsers/modules/02-core-guides-parsers-04-xml-parser.md) - XML patterns
|
|
1094
|
+
|
|
1095
|
+
---
|
|
1096
|
+
|
|
1097
|
+
**[← Previous: Module 06 - Data Sources](./api-reference-06-data-sources.md)** | **[API Reference Home](../api-reference-readme.md)** | **[Next: Module 08 - Types →](./api-reference-08-types.md)**
|