@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/02-CORE-GUIDES/ingestion/readme.md

@@ -1,160 +1,160 @@
-# Data Ingestion Learning Path
-
-> **Complete guide to ingesting data into Fluent Commerce**
-> Progressive learning modules with validated schema examples and production-ready patterns
-
-## Quick Start
-
-```typescript
-import { createClient, S3DataSource, CSVParserService, BatchAPIClient } from '@fluentcommerce/fc-connect-sdk';
-
-// Initialize services
-const client = await createClient({ config: {/*...*/} });
-const s3 = new S3DataSource({/*...*/}, logger);
-const parser = new CSVParserService(logger);
-const batchClient = new BatchAPIClient(client, logger);
-
-// Ingest data
-const files = await s3.listFiles({ prefix: 'data/' });
-for (const file of files) {
-  const data = await s3.downloadFile(file.path);
-  const records = await parser.parse(data);
-  await batchClient.bulkUpdateInventory(records);
-}
-```
-
-## 📚 What You'll Learn
-
-This guide teaches you how to ingest data into Fluent Commerce from various sources using the FC Connect SDK. By the end, you'll understand:
-
-- ✅ How to ingest inventory data using the Batch API (INVENTORY only)
-- ✅ How to read data from multiple sources (S3, SFTP, local files)
-- ✅ How to parse CSV, Parquet, XML, and JSON files
-- ✅ How to transform source data with UniversalMapper and custom resolvers
-- ✅ How to manage state for duplicate prevention
-- ✅ How to optimize performance with batch sizing and job strategies
-- ✅ How to handle errors and implement production-ready patterns
-
-## 🎯 Learning Path Overview
-
-This guide is organized into **9 progressive modules**. Follow them in order for the best learning experience:
-
-| Module | Topic | Level | Time |
-|--------|-------|-------|------|
-| [Module 1](./modules/02-core-guides-ingestion-01-introduction.md) | **Introduction** - What is ingestion, when to use it, key concepts | Beginner | 10 min |
-| [Module 2](./modules/02-core-guides-ingestion-02-quick-start.md) | **Quick Start** - First ingestion workflow with CSV and S3 | Beginner | 15 min |
-| [Module 3](./modules/02-core-guides-ingestion-03-data-sources.md) | **Data Sources & Parsers** - S3, SFTP, CSV parsing, validation | Intermediate | 20 min |
-| [Module 4](./modules/02-core-guides-ingestion-04-field-mapping.md) | **Field Mapping** - Transform data with UniversalMapper | Intermediate | 25 min |
-| [Module 5](./modules/02-core-guides-ingestion-05-advanced-parsers.md) | **Advanced Parsers** - Parquet, XML, JSON parsing and streaming | Intermediate | 25 min |
-| [Module 6](./modules/02-core-guides-ingestion-06-batch-api.md) | **Batch API** - Job creation, batches, status monitoring | Intermediate | 30 min |
-| [Module 7](./modules/02-core-guides-ingestion-07-state-management.md) | **State Management** - Duplicate prevention, file tracking | Advanced | 20 min |
-| [Module 8](./modules/02-core-guides-ingestion-08-performance-optimization.md) | **Performance** - Batch sizing, parallel processing, strategies | Advanced | 25 min |
-| [Module 9](./modules/02-core-guides-ingestion-09-best-practices.md) | **Best Practices** - Error handling, monitoring, production patterns | All Levels | 20 min |
-
-## 🚀 Quick Navigation
-
-### I Want To...
-
-| **Goal** | **Start Here** | **Then See** |
-|----------|----------------|--------------|
-| **Get started quickly** | [Quick Start](./modules/02-core-guides-ingestion-02-quick-start.md) | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) |
-| **Understand ingestion** | [Introduction](./modules/02-core-guides-ingestion-01-introduction.md) | [Data Sources](./modules/02-core-guides-ingestion-03-data-sources.md) |
-| **Read from S3** | [Data Sources: S3](./modules/02-core-guides-ingestion-03-data-sources.md#s3-data-source) | [Quick Start](./modules/02-core-guides-ingestion-02-quick-start.md) |
-| **Read from SFTP** | [Data Sources: SFTP](./modules/02-core-guides-ingestion-03-data-sources.md#sftp-data-source) | [Module 3](./modules/02-core-guides-ingestion-03-data-sources.md) |
-| **Parse CSV files** | [Data Sources: CSV](./modules/02-core-guides-ingestion-03-data-sources.md#integrated-csv-parsing) | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) |
-| **Parse Parquet/XML/JSON** | [Advanced Parsers](./modules/02-core-guides-ingestion-05-advanced-parsers.md) | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) |
-| **Transform data** | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) | [Batch API](./modules/02-core-guides-ingestion-06-batch-api.md) |
-| **Use Batch API** | [Batch API](./modules/02-core-guides-ingestion-06-batch-api.md) | [State Management](./modules/02-core-guides-ingestion-07-state-management.md) |
-| **Prevent duplicates** | [State Management](./modules/02-core-guides-ingestion-07-state-management.md) | [Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md) |
-| **Optimize performance** | [Performance Optimization](./modules/02-core-guides-ingestion-08-performance-optimization.md) | [Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md) |
-| **Handle errors** | [Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md) | [State Management](./modules/02-core-guides-ingestion-07-state-management.md) |
-
-## 📖 Quick Reference
-
-For quick lookups, see:
-- **[Quick Reference Cheat Sheet](../advanced-services/advanced-services-quick-reference.md)** - One-page guide with common patterns, batch sizing, error handling
-- [Batch API Schema Reference](./modules/02-core-guides-ingestion-06-batch-api.md#batch-api-schema) - Complete API structure
-- [Field Mapping Examples](./modules/02-core-guides-ingestion-04-field-mapping.md#complete-mapping-example) - Copy-paste ready configs
-- [Job Strategy Comparison](./modules/02-core-guides-ingestion-08-performance-optimization.md#job-strategy-comparison) - When to use DAILY vs PER_FILE
-- [Common Errors](./modules/02-core-guides-ingestion-09-best-practices.md#common-errors) - Troubleshooting guide
-
-## 🎓 Prerequisites
-
-Before starting this guide, you should:
-
-1. ✅ Have the SDK installed: `npm install @fluentcommerce/fc-connect-sdk`
-2. ✅ Have Fluent API credentials (clientId, clientSecret, retailerId)
-3. ✅ Have access to a data source (S3, SFTP, or local files)
-4. ✅ Understand CSV format basics
-5. ✅ Be familiar with [Universal Mapping Guide](../mapping/readme.md) (recommended)
-
-**New to the SDK?** Start with the [Getting Started Guide](../../00-START-HERE/getting-started.md) first.
-
-## ⚠️ Critical: Batch API Limitations
-
-### INVENTORY Only
-
-The Fluent Commerce Batch API has strict limitations:
-
-| Feature | Supported | Notes |
-|---------|-----------|-------|
-| **Entity Type** | ✅ `INVENTORY` ONLY | ❌ Cannot use for ORDER, PRODUCT, LOCATION |
-| **Action** | ✅ `UPSERT` ONLY | ❌ No DELETE, CREATE-only, or UPDATE-only |
-| **Alternative** | GraphQL mutations | For other entities, use direct mutations |
-
-**What this means:**
-- ✅ **Inventory ingestion** - Use Batch API (this guide)
-- ❌ **Order ingestion** - Use `createOrder` mutation ([Connector Scenarios](../../03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md))
-- ❌ **Product ingestion** - Use `createProduct` or `updateProduct` mutations
-- ❌ **Location ingestion** - Use location-specific mutations
-
-See [Module 6: Batch API](./modules/02-core-guides-ingestion-06-batch-api.md#batch-api-schema) for complete details.
-
-## 🔗 Related Documentation
-
-### Core SDK Guides
-- **[Universal Mapping Guide](../mapping/readme.md)** - Field transformation patterns (companion guide)
-- **[Data Extraction Guide](../extraction/)** - Reverse workflow (Fluent → External) (companion guide)
-- **[S3 Operations Guide](../data-sources/modules/data-sources-02-s3-operations.md)** - Complete S3 integration patterns
-- **[SFTP Operations Guide](../data-sources/modules/data-sources-03-sftp-operations.md)** - Complete SFTP integration patterns
-
-### Advanced Guides
-- **[Resolver Development Guide](../mapping/resolvers/mapping-resolvers-readme.md)** - Custom transformation resolvers
-- **[Auto-Pagination Guide](../auto-pagination/auto-pagination-readme.md)** - For extraction workflows
-- **[Orchestrators Guide](../extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md)** - High-level workflow coordination
-
-### Integration Examples
-- **[Connector Scenarios](../../03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md)** - Real-world integration patterns
-- **[Versori Platform Guide](../../04-REFERENCE/platforms/versori/)** - Deploy to Versori platform
-
-### API Reference
-- **[Batch API Reference](../api-reference/modules/api-reference-01-client-api.md#job-batch-operations)** - Complete API documentation
-- **[Schema File](../../04-REFERENCE/schema/fluent-commerce-schema.json)** - Fluent Commerce GraphQL schema
-
-## 💡 Tips for Best Results
-
-1. **Always validate mappings** - Use schema introspection before deployment
-2. **Start with small batches** - Test with `batchSize: 100` before scaling
-3. **Use state management** - Prevent duplicate processing with `StateService`
-4. **Choose right job strategy** - DAILY for frequent updates, PER_FILE for large datasets
-5. **Implement error handling** - Add retry logic and validation
-6. **Monitor performance** - Track batch processing times and success rates
-7. **Test transformations** - Validate field mappings with sample data first
-
-## 🚦 Getting Started
-
-Ready to begin? Start with [Module 1: Introduction →](./modules/02-core-guides-ingestion-01-introduction.md)
-
----
-
-**Need Help?**
-- 📖 Browse [Examples](./examples/)
-- 🔍 Check [Module 9: Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md)
-- 📧 Contact support@fluentcommerce.com
-- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)
-
----
-
-**Companion Guides:**
-- 📥 **Ingestion** (this guide) - External → Fluent Commerce
-- 📤 **[Extraction](../extraction/)** - Fluent Commerce → External
+# Data Ingestion Learning Path
+
+> **Complete guide to ingesting data into Fluent Commerce**
+> Progressive learning modules with validated schema examples and production-ready patterns
+
+## Quick Start
+
+```typescript
+import { createClient, S3DataSource, CSVParserService, BatchAPIClient } from '@fluentcommerce/fc-connect-sdk';
+
+// Initialize services
+const client = await createClient({ config: {/*...*/} });
+const s3 = new S3DataSource({/*...*/}, logger);
+const parser = new CSVParserService(logger);
+const batchClient = new BatchAPIClient(client, logger);
+
+// Ingest data
+const files = await s3.listFiles({ prefix: 'data/' });
+for (const file of files) {
+  const data = await s3.downloadFile(file.path);
+  const records = await parser.parse(data);
+  await batchClient.bulkUpdateInventory(records);
+}
+```
+
+## 📚 What You'll Learn
+
+This guide teaches you how to ingest data into Fluent Commerce from various sources using the FC Connect SDK. By the end, you'll understand:
+
+- ✅ How to ingest inventory data using the Batch API (INVENTORY only)
+- ✅ How to read data from multiple sources (S3, SFTP, local files)
+- ✅ How to parse CSV, Parquet, XML, and JSON files
+- ✅ How to transform source data with UniversalMapper and custom resolvers
+- ✅ How to manage state for duplicate prevention
+- ✅ How to optimize performance with batch sizing and job strategies
+- ✅ How to handle errors and implement production-ready patterns
+
+## 🎯 Learning Path Overview
+
+This guide is organized into **9 progressive modules**. Follow them in order for the best learning experience:
+
+| Module | Topic | Level | Time |
+|--------|-------|-------|------|
+| [Module 1](./modules/02-core-guides-ingestion-01-introduction.md) | **Introduction** - What is ingestion, when to use it, key concepts | Beginner | 10 min |
+| [Module 2](./modules/02-core-guides-ingestion-02-quick-start.md) | **Quick Start** - First ingestion workflow with CSV and S3 | Beginner | 15 min |
+| [Module 3](./modules/02-core-guides-ingestion-03-data-sources.md) | **Data Sources & Parsers** - S3, SFTP, CSV parsing, validation | Intermediate | 20 min |
+| [Module 4](./modules/02-core-guides-ingestion-04-field-mapping.md) | **Field Mapping** - Transform data with UniversalMapper | Intermediate | 25 min |
+| [Module 5](./modules/02-core-guides-ingestion-05-advanced-parsers.md) | **Advanced Parsers** - Parquet, XML, JSON parsing and streaming | Intermediate | 25 min |
+| [Module 6](./modules/02-core-guides-ingestion-06-batch-api.md) | **Batch API** - Job creation, batches, status monitoring | Intermediate | 30 min |
+| [Module 7](./modules/02-core-guides-ingestion-07-state-management.md) | **State Management** - Duplicate prevention, file tracking | Advanced | 20 min |
+| [Module 8](./modules/02-core-guides-ingestion-08-performance-optimization.md) | **Performance** - Batch sizing, parallel processing, strategies | Advanced | 25 min |
+| [Module 9](./modules/02-core-guides-ingestion-09-best-practices.md) | **Best Practices** - Error handling, monitoring, production patterns | All Levels | 20 min |
+
+## 🚀 Quick Navigation
+
+### I Want To...
+
+| **Goal** | **Start Here** | **Then See** |
+|----------|----------------|--------------|
+| **Get started quickly** | [Quick Start](./modules/02-core-guides-ingestion-02-quick-start.md) | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) |
+| **Understand ingestion** | [Introduction](./modules/02-core-guides-ingestion-01-introduction.md) | [Data Sources](./modules/02-core-guides-ingestion-03-data-sources.md) |
+| **Read from S3** | [Data Sources: S3](./modules/02-core-guides-ingestion-03-data-sources.md#s3-data-source) | [Quick Start](./modules/02-core-guides-ingestion-02-quick-start.md) |
+| **Read from SFTP** | [Data Sources: SFTP](./modules/02-core-guides-ingestion-03-data-sources.md#sftp-data-source) | [Module 3](./modules/02-core-guides-ingestion-03-data-sources.md) |
+| **Parse CSV files** | [Data Sources: CSV](./modules/02-core-guides-ingestion-03-data-sources.md#integrated-csv-parsing) | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) |
+| **Parse Parquet/XML/JSON** | [Advanced Parsers](./modules/02-core-guides-ingestion-05-advanced-parsers.md) | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) |
+| **Transform data** | [Field Mapping](./modules/02-core-guides-ingestion-04-field-mapping.md) | [Batch API](./modules/02-core-guides-ingestion-06-batch-api.md) |
+| **Use Batch API** | [Batch API](./modules/02-core-guides-ingestion-06-batch-api.md) | [State Management](./modules/02-core-guides-ingestion-07-state-management.md) |
+| **Prevent duplicates** | [State Management](./modules/02-core-guides-ingestion-07-state-management.md) | [Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md) |
+| **Optimize performance** | [Performance Optimization](./modules/02-core-guides-ingestion-08-performance-optimization.md) | [Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md) |
+| **Handle errors** | [Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md) | [State Management](./modules/02-core-guides-ingestion-07-state-management.md) |
+
+## 📖 Quick Reference
+
+For quick lookups, see:
+- **[Quick Reference Cheat Sheet](../advanced-services/advanced-services-quick-reference.md)** - One-page guide with common patterns, batch sizing, error handling
+- [Batch API Schema Reference](./modules/02-core-guides-ingestion-06-batch-api.md#batch-api-schema) - Complete API structure
+- [Field Mapping Examples](./modules/02-core-guides-ingestion-04-field-mapping.md#complete-mapping-example) - Copy-paste ready configs
+- [Job Strategy Comparison](./modules/02-core-guides-ingestion-08-performance-optimization.md#job-strategy-comparison) - When to use DAILY vs PER_FILE
+- [Common Errors](./modules/02-core-guides-ingestion-09-best-practices.md#common-errors) - Troubleshooting guide
+
+## 🎓 Prerequisites
+
+Before starting this guide, you should:
+
+1. ✅ Have the SDK installed: `npm install @fluentcommerce/fc-connect-sdk`
+2. ✅ Have Fluent API credentials (clientId, clientSecret, retailerId)
+3. ✅ Have access to a data source (S3, SFTP, or local files)
+4. ✅ Understand CSV format basics
+5. ✅ Be familiar with [Universal Mapping Guide](../mapping/readme.md) (recommended)
+
+**New to the SDK?** Start with the [Getting Started Guide](../../00-START-HERE/getting-started.md) first.
+
+## ⚠️ Critical: Batch API Limitations
+
+### INVENTORY Only
+
+The Fluent Commerce Batch API has strict limitations:
+
+| Feature | Supported | Notes |
+|---------|-----------|-------|
+| **Entity Type** | ✅ `INVENTORY` ONLY | ❌ Cannot use for ORDER, PRODUCT, LOCATION |
+| **Action** | ✅ `UPSERT` ONLY | ❌ No DELETE, CREATE-only, or UPDATE-only |
+| **Alternative** | GraphQL mutations | For other entities, use direct mutations |
+
+**What this means:**
+- ✅ **Inventory ingestion** - Use Batch API (this guide)
+- ❌ **Order ingestion** - Use `createOrder` mutation ([Connector Scenarios](../../03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md))
+- ❌ **Product ingestion** - Use `createProduct` or `updateProduct` mutations
+- ❌ **Location ingestion** - Use location-specific mutations
+
+See [Module 6: Batch API](./modules/02-core-guides-ingestion-06-batch-api.md#batch-api-schema) for complete details.
+
+## 🔗 Related Documentation
+
+### Core SDK Guides
+- **[Universal Mapping Guide](../mapping/readme.md)** - Field transformation patterns (companion guide)
+- **[Data Extraction Guide](../extraction/)** - Reverse workflow (Fluent → External) (companion guide)
+- **[S3 Operations Guide](../data-sources/modules/data-sources-02-s3-operations.md)** - Complete S3 integration patterns
+- **[SFTP Operations Guide](../data-sources/modules/data-sources-03-sftp-operations.md)** - Complete SFTP integration patterns
+
+### Advanced Guides
+- **[Resolver Development Guide](../mapping/resolvers/mapping-resolvers-readme.md)** - Custom transformation resolvers
+- **[Auto-Pagination Guide](../auto-pagination/auto-pagination-readme.md)** - For extraction workflows
+- **[Orchestrators Guide](../extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md)** - High-level workflow coordination
+
+### Integration Examples
+- **[Connector Scenarios](../../03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md)** - Real-world integration patterns
+- **[Versori Platform Guide](../../04-REFERENCE/platforms/versori/)** - Deploy to Versori platform
+
+### API Reference
+- **[Batch API Reference](../api-reference/modules/api-reference-01-client-api.md#job-batch-operations)** - Complete API documentation
+- **[Schema File](../../04-REFERENCE/schema/fluent-commerce-schema.json)** - Fluent Commerce GraphQL schema
+
+## 💡 Tips for Best Results
+
+1. **Always validate mappings** - Use schema introspection before deployment
+2. **Start with small batches** - Test with `batchSize: 100` before scaling
+3. **Use state management** - Prevent duplicate processing with `StateService`
+4. **Choose right job strategy** - DAILY for frequent updates, PER_FILE for large datasets
+5. **Implement error handling** - Add retry logic and validation
+6. **Monitor performance** - Track batch processing times and success rates
+7. **Test transformations** - Validate field mappings with sample data first
+
+## 🚦 Getting Started
+
+Ready to begin? Start with [Module 1: Introduction →](./modules/02-core-guides-ingestion-01-introduction.md)
+
+---
+
+**Need Help?**
+- 📖 Browse [Examples](./examples/)
+- 🔍 Check [Module 9: Best Practices](./modules/02-core-guides-ingestion-09-best-practices.md)
+- 📧 Contact support@fluentcommerce.com
+- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)
+
+---
+
+**Companion Guides:**
+- 📥 **Ingestion** (this guide) - External → Fluent Commerce
+- 📤 **[Extraction](../extraction/)** - Fluent Commerce → External