@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md

@@ -1,637 +1,637 @@
-# Module 3: Intermediate Scenarios
-
-**Level:** Intermediate
-**Estimated Time:** 45 minutes
-
-## Overview
-
-This module covers intermediate connector scenarios including S3 Parquet processing, JSON order creation, and advanced validation patterns. You'll learn how to handle different file formats and more complex data structures.
-
-## Learning Objectives
-
-By the end of this module, you will:
-
-- ✅ Build S3 Parquet to Fluent product catalog sync
-- ✅ Process SFTP JSON files for order creation
-- ✅ Implement schema-based validation
-- ✅ Handle nested data transformations
-- ✅ Use GraphQL mutations for complex entity creation
-- ✅ Implement proper enrichment and data augmentation
-
-## Scenario 2: S3 Parquet to Fluent Product Catalog
-
-**Goal:** Read product data from Parquet files in S3, enrich with additional data, and update Fluent product catalog.
-
-**Complexity:** Medium
-**Components:** S3DataSource, ParquetParserService, UniversalMapper, FluentClient
-**Estimated Lines:** ~200 lines
-
-### Problem Statement
-
-Your data warehouse exports product catalog as Parquet files to S3 daily. You need to:
-
-1. Read Parquet files from S3
-2. Parse columnar format efficiently
-3. Enrich with calculated fields
-4. Transform nested attributes
-5. Send to Fluent as product updates
-
-### Complete Implementation
-
-```typescript
-import {
-  createClient,
-  S3DataSource,
-  ParquetParserService,
-  UniversalMapper,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Configuration
-const config = {
-  s3: {
-    bucketName: 'product-data-lake',
-    region: 'us-east-1',
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    prefix: 'products/daily/',
-  },
-  fluent: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    username: process.env.FLUENT_USERNAME!,
-    password: process.env.FLUENT_PASSWORD!,
-    retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-};
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-// Initialize S3 Data Source
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_PARQUET',
-    connectionId: 's3-products',
-    name: 'Product Data Lake',
-    settings: {
-      ...config.s3,
-      filePattern: '*.parquet',
-    },
-  },
-  logger
-);
-
-// Initialize Parquet Parser
-const parquetParser = new ParquetParserService(logger);
-
-// Field Mapping with enrichment
-const fieldMapper = new UniversalMapper({
-  fields: {
-    ref: {
-      source: 'product_id',
-      resolver: value => `PROD-${value}`,
-    },
-    name: {
-      source: 'product_name',
-      resolver: 'sdk.trim',
-    },
-    type: {
-      source: 'category',
-      resolver: 'sdk.uppercase',
-    },
-    standardPrice: {
-      source: 'price',
-      resolver: value => ({
-        value: parseFloat(value),
-        currency: 'USD',
-      }),
-    },
-    'attributes.weight': {
-      source: 'weight',
-      resolver: value => ({
-        value: parseFloat(value),
-        unit: 'kg',
-      }),
-    },
-    'attributes.dimensions': {
-      source: 'dimensions',
-      resolver: value => {
-        const [l, w, h] = value.split('x').map(Number);
-        return {
-          length: l,
-          width: w,
-          height: h,
-          unit: 'cm',
-        };
-      },
-    },
-  },
-});
-
-// Create Fluent Client
-const fluentClient = await createClient({ config: config.fluent });
-
-async function syncProductCatalog() {
-  logger.info('Starting product catalog sync from S3 Parquet files');
-
-  try {
-    // List Parquet files from S3
-    const files = await s3Source.listFiles({
-      prefix: 'products/daily/',
-      filePattern: '*.parquet',
-    });
-
-    logger.info(`Found ${files.length} Parquet files`);
-
-    for (const file of files) {
-      logger.info(`Processing: ${file.name}`);
-
-      // Download Parquet file
-      const buffer = await s3Source.downloadFile(file.name);
-
-      // Parse Parquet data
-      const records = await parquetParser.parse(buffer, {
-        columns: ['product_id', 'product_name', 'category', 'price', 'weight', 'dimensions'],
-        maxRows: 10000,
-      });
-
-      logger.info(`Parsed ${records.length} products`);
-
-      // Transform records
-      const products = [];
-      for (const record of records) {
-        const result = await fieldMapper.map(record);
-        if (!result.success) {
-          logger.warn('Mapping errors', result.errors);
-          continue;
-        }
-
-        // Add additional enrichment
-        products.push({
-          ...result.data,
-          status: 'ACTIVE',
-          gtin: generateGTIN(result.data.ref),
-          catalogueRef: 'MAIN_CATALOG',
-          attributes: {
-            ...result.data.attributes,
-            source: 'S3_PARQUET',
-            lastUpdated: new Date().toISOString(),
-          },
-        });
-      }
-
-      // GraphQL mutation for product creation/update
-      const mutation = `
-        mutation UpsertProducts($products: [ProductInput]!) {
-          upsertProducts(input: { products: $products }) {
-            created
-            updated
-            failed
-            errors {
-              ref
-              message
-            }
-          }
-        }
-      `;
-
-      const result = await fluentClient.graphql({ query: mutation, variables: { products } });
-
-      logger.info('Product sync result:', result);
-
-      // Archive processed file
-      await s3Source.moveFile(
-        file.name,
-        `products/archive/${new Date().toISOString().split('T')[0]}/${file.name}`
-      );
-    }
-
-    logger.info('Product catalog sync completed');
-  } catch (error) {
-    logger.error('Product sync failed:', error);
-    throw error;
-  }
-}
-
-// Helper function for GTIN generation
-function generateGTIN(ref: string): string {
-  // Simple GTIN generation (in production, use proper algorithm)
-  const hash = ref.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0);
-  return `0${String(hash).padStart(13, '0')}`.slice(0, 14);
-}
-
-// Execute
-syncProductCatalog();
-```
-
-### Key Concepts: Parquet Processing
-
-**Why Parquet?**
-
-- Columnar format (efficient for analytics)
-- Compressed (smaller file sizes)
-- Typed data (enforces schema)
-- Fast reading (only read needed columns)
-
-**Reading Parquet:**
-
-```typescript
-const records = await parquetParser.parse(buffer, {
-  columns: ['product_id', 'product_name', 'price'], // Only read these columns
-  maxRows: 10000, // Limit for safety
-});
-```
-
-**Nested Field Mapping:**
-
-```typescript
-// Map flat field to nested structure
-'attributes.weight': {
-  source: 'weight',
-  resolver: (value) => ({
-    value: parseFloat(value),
-    unit: 'kg'
-  })
-}
-
-// Result: { attributes: { weight: { value: 10.5, unit: 'kg' } } }
-```
-
-## Scenario 3: SFTP JSON to Fluent Order Creation
-
-**Goal:** Process order JSON files from SFTP and create orders in Fluent Commerce.
-
-**Complexity:** Medium
-**Components:** SftpDataSource, JSONParserService, SchemaValidationService
-**Estimated Lines:** ~300 lines
-
-### Problem Statement
-
-Your e-commerce platform exports orders as JSON files to SFTP. Each order contains:
-
-- Order ID and customer details
-- Line items with SKUs and quantities
-- Shipping address
-- Pricing information
-
-You need to validate the structure and create orders in Fluent.
-
-### JSON Order Format
-
-```json
-{
-  "orderId": "ORD-12345",
-  "customer": {
-    "email": "customer@example.com",
-    "firstName": "John",
-    "lastName": "Doe",
-    "phone": "+1234567890"
-  },
-  "items": [
-    {
-      "sku": "PROD-001",
-      "quantity": 2,
-      "price": 29.99
-    }
-  ],
-  "total": 59.98,
-  "tax": 5.4,
-  "shippingAddress": {
-    "street": "123 Main St",
-    "city": "New York",
-    "postcode": "10001",
-    "country": "US"
-  }
-}
-```
-
-### Complete Implementation
-
-```typescript
-import {
-  createClient,
-  SftpDataSource,
-  SchemaValidationService,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-// SFTP Configuration for JSON files
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_JSON',
-    connectionId: 'orders-sftp',
-    name: 'Orders SFTP',
-    settings: {
-      host: process.env.SFTP_HOST!,
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD!,
-      remotePath: '/orders/new',
-      filePattern: 'ORDER_*.json',
-      jsonValidate: true,
-      jsonFormat: 'json',
-    },
-  },
-  logger
-);
-
-// Fluent Client
-const fluentClient = await createClient({
-  config: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    username: process.env.FLUENT_USERNAME!,
-    password: process.env.FLUENT_PASSWORD!,
-    retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-});
-
-// Custom validation function for order data
-// Note: SchemaValidationService validates GraphQL payloads against Fluent schema
-// For JSON data validation, use custom logic or third-party libraries
-function validateOrder(order: any): { valid: boolean; errors: string[] } {
-  const errors: string[] = [];
-
-  if (!order.orderId) errors.push('orderId is required');
-  if (!order.customer?.email) errors.push('customer.email is required');
-  if (!order.items || order.items.length === 0) errors.push('items array is required');
-  if (order.total === undefined) errors.push('total is required');
-
-  return { valid: errors.length === 0, errors };
-}
-
-// Initialize GraphQL schema validator (for validating mutations to Fluent)
-const schemaValidator = new SchemaValidationService(fluentClient);
-
-async function processOrders() {
-  logger.info('Starting order processing from SFTP');
-
-  try {
-    // List JSON order files
-    const files = await sftpSource.listFiles({
-      filePattern: 'ORDER_*.json',
-    });
-
-    logger.info(`Found ${files.length} order files`);
-
-    for (const file of files) {
-      logger.info(`Processing order file: ${file.name}`);
-
-      try {
-        // Download JSON file
-        const content = await sftpSource.downloadFile(file.name);
-
-        // Parse JSON
-        const orderData = JSON.parse(content.toString());
-
-        // Validate order data
-        const validationResult = validateOrder(orderData);
-        if (!validationResult.valid) {
-          logger.error('Order validation failed:', validationResult.errors);
-          throw new Error(`Invalid order data: ${validationResult.errors.join(', ')}`);
-        }
-
-        // Transform to Fluent order format
-        const fluentOrder = {
-          ref: orderData.orderId,
-          type: 'HD',
-          retailer: { id: process.env.FLUENT_RETAILER_ID },
-          customer: {
-            email: orderData.customer.email,
-            firstName: orderData.customer.firstName,
-            lastName: orderData.customer.lastName,
-            phone: orderData.customer.phone,
-          },
-          fulfilmentChoice: {
-            deliveryType: 'STANDARD',
-            deliveryAddress: {
-              street: orderData.shippingAddress.street,
-              city: orderData.shippingAddress.city,
-              postcode: orderData.shippingAddress.postcode,
-              state: orderData.shippingAddress.state || '',
-              country: orderData.shippingAddress.country,
-            },
-          },
-          items: orderData.items.map((item, index) => ({
-            ref: `${orderData.orderId}-${index + 1}`,
-            productRef: item.sku,
-            quantity: item.quantity,
-            price: item.price,
-            totalPrice: item.quantity * item.price,
-            currency: 'USD',
-          })),
-          totalPrice: orderData.total,
-          totalTaxPrice: orderData.tax || 0,
-          attributes: {
-            source: 'SFTP_JSON',
-            originalFile: file.name,
-            processedAt: new Date().toISOString(),
-          },
-        };
-
-        // Direct GraphQL mutation
-        const mutation = `
-          mutation CreateOrder($input: CreateOrderInput!) {
-            createOrder(input: $input) {
-              id
-              ref
-              status
-              createdOn
-            }
-          }
-        `;
-
-        const mutationResult = await fluentClient.graphql({
-          query: mutation,
-          variables: { input: fluentOrder },
-        });
-
-        logger.info('Order created via mutation:', mutationResult);
-
-        // Archive processed file
-        await sftpSource.moveFile(
-          `/orders/new/${file.name}`,
-          `/orders/processed/${file.name}`,
-          false
-        );
-      } catch (error) {
-        logger.error(`Failed to process order ${file.name}:`, error);
-
-        // Move to error folder
-        await sftpSource.moveFile(`/orders/new/${file.name}`, `/orders/failed/${file.name}`, true);
-
-        // Create error report
-        const errorReport = {
-          file: file.name,
-          error: (error as Error).message,
-          timestamp: new Date().toISOString(),
-        };
-
-        await sftpSource.uploadFile(
-          `/orders/errors/${file.name}.error.json`,
-          JSON.stringify(errorReport)
-        );
-      }
-    }
-
-    logger.info('Order processing completed');
-  } catch (error) {
-    logger.error('Order processing failed:', error);
-    throw error;
-  }
-}
-
-// Execute
-processOrders();
-```
-
-### Key Concepts: JSON Processing and Validation
-
-**Schema-Based Validation:**
-
-```typescript
-const orderSchema = {
-  type: 'object',
-  required: ['orderId', 'customer', 'items'],
-  properties: {
-    orderId: { type: 'string' },
-    items: {
-      type: 'array',
-      minItems: 1,
-      items: {
-        required: ['sku', 'quantity'],
-        properties: {
-          sku: { type: 'string' },
-          quantity: { type: 'number', minimum: 1 },
-        },
-      },
-    },
-  },
-};
-
-// Custom validation function (JSON schema validation not built into SDK)
-function validateOrder(data: any) {
-  const errors: string[] = [];
-  if (!data.orderId) errors.push('orderId required');
-  if (!data.items?.length) errors.push('items array required');
-  return { valid: errors.length === 0, errors };
-}
-
-const result = validateOrder(orderData);
-if (!result.valid) {
-  console.error('Validation errors:', result.errors);
-}
-```
-
-**Nested Data Transformation:**
-
-```typescript
-// Transform nested items array
-items: orderData.items.map((item, index) => ({
-  ref: `${orderData.orderId}-${index + 1}`, // Generate item ref
-  productRef: item.sku,
-  quantity: item.quantity,
-  price: item.price,
-  totalPrice: item.quantity * item.price, // Calculate total
-  currency: 'USD',
-}));
-```
-
-**Error Reporting:**
-
-```typescript
-catch (error) {
-  // Create detailed error report
-  const errorReport = {
-    file: file.name,
-    error: error.message,
-    timestamp: new Date().toISOString(),
-    orderData: content.toString()  // Include original data
-  };
-
-  // Save error report for investigation
-  await sftpSource.uploadFile(
-    `/orders/errors/${file.name}.error.json`,
-    JSON.stringify(errorReport)
-  );
-}
-```
-
-## Comparison: Batch API vs GraphQL Mutations
-
-| Feature               | Batch API                     | GraphQL Mutations                          |
-| --------------------- | ----------------------------- | ------------------------------------------ |
-| **Best For**          | High-volume inventory updates | Complex entity creation (orders, products) |
-| **Entity Types**      | INVENTORY only                | All entity types                           |
-| **Batch Size**        | Up to 10,000 records          | Typically 50-100 at a time                 |
-| **Field Flexibility** | Limited fields                | Full entity schema                         |
-| **Validation**        | Server-side                   | Schema validation                          |
-| **Performance**       | Optimized for bulk            | Better for complex data                    |
-| **Error Handling**    | Batch-level errors            | Detailed field-level errors                |
-
-## Practice Exercise
-
-**Challenge:** Add data enrichment to the product sync
-
-**Requirements:**
-
-1. Calculate product availability from external API
-2. Add image URLs from CDN (based on product ID)
-3. Add category hierarchy lookup
-4. Include custom attributes for searchability
-
-**Solution approach:**
-
-```typescript
-// Enrichment function
-async function enrichProduct(product: any) {
-  // 1. Fetch availability from external API
-  const availability = await fetchAvailability(product.ref);
-
-  // 2. Generate CDN image URLs
-  const images = [
-    `https://cdn.example.com/products/${product.ref}/main.jpg`,
-    `https://cdn.example.com/products/${product.ref}/thumb.jpg`,
-  ];
-
-  // 3. Lookup category hierarchy
-  const categoryPath = await lookupCategoryHierarchy(product.type);
-
-  return {
-    ...product,
-    availability,
-    images,
-    categories: categoryPath,
-    attributes: {
-      ...product.attributes,
-      searchable: true,
-      enrichedAt: new Date().toISOString(),
-    },
-  };
-}
-
-// Apply enrichment
-const enrichedProducts = await Promise.all(products.map(p => enrichProduct(p)));
-```
-
-## Key Takeaways
-
-- 🎯 Parquet format is efficient for large datasets (columnar, compressed)
-- 🎯 JSON Schema validation catches structural errors early
-- 🎯 Nested data transformations require careful mapping
-- 🎯 GraphQL mutations provide full flexibility for complex entities
-- 🎯 Enrichment adds value (calculated fields, external data)
-- 🎯 Error reports should include context for debugging
-
-## Next Steps
-
-Continue to [Module 4: Advanced Scenarios](./connector-scenarios-04-advanced-scenarios.md) to learn state management, multi-source aggregation, and streaming patterns.
+# Module 3: Intermediate Scenarios
+
+**Level:** Intermediate
+**Estimated Time:** 45 minutes
+
+## Overview
+
+This module covers intermediate connector scenarios including S3 Parquet processing, JSON order creation, and advanced validation patterns. You'll learn how to handle different file formats and more complex data structures.
+
+## Learning Objectives
+
+By the end of this module, you will:
+
+- ✅ Build S3 Parquet to Fluent product catalog sync
+- ✅ Process SFTP JSON files for order creation
+- ✅ Implement schema-based validation
+- ✅ Handle nested data transformations
+- ✅ Use GraphQL mutations for complex entity creation
+- ✅ Implement proper enrichment and data augmentation
+
+## Scenario 2: S3 Parquet to Fluent Product Catalog
+
+**Goal:** Read product data from Parquet files in S3, enrich with additional data, and update Fluent product catalog.
+
+**Complexity:** Medium
+**Components:** S3DataSource, ParquetParserService, UniversalMapper, FluentClient
+**Estimated Lines:** ~200 lines
+
+### Problem Statement
+
+Your data warehouse exports product catalog as Parquet files to S3 daily. You need to:
+
+1. Read Parquet files from S3
+2. Parse columnar format efficiently
+3. Enrich with calculated fields
+4. Transform nested attributes
+5. Send to Fluent as product updates
+
+### Complete Implementation
+
+```typescript
+import {
+  createClient,
+  S3DataSource,
+  ParquetParserService,
+  UniversalMapper,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Configuration
+const config = {
+  s3: {
+    bucketName: 'product-data-lake',
+    region: 'us-east-1',
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    prefix: 'products/daily/',
+  },
+  fluent: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    username: process.env.FLUENT_USERNAME!,
+    password: process.env.FLUENT_PASSWORD!,
+    retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+};
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+// Initialize S3 Data Source
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_PARQUET',
+    connectionId: 's3-products',
+    name: 'Product Data Lake',
+    settings: {
+      ...config.s3,
+      filePattern: '*.parquet',
+    },
+  },
+  logger
+);
+
+// Initialize Parquet Parser
+const parquetParser = new ParquetParserService(logger);
+
+// Field Mapping with enrichment
+const fieldMapper = new UniversalMapper({
+  fields: {
+    ref: {
+      source: 'product_id',
+      resolver: value => `PROD-${value}`,
+    },
+    name: {
+      source: 'product_name',
+      resolver: 'sdk.trim',
+    },
+    type: {
+      source: 'category',
+      resolver: 'sdk.uppercase',
+    },
+    standardPrice: {
+      source: 'price',
+      resolver: value => ({
+        value: parseFloat(value),
+        currency: 'USD',
+      }),
+    },
+    'attributes.weight': {
+      source: 'weight',
+      resolver: value => ({
+        value: parseFloat(value),
+        unit: 'kg',
+      }),
+    },
+    'attributes.dimensions': {
+      source: 'dimensions',
+      resolver: value => {
+        const [l, w, h] = value.split('x').map(Number);
+        return {
+          length: l,
+          width: w,
+          height: h,
+          unit: 'cm',
+        };
+      },
+    },
+  },
+});
+
+// Create Fluent Client
+const fluentClient = await createClient({ config: config.fluent });
+
+async function syncProductCatalog() {
+  logger.info('Starting product catalog sync from S3 Parquet files');
+
+  try {
+    // List Parquet files from S3
+    const files = await s3Source.listFiles({
+      prefix: 'products/daily/',
+      filePattern: '*.parquet',
+    });
+
+    logger.info(`Found ${files.length} Parquet files`);
+
+    for (const file of files) {
+      logger.info(`Processing: ${file.name}`);
+
+      // Download Parquet file
+      const buffer = await s3Source.downloadFile(file.name);
+
+      // Parse Parquet data
+      const records = await parquetParser.parse(buffer, {
+        columns: ['product_id', 'product_name', 'category', 'price', 'weight', 'dimensions'],
+        maxRows: 10000,
+      });
+
+      logger.info(`Parsed ${records.length} products`);
+
+      // Transform records
+      const products = [];
+      for (const record of records) {
+        const result = await fieldMapper.map(record);
+        if (!result.success) {
+          logger.warn('Mapping errors', result.errors);
+          continue;
+        }
+
+        // Add additional enrichment
+        products.push({
+          ...result.data,
+          status: 'ACTIVE',
+          gtin: generateGTIN(result.data.ref),
+          catalogueRef: 'MAIN_CATALOG',
+          attributes: {
+            ...result.data.attributes,
+            source: 'S3_PARQUET',
+            lastUpdated: new Date().toISOString(),
+          },
+        });
+      }
+
+      // GraphQL mutation for product creation/update
+      const mutation = `
+        mutation UpsertProducts($products: [ProductInput]!) {
+          upsertProducts(input: { products: $products }) {
+            created
+            updated
+            failed
+            errors {
+              ref
+              message
+            }
+          }
+        }
+      `;
+
+      const result = await fluentClient.graphql({ query: mutation, variables: { products } });
+
+      logger.info('Product sync result:', result);
+
+      // Archive processed file
+      await s3Source.moveFile(
+        file.name,
+        `products/archive/${new Date().toISOString().split('T')[0]}/${file.name}`
+      );
+    }
+
+    logger.info('Product catalog sync completed');
+  } catch (error) {
+    logger.error('Product sync failed:', error);
+    throw error;
+  }
+}
+
+// Helper function for GTIN generation
+function generateGTIN(ref: string): string {
+  // Simple GTIN generation (in production, use proper algorithm)
+  const hash = ref.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0);
+  return `0${String(hash).padStart(13, '0')}`.slice(0, 14);
+}
+
+// Execute
+syncProductCatalog();
+```
+
+### Key Concepts: Parquet Processing
+
+**Why Parquet?**
+
+- Columnar format (efficient for analytics)
+- Compressed (smaller file sizes)
+- Typed data (enforces schema)
+- Fast reading (only read needed columns)
+
+**Reading Parquet:**
+
+```typescript
+const records = await parquetParser.parse(buffer, {
+  columns: ['product_id', 'product_name', 'price'], // Only read these columns
+  maxRows: 10000, // Limit for safety
+});
+```
+
+**Nested Field Mapping:**
+
+```typescript
+// Map flat field to nested structure
+'attributes.weight': {
+  source: 'weight',
+  resolver: (value) => ({
+    value: parseFloat(value),
+    unit: 'kg'
+  })
+}
+
+// Result: { attributes: { weight: { value: 10.5, unit: 'kg' } } }
+```
+
+## Scenario 3: SFTP JSON to Fluent Order Creation
+
+**Goal:** Process order JSON files from SFTP and create orders in Fluent Commerce.
+
+**Complexity:** Medium
+**Components:** SftpDataSource, JSONParserService, SchemaValidationService
+**Estimated Lines:** ~300 lines
+
+### Problem Statement
+
+Your e-commerce platform exports orders as JSON files to SFTP. Each order contains:
+
+- Order ID and customer details
+- Line items with SKUs and quantities
+- Shipping address
+- Pricing information
+
+You need to validate the structure and create orders in Fluent.
+
+### JSON Order Format
+
+```json
+{
+  "orderId": "ORD-12345",
+  "customer": {
+    "email": "customer@example.com",
+    "firstName": "John",
+    "lastName": "Doe",
+    "phone": "+1234567890"
+  },
+  "items": [
+    {
+      "sku": "PROD-001",
+      "quantity": 2,
+      "price": 29.99
+    }
+  ],
+  "total": 59.98,
+  "tax": 5.4,
+  "shippingAddress": {
+    "street": "123 Main St",
+    "city": "New York",
+    "postcode": "10001",
+    "country": "US"
+  }
+}
+```
+
+### Complete Implementation
+
+```typescript
+import {
+  createClient,
+  SftpDataSource,
+  SchemaValidationService,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+// SFTP Configuration for JSON files
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_JSON',
+    connectionId: 'orders-sftp',
+    name: 'Orders SFTP',
+    settings: {
+      host: process.env.SFTP_HOST!,
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD!,
+      remotePath: '/orders/new',
+      filePattern: 'ORDER_*.json',
+      jsonValidate: true,
+      jsonFormat: 'json',
+    },
+  },
+  logger
+);
+
+// Fluent Client
+const fluentClient = await createClient({
+  config: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    username: process.env.FLUENT_USERNAME!,
+    password: process.env.FLUENT_PASSWORD!,
+    retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+});
+
+// Custom validation function for order data
+// Note: SchemaValidationService validates GraphQL payloads against Fluent schema
+// For JSON data validation, use custom logic or third-party libraries
+function validateOrder(order: any): { valid: boolean; errors: string[] } {
+  const errors: string[] = [];
+
+  if (!order.orderId) errors.push('orderId is required');
+  if (!order.customer?.email) errors.push('customer.email is required');
+  if (!order.items || order.items.length === 0) errors.push('items array is required');
+  if (order.total === undefined) errors.push('total is required');
+
+  return { valid: errors.length === 0, errors };
+}
+
+// Initialize GraphQL schema validator (for validating mutations to Fluent)
+const schemaValidator = new SchemaValidationService(fluentClient);
+
+async function processOrders() {
+  logger.info('Starting order processing from SFTP');
+
+  try {
+    // List JSON order files
+    const files = await sftpSource.listFiles({
+      filePattern: 'ORDER_*.json',
+    });
+
+    logger.info(`Found ${files.length} order files`);
+
+    for (const file of files) {
+      logger.info(`Processing order file: ${file.name}`);
+
+      try {
+        // Download JSON file
+        const content = await sftpSource.downloadFile(file.name);
+
+        // Parse JSON
+        const orderData = JSON.parse(content.toString());
+
+        // Validate order data
+        const validationResult = validateOrder(orderData);
+        if (!validationResult.valid) {
+          logger.error('Order validation failed:', validationResult.errors);
+          throw new Error(`Invalid order data: ${validationResult.errors.join(', ')}`);
+        }
+
+        // Transform to Fluent order format
+        const fluentOrder = {
+          ref: orderData.orderId,
+          type: 'HD',
+          retailer: { id: process.env.FLUENT_RETAILER_ID },
+          customer: {
+            email: orderData.customer.email,
+            firstName: orderData.customer.firstName,
+            lastName: orderData.customer.lastName,
+            phone: orderData.customer.phone,
+          },
+          fulfilmentChoice: {
+            deliveryType: 'STANDARD',
+            deliveryAddress: {
+              street: orderData.shippingAddress.street,
+              city: orderData.shippingAddress.city,
+              postcode: orderData.shippingAddress.postcode,
+              state: orderData.shippingAddress.state || '',
+              country: orderData.shippingAddress.country,
+            },
+          },
+          items: orderData.items.map((item, index) => ({
+            ref: `${orderData.orderId}-${index + 1}`,
+            productRef: item.sku,
+            quantity: item.quantity,
+            price: item.price,
+            totalPrice: item.quantity * item.price,
+            currency: 'USD',
+          })),
+          totalPrice: orderData.total,
+          totalTaxPrice: orderData.tax || 0,
+          attributes: {
+            source: 'SFTP_JSON',
+            originalFile: file.name,
+            processedAt: new Date().toISOString(),
+          },
+        };
+
+        // Direct GraphQL mutation
+        const mutation = `
+          mutation CreateOrder($input: CreateOrderInput!) {
+            createOrder(input: $input) {
+              id
+              ref
+              status
+              createdOn
+            }
+          }
+        `;
+
+        const mutationResult = await fluentClient.graphql({
+          query: mutation,
+          variables: { input: fluentOrder },
+        });
+
+        logger.info('Order created via mutation:', mutationResult);
+
+        // Archive processed file
+        await sftpSource.moveFile(
+          `/orders/new/${file.name}`,
+          `/orders/processed/${file.name}`,
+          false
+        );
+      } catch (error) {
+        logger.error(`Failed to process order ${file.name}:`, error);
+
+        // Move to error folder
+        await sftpSource.moveFile(`/orders/new/${file.name}`, `/orders/failed/${file.name}`, true);
+
+        // Create error report
+        const errorReport = {
+          file: file.name,
+          error: (error as Error).message,
+          timestamp: new Date().toISOString(),
+        };
+
+        await sftpSource.uploadFile(
+          `/orders/errors/${file.name}.error.json`,
+          JSON.stringify(errorReport)
+        );
+      }
+    }
+
+    logger.info('Order processing completed');
+  } catch (error) {
+    logger.error('Order processing failed:', error);
+    throw error;
+  }
+}
+
+// Execute
+processOrders();
+```
+
+### Key Concepts: JSON Processing and Validation
+
+**Schema-Based Validation:**
+
+```typescript
+const orderSchema = {
+  type: 'object',
+  required: ['orderId', 'customer', 'items'],
+  properties: {
+    orderId: { type: 'string' },
+    items: {
+      type: 'array',
+      minItems: 1,
+      items: {
+        required: ['sku', 'quantity'],
+        properties: {
+          sku: { type: 'string' },
+          quantity: { type: 'number', minimum: 1 },
+        },
+      },
+    },
+  },
+};
+
+// Custom validation function (JSON schema validation not built into SDK)
+function validateOrder(data: any) {
+  const errors: string[] = [];
+  if (!data.orderId) errors.push('orderId required');
+  if (!data.items?.length) errors.push('items array required');
+  return { valid: errors.length === 0, errors };
+}
+
+const result = validateOrder(orderData);
+if (!result.valid) {
+  console.error('Validation errors:', result.errors);
+}
+```
+
+**Nested Data Transformation:**
+
+```typescript
+// Transform nested items array
+items: orderData.items.map((item, index) => ({
+  ref: `${orderData.orderId}-${index + 1}`, // Generate item ref
+  productRef: item.sku,
+  quantity: item.quantity,
+  price: item.price,
+  totalPrice: item.quantity * item.price, // Calculate total
+  currency: 'USD',
+}));
+```
+
+**Error Reporting:**
+
+```typescript
+catch (error) {
+  // Create detailed error report
+  const errorReport = {
+    file: file.name,
+    error: error.message,
+    timestamp: new Date().toISOString(),
+    orderData: content.toString()  // Include original data
+  };
+
+  // Save error report for investigation
+  await sftpSource.uploadFile(
+    `/orders/errors/${file.name}.error.json`,
+    JSON.stringify(errorReport)
+  );
+}
+```
+
+## Comparison: Batch API vs GraphQL Mutations
+
+| Feature               | Batch API                     | GraphQL Mutations                          |
+| --------------------- | ----------------------------- | ------------------------------------------ |
+| **Best For**          | High-volume inventory updates | Complex entity creation (orders, products) |
+| **Entity Types**      | INVENTORY only                | All entity types                           |
+| **Batch Size**        | Up to 10,000 records          | Typically 50-100 at a time                 |
+| **Field Flexibility** | Limited fields                | Full entity schema                         |
+| **Validation**        | Server-side                   | Schema validation                          |
+| **Performance**       | Optimized for bulk            | Better for complex data                    |
+| **Error Handling**    | Batch-level errors            | Detailed field-level errors                |
+
+## Practice Exercise
+
+**Challenge:** Add data enrichment to the product sync
+
+**Requirements:**
+
+1. Calculate product availability from external API
+2. Add image URLs from CDN (based on product ID)
+3. Add category hierarchy lookup
+4. Include custom attributes for searchability
+
+**Solution approach:**
+
+```typescript
+// Enrichment function
+async function enrichProduct(product: any) {
+  // 1. Fetch availability from external API
+  const availability = await fetchAvailability(product.ref);
+
+  // 2. Generate CDN image URLs
+  const images = [
+    `https://cdn.example.com/products/${product.ref}/main.jpg`,
+    `https://cdn.example.com/products/${product.ref}/thumb.jpg`,
+  ];
+
+  // 3. Lookup category hierarchy
+  const categoryPath = await lookupCategoryHierarchy(product.type);
+
+  return {
+    ...product,
+    availability,
+    images,
+    categories: categoryPath,
+    attributes: {
+      ...product.attributes,
+      searchable: true,
+      enrichedAt: new Date().toISOString(),
+    },
+  };
+}
+
+// Apply enrichment
+const enrichedProducts = await Promise.all(products.map(p => enrichProduct(p)));
+```
+
+## Key Takeaways
+
+- 🎯 Parquet format is efficient for large datasets (columnar, compressed)
+- 🎯 JSON Schema validation catches structural errors early
+- 🎯 Nested data transformations require careful mapping
+- 🎯 GraphQL mutations provide full flexibility for complex entities
+- 🎯 Enrichment adds value (calculated fields, external data)
+- 🎯 Error reports should include context for debugging
+
+## Next Steps
+
+Continue to [Module 4: Advanced Scenarios](./connector-scenarios-04-advanced-scenarios.md) to learn state management, multi-source aggregation, and streaming patterns.