@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md

@@ -1,603 +1,603 @@
-# Module 5: Parquet Parser
-
-**Level:** Advanced
-**Estimated Time:** 20 minutes
-
-## Overview
-
-This module covers the Parquet Parser (`ParquetParserService`), designed for efficient large-scale data extraction and analytics. You'll learn how to parse Parquet files, leverage columnar storage, handle compression, and integrate with S3 for data extraction workflows.
-
-## Learning Objectives
-
-By the end of this module, you will:
-- ✅ Understand Parquet columnar format and its benefits
-- ✅ Parse Parquet files into JavaScript objects
-- ✅ Stream large Parquet files efficiently
-- ✅ Leverage compression and columnar storage
-- ✅ Extract Fluent data to Parquet for analytics
-- ✅ Integrate with S3 extraction workflows
-
-## What is Parquet?
-
-**Apache Parquet** is a columnar storage format that:
-- Stores data by column instead of by row
-- Uses efficient compression algorithms
-- Optimizes for analytical queries
-- Reduces file size significantly (often 10:1 compression)
-- Is the standard for big data and analytics
-
-**Row-based vs Column-based storage:**
-
-```
-Row-based (CSV):
-SKU001,100,WH-01
-SKU002,50,WH-01
-SKU003,75,WH-02
-
-Column-based (Parquet):
-Column[sku]:      [SKU001, SKU002, SKU003]
-Column[qty]:      [100, 50, 75]
-Column[location]: [WH-01, WH-01, WH-02]
-```
-
-**Benefits:**
-- ✅ Better compression (similar values in a column compress well)
-- ✅ Faster column-specific queries
-- ✅ Smaller file sizes
-- ✅ Efficient for analytics
-
-## Basic Parquet Parsing
-
-### Simple Parsing
-
-```typescript
-import { ParquetParserService, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
-import * as fs from 'fs/promises';
-
-// ParquetParserService requires a StructuredLogger (not a simple logger)
-const consoleLogger = createConsoleLogger();
-const logger = toStructuredLogger(consoleLogger, { service: 'ParquetParser' });
-
-// CRITICAL: Constructor requires logger parameter (NOT optional)
-const parquetParser = new ParquetParserService(logger);
-
-// Read Parquet file
-const fileBuffer = await fs.readFile('inventory.parquet');
-
-// Convert to ArrayBuffer (parseSimple requires ArrayBuffer, not Buffer)
-const arrayBuffer = fileBuffer.buffer.slice(
-  fileBuffer.byteOffset,
-  fileBuffer.byteOffset + fileBuffer.byteLength
-);
-
-// Parse Parquet (simple mode - easiest)
-const records = await parquetParser.parseSimple(arrayBuffer, 'inventory.parquet');
-
-console.log(`Parsed ${records.length} records`);
-console.log('First record:', records[0]);
-// { sku: 'SKU001', qty: 100, location: 'WH-01' }
-```
-
-### Type Safety with TypeScript
-
-```typescript
-// Define your record structure
-interface InventoryRecord {
-  sku: string;
-  qty: number;
-  location: string;
-  updatedAt: Date;
-}
-
-// Parse with type assertion (simple mode)
-const records = await parquetParser.parseSimple(arrayBuffer, 'inventory.parquet');
-
-// Or use parse() for detailed stats
-const result = await parquetParser.parse(arrayBuffer, 'inventory.parquet');
-console.log(`Processed ${result.stats.totalRecords} records in ${result.stats.processingTimeMs}ms`);
-
-// Full type safety
-result.records.forEach(record => {
-  console.log(`${record.sku}: ${record.qty} at ${record.location}`);
-});
-```
-
-## Streaming Large Files
-
-For large Parquet files, use streaming to avoid memory issues:
-
-### Basic Streaming
-
-```typescript
-import { ParquetParserService, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
-
-const consoleLogger = createConsoleLogger();
-const logger = toStructuredLogger(consoleLogger, { service: 'ParquetParser' });
-
-const parquetParser = new ParquetParserService(logger);
-
-// Stream Parquet file
-const fileBuffer = await fs.readFile('large-inventory.parquet');
-
-// Convert to ArrayBuffer
-const arrayBuffer = fileBuffer.buffer.slice(
-  fileBuffer.byteOffset,
-  fileBuffer.byteOffset + fileBuffer.byteLength
-);
-
-// Process records in batches
-const batchSize = 1000;
-
-const streamingResult = await parquetParser.parseStreaming(
-  arrayBuffer,
-  'large-inventory.parquet',
-  { batchSize }
-);
-
-for await (const batch of streamingResult.records) {
-  console.log(`Processing batch of ${batch.length} records`);
-  await processBatch(batch);
-}
-
-// Get final stats
-const stats = await streamingResult.stats;
-console.log(`Processed ${stats.totalRecords} total records`);
-
-async function processBatch(records: any[]) {
-  // Transform and process
-  console.log(`Batch processing ${records.length} records`);
-}
-```
-
-### Streaming with Progress Tracking
-
-```typescript
-async function streamWithProgress(parquetBuffer: ArrayBuffer) {
-  const logger = createConsoleLogger();
-  const parquetParser = new ParquetParserService(logger);
-  const startTime = Date.now();
-  let totalRecords = 0;
-
-  const streamingResult = await parquetParser.parseStreaming(
-    parquetBuffer,
-    'file.parquet',
-    { batchSize: 1000 }
-  );
-
-  for await (const batch of streamingResult.records) {
-    totalRecords += batch.length;
-
-    // Process batch
-    await processBatch(batch);
-
-    // Report progress
-    const elapsed = (Date.now() - startTime) / 1000;
-    const rate = totalRecords / elapsed;
-    console.log(`Processed ${totalRecords} records (${rate.toFixed(2)} records/sec)`);
-  }
-
-  const stats = await streamingResult.stats;
-  console.log(`Completed: ${stats.totalRecords} total records`);
-}
-```
-
-## Parser Configuration
-
-The Parquet parser supports various options:
-
-```typescript
-interface ParquetParsingOptions {
-  // Batch size for streaming (default: 1000)
-  batchSize?: number;
-
-  // Maximum file size in bytes (default: 100MB)
-  maxFileSize?: number;
-
-  // Enable streaming mode (default: true)
-  enableStreaming?: boolean;
-
-  // Validation options
-  validation?: {
-    enabled?: boolean;
-    requiredColumns?: string[];
-    columnTypes?: Record<string, ParquetDataType>;
-    allowAdditionalColumns?: boolean;
-  };
-}
-```
-
-### Example Configuration
-
-```typescript
-const options: ParquetParsingOptions = {
-  batchSize: 5000,
-  maxFileSize: 500 * 1024 * 1024, // 500MB
-  enableStreaming: true,
-  validation: {
-    enabled: true,
-    requiredColumns: ['sku', 'qty', 'location'],
-    columnTypes: {
-      sku: 'string',
-      qty: 'number',
-      location: 'string'
-    }
-  }
-};
-
-const result = await parquetParser.parse(parquetBuffer, 'file.parquet', options);
-const records = result.records;
-```
-
-## Data Extraction Workflow
-
-The primary use case for Parquet is extracting large datasets from Fluent Commerce to S3.
-
-### Complete Extraction Example
-
-```typescript
-import {
-  createClient,
-  ParquetParserService,
-  S3DataSource,
-  createConsoleLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function extractInventoryToParquet() {
-  // 1. Setup logger
-  const logger = createConsoleLogger();
-
-  // 2. Create client
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-
-  // 3. Fetch data from Fluent (with auto-pagination)
-  const result = await client.graphql({
-    query: `
-      query GetVirtualPositions($first: Int!, $after: String) {
-        virtualPositions(first: $first, after: $after) {
-          edges {
-            node {
-              id
-              productRef
-              quantity
-              groupRef
-              createdOn
-              updatedOn
-            }
-            cursor
-          }
-          pageInfo {
-            hasNextPage
-          }
-        }
-      }
-    `,
-    variables: { first: 1000 },
-    pagination: { maxPages: 100 }
-  });
-
-  // 4. Transform to flat structure
-  const records = result.data.virtualPositions.edges.map(edge => ({
-    id: edge.node.id,
-    productRef: edge.node.productRef,
-    quantity: edge.node.quantity,
-    groupRef: edge.node.groupRef,
-    createdOn: edge.node.createdOn,
-    updatedOn: edge.node.updatedOn
-  }));
-
-  console.log(`Extracted ${records.length} virtual positions`);
-
-  // 5. Convert to Parquet (using parquetjs or hyparquet)
-  const parquetBuffer = await convertToParquet(records);
-
-  // 6. Upload to S3 using S3DataSource
-  const s3DataSource = new S3DataSource({
-    type: 'S3_PARQUET',
-    connectionId: 's3-parquet-extract',
-    name: 'S3 Parquet Extract',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION!,
-      bucket: 'my-analytics-bucket'
-    }
-  }, logger);
-
-  await s3DataSource.uploadFile(
-    `extracts/virtual-positions-${new Date().toISOString()}.parquet`,
-    parquetBuffer,
-    { contentType: 'application/octet-stream' }
-  );
-
-  console.log('✅ Extraction complete!');
-}
-
-async function convertToParquet(records: any[]): Promise<Buffer> {
-  // Use parquetjs or hyparquet to convert
-  // Implementation depends on your choice of Parquet library
-  const parquetWriter = /* ... */;
-  // Write records and return buffer
-  return buffer;
-}
-```
-
-## Performance Metrics
-
-Get performance statistics from parsing operations:
-
-```typescript
-const logger = createConsoleLogger();
-const parquetParser = new ParquetParserService(logger);
-
-// Parse file
-const result = await parquetParser.parse(parquetBuffer, 'file.parquet');
-
-// Access parsing statistics
-console.log('Parsing statistics:');
-console.log(`Total records: ${result.stats.totalRecords}`);
-console.log(`Processing time: ${result.stats.processingTimeMs}ms`);
-console.log(`Peak memory usage: ${(result.stats.peakMemoryUsage / 1024 / 1024).toFixed(2)}MB`);
-console.log(`Records per second: ${result.stats.recordsPerSecond}`);
-console.log(`Batches processed: ${result.stats.batchesProcessed}`);
-
-// Get overall performance metrics
-const metrics = parquetParser.getPerformanceMetrics();
-console.log('Overall metrics:', metrics);
-```
-
-## Validation
-
-Validate Parquet structure:
-
-```typescript
-const logger = createConsoleLogger();
-const parquetParser = new ParquetParserService(logger);
-
-// Validate Parquet file
-const validation = await parquetParser.validateFile(parquetBuffer, 'file.parquet');
-
-if (validation.isValid) {
-  console.log('Parquet file is valid');
-  const records = await parquetParser.parseSimple(parquetBuffer, 'file.parquet');
-} else {
-  console.error('Validation errors:', validation.errors);
-}
-```
-
-## Integration with S3
-
-### Read Parquet from S3
-
-```typescript
-import { S3DataSource, ParquetParserService, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function readParquetFromS3() {
-  const logger = createConsoleLogger();
-
-  // Initialize S3 data source
-  const s3DataSource = new S3DataSource({
-    type: 'S3_PARQUET',
-    connectionId: 's3-parquet-read',
-    name: 'S3 Parquet Read',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION!,
-      bucket: 'my-data-bucket'
-    }
-  }, logger);
-
-  // Download Parquet file from S3
-  const parquetBuffer = await s3DataSource.downloadFile('extracts/inventory.parquet');
-
-  // Parse Parquet
-  const parquetParser = new ParquetParserService(logger);
-  const records = await parquetParser.parseSimple(parquetBuffer, 'inventory.parquet');
-
-  console.log(`Loaded ${records.length} records from S3`);
-  return records;
-}
-```
-
-### Write Parquet to S3
-
-```typescript
-async function writeParquetToS3(records: any[]) {
-  const logger = createConsoleLogger();
-
-  // Convert to Parquet
-  const parquetBuffer = await convertToParquet(records);
-
-  // Upload to S3 using S3DataSource
-  const s3DataSource = new S3DataSource({
-    type: 'S3_PARQUET',
-    connectionId: 's3-parquet-write',
-    name: 'S3 Parquet Write',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION!,
-      bucket: 'my-analytics-bucket'
-    }
-  }, logger);
-
-  await s3DataSource.uploadFile(
-    `extracts/data-${Date.now()}.parquet`,
-    parquetBuffer,
-    { contentType: 'application/octet-stream' }
-  );
-
-  console.log('✅ Parquet file uploaded to S3');
-}
-```
-
-## Performance Optimization
-
-### 1. Use Appropriate Batch Sizes
-
-```typescript
-// Small batches for memory-constrained environments
-const smallBatch = await parquetParser.parseStreaming(parquetBuffer, {
-  batchSize: 500
-});
-
-// Large batches for high-memory environments
-const largeBatch = await parquetParser.parseStreaming(parquetBuffer, {
-  batchSize: 10000
-});
-```
-
-### 2. Leverage Compression
-
-Parquet automatically compresses data, but you can optimize:
-
-```typescript
-// When creating Parquet files, use optimal compression
-const parquetOptions = {
-  compression: 'SNAPPY', // Fast compression/decompression
-  // or 'GZIP' for better compression ratio
-};
-```
-
-### 3. Column Pruning
-
-Only read columns you need (when using advanced Parquet libraries):
-
-```typescript
-// Only read specific columns
-const selectedColumns = await parquetParser.parse(parquetBuffer, {
-  columns: ['sku', 'qty'] // Only read these columns
-});
-```
-
-## Real-World Example: Analytics Pipeline
-
-Complete pipeline for extracting Fluent data to analytics warehouse:
-
-```typescript
-import {
-  createClient,
-  ParquetParserService,
-  S3DataSource,
-  createConsoleLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function dailyInventoryExtract() {
-  const logger = createConsoleLogger();
-  const client = await createClient({ config });
-
-  const s3DataSource = new S3DataSource({
-    type: 'S3_PARQUET',
-    connectionId: 's3-analytics',
-    name: 'S3 Analytics',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION!,
-      bucket: 'analytics-warehouse'
-    }
-  }, logger);
-
-  // 1. Extract inventory from Fluent
-  const result = await client.graphql({
-    query: inventoryQuery,
-    variables: { first: 1000 },
-    pagination: { maxRecords: 1000000 }
-  });
-
-  const records = result.data.inventoryPositions.edges.map(e => e.node);
-
-  console.log(`Extracted ${records.length} inventory positions`);
-
-  // 2. Convert to Parquet
-  const parquetBuffer = await convertToParquet(records);
-
-  // 3. Upload to S3 analytics bucket
-  const timestamp = new Date().toISOString().split('T')[0];
-  await s3DataSource.uploadFile(
-    `inventory/date=${timestamp}/inventory.parquet`,
-    parquetBuffer,
-    { contentType: 'application/octet-stream' }
-  );
-
-  // 4. Trigger data warehouse ingestion (e.g., Snowflake, Redshift)
-  await triggerWarehouseIngestion(timestamp);
-
-  console.log('✅ Daily extract complete!');
-}
-
-async function triggerWarehouseIngestion(date: string) {
-  // Trigger Snowflake/Redshift/BigQuery ingestion
-  console.log(`Triggering warehouse ingestion for ${date}`);
-}
-```
-
-## Error Handling
-
-```typescript
-async function robustParquetParsing(parquetBuffer: ArrayBuffer) {
-  const logger = createConsoleLogger();
-  const parquetParser = new ParquetParserService(logger);
-
-  try {
-    // Validate first
-    const validation = await parquetParser.validateFile(parquetBuffer, 'file.parquet');
-
-    if (!validation.isValid) {
-      throw new Error(`Validation failed: ${validation.errors.map(e => e.message).join(', ')}`);
-    }
-
-    // Parse
-    const records = await parquetParser.parseSimple(parquetBuffer, 'file.parquet');
-    return records;
-
-  } catch (error) {
-    if (error.message.includes('Invalid Parquet file')) {
-      console.error('File is not a valid Parquet file');
-    } else if (error.message.includes('Corrupted')) {
-      console.error('Parquet file is corrupted');
-    } else {
-      console.error('Unexpected error:', error);
-    }
-
-    throw error;
-  }
-}
-```
-
-## Key Takeaways
-
-- 🎯 Parquet is columnar format optimized for analytics
-- 🎯 Use for large-scale data extraction (1M+ records)
-- 🎯 Provides 10:1 compression ratio vs CSV/JSON
-- 🎯 Stream large files with `parseStreaming()` for memory efficiency
-- 🎯 Integrates with S3 for data warehouse pipelines
-- 🎯 Ideal for Fluent → Analytics warehouse workflows
-- 🎯 Column statistics enable query optimization
-
-## Practice Exercise
-
-**Challenge:** Extract virtual positions from Fluent and save as Parquet:
-
-**Requirements:**
-1. Fetch all virtual positions with auto-pagination
-2. Transform to flat structure (no nested objects)
-3. Convert to Parquet format
-4. Upload to S3 with date partition: `data/date=2025-01-15/positions.parquet`
-
-**Solution:** See [Parquet Examples](../examples/parquet-parser-examples.ts)
-
-## Next Steps
-
-Continue to [Module 6: Integration Patterns →](./02-core-guides-parsers-06-integration-patterns.md) to learn how to combine parsers with other SDK services.
-
----
-
-**Module 5 Complete!** ✅
+# Module 5: Parquet Parser
+
+**Level:** Advanced
+**Estimated Time:** 20 minutes
+
+## Overview
+
+This module covers the Parquet Parser (`ParquetParserService`), designed for efficient large-scale data extraction and analytics. You'll learn how to parse Parquet files, leverage columnar storage, handle compression, and integrate with S3 for data extraction workflows.
+
+## Learning Objectives
+
+By the end of this module, you will:
+- ✅ Understand Parquet columnar format and its benefits
+- ✅ Parse Parquet files into JavaScript objects
+- ✅ Stream large Parquet files efficiently
+- ✅ Leverage compression and columnar storage
+- ✅ Extract Fluent data to Parquet for analytics
+- ✅ Integrate with S3 extraction workflows
+
+## What is Parquet?
+
+**Apache Parquet** is a columnar storage format that:
+- Stores data by column instead of by row
+- Uses efficient compression algorithms
+- Optimizes for analytical queries
+- Reduces file size significantly (often 10:1 compression)
+- Is the standard for big data and analytics
+
+**Row-based vs Column-based storage:**
+
+```
+Row-based (CSV):
+SKU001,100,WH-01
+SKU002,50,WH-01
+SKU003,75,WH-02
+
+Column-based (Parquet):
+Column[sku]:      [SKU001, SKU002, SKU003]
+Column[qty]:      [100, 50, 75]
+Column[location]: [WH-01, WH-01, WH-02]
+```
+
+**Benefits:**
+- ✅ Better compression (similar values in a column compress well)
+- ✅ Faster column-specific queries
+- ✅ Smaller file sizes
+- ✅ Efficient for analytics
+
+## Basic Parquet Parsing
+
+### Simple Parsing
+
+```typescript
+import { ParquetParserService, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
+import * as fs from 'fs/promises';
+
+// ParquetParserService requires a StructuredLogger (not a simple logger)
+const consoleLogger = createConsoleLogger();
+const logger = toStructuredLogger(consoleLogger, { service: 'ParquetParser' });
+
+// CRITICAL: Constructor requires logger parameter (NOT optional)
+const parquetParser = new ParquetParserService(logger);
+
+// Read Parquet file
+const fileBuffer = await fs.readFile('inventory.parquet');
+
+// Convert to ArrayBuffer (parseSimple requires ArrayBuffer, not Buffer)
+const arrayBuffer = fileBuffer.buffer.slice(
+  fileBuffer.byteOffset,
+  fileBuffer.byteOffset + fileBuffer.byteLength
+);
+
+// Parse Parquet (simple mode - easiest)
+const records = await parquetParser.parseSimple(arrayBuffer, 'inventory.parquet');
+
+console.log(`Parsed ${records.length} records`);
+console.log('First record:', records[0]);
+// { sku: 'SKU001', qty: 100, location: 'WH-01' }
+```
+
+### Type Safety with TypeScript
+
+```typescript
+// Define your record structure
+interface InventoryRecord {
+  sku: string;
+  qty: number;
+  location: string;
+  updatedAt: Date;
+}
+
+// Parse with type assertion (simple mode)
+const records = await parquetParser.parseSimple(arrayBuffer, 'inventory.parquet');
+
+// Or use parse() for detailed stats
+const result = await parquetParser.parse(arrayBuffer, 'inventory.parquet');
+console.log(`Processed ${result.stats.totalRecords} records in ${result.stats.processingTimeMs}ms`);
+
+// Full type safety
+result.records.forEach(record => {
+  console.log(`${record.sku}: ${record.qty} at ${record.location}`);
+});
+```
+
+## Streaming Large Files
+
+For large Parquet files, use streaming to avoid memory issues:
+
+### Basic Streaming
+
+```typescript
+import { ParquetParserService, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
+
+const consoleLogger = createConsoleLogger();
+const logger = toStructuredLogger(consoleLogger, { service: 'ParquetParser' });
+
+const parquetParser = new ParquetParserService(logger);
+
+// Stream Parquet file
+const fileBuffer = await fs.readFile('large-inventory.parquet');
+
+// Convert to ArrayBuffer
+const arrayBuffer = fileBuffer.buffer.slice(
+  fileBuffer.byteOffset,
+  fileBuffer.byteOffset + fileBuffer.byteLength
+);
+
+// Process records in batches
+const batchSize = 1000;
+
+const streamingResult = await parquetParser.parseStreaming(
+  arrayBuffer,
+  'large-inventory.parquet',
+  { batchSize }
+);
+
+for await (const batch of streamingResult.records) {
+  console.log(`Processing batch of ${batch.length} records`);
+  await processBatch(batch);
+}
+
+// Get final stats
+const stats = await streamingResult.stats;
+console.log(`Processed ${stats.totalRecords} total records`);
+
+async function processBatch(records: any[]) {
+  // Transform and process
+  console.log(`Batch processing ${records.length} records`);
+}
+```
+
+### Streaming with Progress Tracking
+
+```typescript
+async function streamWithProgress(parquetBuffer: ArrayBuffer) {
+  const logger = createConsoleLogger();
+  const parquetParser = new ParquetParserService(logger);
+  const startTime = Date.now();
+  let totalRecords = 0;
+
+  const streamingResult = await parquetParser.parseStreaming(
+    parquetBuffer,
+    'file.parquet',
+    { batchSize: 1000 }
+  );
+
+  for await (const batch of streamingResult.records) {
+    totalRecords += batch.length;
+
+    // Process batch
+    await processBatch(batch);
+
+    // Report progress
+    const elapsed = (Date.now() - startTime) / 1000;
+    const rate = totalRecords / elapsed;
+    console.log(`Processed ${totalRecords} records (${rate.toFixed(2)} records/sec)`);
+  }
+
+  const stats = await streamingResult.stats;
+  console.log(`Completed: ${stats.totalRecords} total records`);
+}
+```
+
+## Parser Configuration
+
+The Parquet parser supports various options:
+
+```typescript
+interface ParquetParsingOptions {
+  // Batch size for streaming (default: 1000)
+  batchSize?: number;
+
+  // Maximum file size in bytes (default: 100MB)
+  maxFileSize?: number;
+
+  // Enable streaming mode (default: true)
+  enableStreaming?: boolean;
+
+  // Validation options
+  validation?: {
+    enabled?: boolean;
+    requiredColumns?: string[];
+    columnTypes?: Record<string, ParquetDataType>;
+    allowAdditionalColumns?: boolean;
+  };
+}
+```
+
+### Example Configuration
+
+```typescript
+const options: ParquetParsingOptions = {
+  batchSize: 5000,
+  maxFileSize: 500 * 1024 * 1024, // 500MB
+  enableStreaming: true,
+  validation: {
+    enabled: true,
+    requiredColumns: ['sku', 'qty', 'location'],
+    columnTypes: {
+      sku: 'string',
+      qty: 'number',
+      location: 'string'
+    }
+  }
+};
+
+const result = await parquetParser.parse(parquetBuffer, 'file.parquet', options);
+const records = result.records;
+```
+
+## Data Extraction Workflow
+
+The primary use case for Parquet is extracting large datasets from Fluent Commerce to S3.
+
+### Complete Extraction Example
+
+```typescript
+import {
+  createClient,
+  ParquetParserService,
+  S3DataSource,
+  createConsoleLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function extractInventoryToParquet() {
+  // 1. Setup logger
+  const logger = createConsoleLogger();
+
+  // 2. Create client
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+
+  // 3. Fetch data from Fluent (with auto-pagination)
+  const result = await client.graphql({
+    query: `
+      query GetVirtualPositions($first: Int!, $after: String) {
+        virtualPositions(first: $first, after: $after) {
+          edges {
+            node {
+              id
+              productRef
+              quantity
+              groupRef
+              createdOn
+              updatedOn
+            }
+            cursor
+          }
+          pageInfo {
+            hasNextPage
+          }
+        }
+      }
+    `,
+    variables: { first: 1000 },
+    pagination: { maxPages: 100 }
+  });
+
+  // 4. Transform to flat structure
+  const records = result.data.virtualPositions.edges.map(edge => ({
+    id: edge.node.id,
+    productRef: edge.node.productRef,
+    quantity: edge.node.quantity,
+    groupRef: edge.node.groupRef,
+    createdOn: edge.node.createdOn,
+    updatedOn: edge.node.updatedOn
+  }));
+
+  console.log(`Extracted ${records.length} virtual positions`);
+
+  // 5. Convert to Parquet (using parquetjs or hyparquet)
+  const parquetBuffer = await convertToParquet(records);
+
+  // 6. Upload to S3 using S3DataSource
+  const s3DataSource = new S3DataSource({
+    type: 'S3_PARQUET',
+    connectionId: 's3-parquet-extract',
+    name: 'S3 Parquet Extract',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION!,
+      bucket: 'my-analytics-bucket'
+    }
+  }, logger);
+
+  await s3DataSource.uploadFile(
+    `extracts/virtual-positions-${new Date().toISOString()}.parquet`,
+    parquetBuffer,
+    { contentType: 'application/octet-stream' }
+  );
+
+  console.log('✅ Extraction complete!');
+}
+
+async function convertToParquet(records: any[]): Promise<Buffer> {
+  // Use parquetjs or hyparquet to convert
+  // Implementation depends on your choice of Parquet library
+  const parquetWriter = /* ... */;
+  // Write records and return buffer
+  return buffer;
+}
+```
+
+## Performance Metrics
+
+Get performance statistics from parsing operations:
+
+```typescript
+const logger = createConsoleLogger();
+const parquetParser = new ParquetParserService(logger);
+
+// Parse file
+const result = await parquetParser.parse(parquetBuffer, 'file.parquet');
+
+// Access parsing statistics
+console.log('Parsing statistics:');
+console.log(`Total records: ${result.stats.totalRecords}`);
+console.log(`Processing time: ${result.stats.processingTimeMs}ms`);
+console.log(`Peak memory usage: ${(result.stats.peakMemoryUsage / 1024 / 1024).toFixed(2)}MB`);
+console.log(`Records per second: ${result.stats.recordsPerSecond}`);
+console.log(`Batches processed: ${result.stats.batchesProcessed}`);
+
+// Get overall performance metrics
+const metrics = parquetParser.getPerformanceMetrics();
+console.log('Overall metrics:', metrics);
+```
+
+## Validation
+
+Validate Parquet structure:
+
+```typescript
+const logger = createConsoleLogger();
+const parquetParser = new ParquetParserService(logger);
+
+// Validate Parquet file
+const validation = await parquetParser.validateFile(parquetBuffer, 'file.parquet');
+
+if (validation.isValid) {
+  console.log('Parquet file is valid');
+  const records = await parquetParser.parseSimple(parquetBuffer, 'file.parquet');
+} else {
+  console.error('Validation errors:', validation.errors);
+}
+```
+
+## Integration with S3
+
+### Read Parquet from S3
+
+```typescript
+import { S3DataSource, ParquetParserService, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function readParquetFromS3() {
+  const logger = createConsoleLogger();
+
+  // Initialize S3 data source
+  const s3DataSource = new S3DataSource({
+    type: 'S3_PARQUET',
+    connectionId: 's3-parquet-read',
+    name: 'S3 Parquet Read',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION!,
+      bucket: 'my-data-bucket'
+    }
+  }, logger);
+
+  // Download Parquet file from S3
+  const parquetBuffer = await s3DataSource.downloadFile('extracts/inventory.parquet');
+
+  // Parse Parquet
+  const parquetParser = new ParquetParserService(logger);
+  const records = await parquetParser.parseSimple(parquetBuffer, 'inventory.parquet');
+
+  console.log(`Loaded ${records.length} records from S3`);
+  return records;
+}
+```
+
+### Write Parquet to S3
+
+```typescript
+async function writeParquetToS3(records: any[]) {
+  const logger = createConsoleLogger();
+
+  // Convert to Parquet
+  const parquetBuffer = await convertToParquet(records);
+
+  // Upload to S3 using S3DataSource
+  const s3DataSource = new S3DataSource({
+    type: 'S3_PARQUET',
+    connectionId: 's3-parquet-write',
+    name: 'S3 Parquet Write',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION!,
+      bucket: 'my-analytics-bucket'
+    }
+  }, logger);
+
+  await s3DataSource.uploadFile(
+    `extracts/data-${Date.now()}.parquet`,
+    parquetBuffer,
+    { contentType: 'application/octet-stream' }
+  );
+
+  console.log('✅ Parquet file uploaded to S3');
+}
+```
+
+## Performance Optimization
+
+### 1. Use Appropriate Batch Sizes
+
+```typescript
+// Small batches for memory-constrained environments
+const smallBatch = await parquetParser.parseStreaming(parquetBuffer, {
+  batchSize: 500
+});
+
+// Large batches for high-memory environments
+const largeBatch = await parquetParser.parseStreaming(parquetBuffer, {
+  batchSize: 10000
+});
+```
+
+### 2. Leverage Compression
+
+Parquet automatically compresses data, but you can optimize:
+
+```typescript
+// When creating Parquet files, use optimal compression
+const parquetOptions = {
+  compression: 'SNAPPY', // Fast compression/decompression
+  // or 'GZIP' for better compression ratio
+};
+```
+
+### 3. Column Pruning
+
+Only read columns you need (when using advanced Parquet libraries):
+
+```typescript
+// Only read specific columns
+const selectedColumns = await parquetParser.parse(parquetBuffer, {
+  columns: ['sku', 'qty'] // Only read these columns
+});
+```
+
+## Real-World Example: Analytics Pipeline
+
+Complete pipeline for extracting Fluent data to analytics warehouse:
+
+```typescript
+import {
+  createClient,
+  ParquetParserService,
+  S3DataSource,
+  createConsoleLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function dailyInventoryExtract() {
+  const logger = createConsoleLogger();
+  const client = await createClient({ config });
+
+  const s3DataSource = new S3DataSource({
+    type: 'S3_PARQUET',
+    connectionId: 's3-analytics',
+    name: 'S3 Analytics',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION!,
+      bucket: 'analytics-warehouse'
+    }
+  }, logger);
+
+  // 1. Extract inventory from Fluent
+  const result = await client.graphql({
+    query: inventoryQuery,
+    variables: { first: 1000 },
+    pagination: { maxRecords: 1000000 }
+  });
+
+  const records = result.data.inventoryPositions.edges.map(e => e.node);
+
+  console.log(`Extracted ${records.length} inventory positions`);
+
+  // 2. Convert to Parquet
+  const parquetBuffer = await convertToParquet(records);
+
+  // 3. Upload to S3 analytics bucket
+  const timestamp = new Date().toISOString().split('T')[0];
+  await s3DataSource.uploadFile(
+    `inventory/date=${timestamp}/inventory.parquet`,
+    parquetBuffer,
+    { contentType: 'application/octet-stream' }
+  );
+
+  // 4. Trigger data warehouse ingestion (e.g., Snowflake, Redshift)
+  await triggerWarehouseIngestion(timestamp);
+
+  console.log('✅ Daily extract complete!');
+}
+
+async function triggerWarehouseIngestion(date: string) {
+  // Trigger Snowflake/Redshift/BigQuery ingestion
+  console.log(`Triggering warehouse ingestion for ${date}`);
+}
+```
+
+## Error Handling
+
+```typescript
+async function robustParquetParsing(parquetBuffer: ArrayBuffer) {
+  const logger = createConsoleLogger();
+  const parquetParser = new ParquetParserService(logger);
+
+  try {
+    // Validate first
+    const validation = await parquetParser.validateFile(parquetBuffer, 'file.parquet');
+
+    if (!validation.isValid) {
+      throw new Error(`Validation failed: ${validation.errors.map(e => e.message).join(', ')}`);
+    }
+
+    // Parse
+    const records = await parquetParser.parseSimple(parquetBuffer, 'file.parquet');
+    return records;
+
+  } catch (error) {
+    if (error.message.includes('Invalid Parquet file')) {
+      console.error('File is not a valid Parquet file');
+    } else if (error.message.includes('Corrupted')) {
+      console.error('Parquet file is corrupted');
+    } else {
+      console.error('Unexpected error:', error);
+    }
+
+    throw error;
+  }
+}
+```
+
+## Key Takeaways
+
+- 🎯 Parquet is columnar format optimized for analytics
+- 🎯 Use for large-scale data extraction (1M+ records)
+- 🎯 Provides 10:1 compression ratio vs CSV/JSON
+- 🎯 Stream large files with `parseStreaming()` for memory efficiency
+- 🎯 Integrates with S3 for data warehouse pipelines
+- 🎯 Ideal for Fluent → Analytics warehouse workflows
+- 🎯 Column statistics enable query optimization
+
+## Practice Exercise
+
+**Challenge:** Extract virtual positions from Fluent and save as Parquet:
+
+**Requirements:**
+1. Fetch all virtual positions with auto-pagination
+2. Transform to flat structure (no nested objects)
+3. Convert to Parquet format
+4. Upload to S3 with date partition: `data/date=2025-01-15/positions.parquet`
+
+**Solution:** See [Parquet Examples](../examples/parquet-parser-examples.ts)
+
+## Next Steps
+
+Continue to [Module 6: Integration Patterns →](./02-core-guides-parsers-06-integration-patterns.md) to learn how to combine parsers with other SDK services.
+
+---
+
+**Module 5 Complete!** ✅