npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.54 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (475) hide show

package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md CHANGED Viewed

@@ -1,554 +1,554 @@
-# Module 4: Writing Parquet Files
-[← Back to Parquet Guide](../parquet-readme.md)
-**Module 4 of 8** | **Level**: Intermediate | **Time**: 20 minutes
----
-## Overview
-Learn how to write Parquet files using external libraries. The SDK focuses on **reading** Parquet files, but this module shows integration patterns for **writing** using `parquetjs`.
-## Learning Objectives
-By the end of this module, you will:
-- ✅ Understand why SDK doesn't include writing (and what to use instead)
-- ✅ Write Parquet files using `parquetjs` library
-- ✅ Define schemas and compression options
-- ✅ Integrate writing with SDK extraction workflows
----
-## SDK Writing Support
-⚠️ **IMPORTANT**: The FC Connect SDK **ParquetParserService focuses on reading**, not writing.
-**Why?**
-- Reading is the primary use case for integrations (parse incoming files)
-- Writing libraries (`parquetjs`) are mature and well-maintained
-- Keeps SDK focused and lightweight
-**For Writing**: Use `parquetjs` library alongside the SDK.
----
-## Writing with parquetjs
-### Installation
-```bash
-npm install parquetjs
-npm install @types/parquetjs --save-dev  # TypeScript types
-```
-### Basic Writing Example
-```typescript
-import * as parquet from 'parquetjs';
-// Define schema
-const schema = new parquet.ParquetSchema({
-  ref: { type: 'UTF8', optional: false },
-  productRef: { type: 'UTF8', optional: false },
-  locationRef: { type: 'UTF8', optional: false },
-  qty: { type: 'INT32', optional: false },
-  status: { type: 'UTF8', optional: true },
-  updatedOn: { type: 'TIMESTAMP_MILLIS', optional: true },
-});
-// Create writer
-const writer = await parquet.ParquetWriter.openFile(schema, 'inventory.parquet', {
-  compression: 'SNAPPY', // Fast compression
-  rowGroupSize: 10000, // Records per row group
-});
-// Write records
-const records = [
-  {
-    ref: 'IP-001',
-    productRef: 'SKU-001',
-    locationRef: 'WH-001',
-    qty: 100,
-    status: 'AVAILABLE',
-    updatedOn: new Date(),
-  },
-  {
-    ref: 'IP-002',
-    productRef: 'SKU-002',
-    locationRef: 'WH-001',
-    qty: 50,
-    status: 'RESERVED',
-    updatedOn: new Date(),
-  },
-];
-for (const record of records) {
-  await writer.appendRow(record);
-}
-// Close writer
-await writer.close();
-console.log('✅ Parquet file written successfully');
-```
----
-## Schema Definition
-### Supported Data Types
-```typescript
-const schema = new parquet.ParquetSchema({
-  // Strings
-  stringField: { type: 'UTF8' },
-  // Integers
-  int32Field: { type: 'INT32' },
-  int64Field: { type: 'INT64' },
-  // Floating point
-  floatField: { type: 'FLOAT' },
-  doubleField: { type: 'DOUBLE' },
-  // Boolean
-  boolField: { type: 'BOOLEAN' },
-  // Dates/Times
-  dateField: { type: 'DATE' },
-  timestampField: { type: 'TIMESTAMP_MILLIS' },
-  // Optional fields
-  optionalField: { type: 'UTF8', optional: true },
-  // Arrays (repeated fields)
-  arrayField: {
-    type: 'UTF8',
-    repeated: true,
-  },
-  // Nested structures
-  nestedField: {
-    fields: {
-      subField1: { type: 'UTF8' },
-      subField2: { type: 'INT32' },
-    },
-  },
-});
-```
-### Inventory Position Schema
-```typescript
-const inventorySchema = new parquet.ParquetSchema({
-  ref: { type: 'UTF8', optional: false },
-  productRef: { type: 'UTF8', optional: false },
-  locationRef: { type: 'UTF8', optional: false },
-  qty: { type: 'INT32', optional: false },
-  status: { type: 'UTF8', optional: true },
-  type: { type: 'UTF8', optional: true },
-  createdOn: { type: 'TIMESTAMP_MILLIS', optional: true },
-  updatedOn: { type: 'TIMESTAMP_MILLIS', optional: true },
-});
-```
----
-## Compression Options
-### Available Compression Algorithms
-```typescript
-// UNCOMPRESSED - No compression (fastest write, largest size)
-const writer1 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
-  compression: 'UNCOMPRESSED',
-});
-// SNAPPY - Fast compression (recommended for most cases)
-const writer2 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
-  compression: 'SNAPPY', // Default, good balance
-});
-// GZIP - Better compression ratio (slower)
-const writer3 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
-  compression: 'GZIP',
-  compressionLevel: 6, // 1-9, higher = better compression, slower
-});
-// BROTLI - Best compression (slowest)
-const writer4 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
-  compression: 'BROTLI',
-  compressionLevel: 4, // 1-11
-});
-```
-### Compression Comparison
-| Algorithm        | Compression Ratio | Write Speed | Read Speed | Use When                      |
-| ---------------- | ----------------- | ----------- | ---------- | ----------------------------- |
-| **UNCOMPRESSED** | 1x                | Fastest     | Fastest    | Local testing only            |
-| **SNAPPY**       | 2-3x              | Very Fast   | Very Fast  | General purpose (recommended) |
-| **GZIP**         | 3-4x              | Moderate    | Fast       | Storage optimization          |
-| **BROTLI**       | 4-5x              | Slow        | Fast       | Maximum compression needed    |
-**Recommendation**: Use **SNAPPY** for most cases - excellent balance of speed and compression.
----
-## Integration with SDK Extraction
-### Pattern: GraphQL → Transform → Parquet
-```typescript
-import { createClient } from '@fluentcommerce/fc-connect-sdk';
-import * as parquet from 'parquetjs';
-/**
- * Extract inventory from Fluent Commerce and save as Parquet
- * Business use case: Daily inventory snapshot for analytics
- */
-async function extractToParquet(outputFile: string) {
-  // 1. Setup SDK client
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!,
-    },
-  });
-  // 2. Define Parquet schema
-  const schema = new parquet.ParquetSchema({
-    ref: { type: 'UTF8' },
-    productRef: { type: 'UTF8' },
-    locationRef: { type: 'UTF8' },
-    qty: { type: 'INT32' },
-    status: { type: 'UTF8', optional: true },
-    extractedAt: { type: 'TIMESTAMP_MILLIS' },
-  });
-  // 3. Create Parquet writer
-  const writer = await parquet.ParquetWriter.openFile(schema, outputFile, {
-    compression: 'SNAPPY',
-    rowGroupSize: 10000,
-  });
-  // 4. Extract data with pagination
-  let hasNextPage = true;
-  let cursor: string | null = null;
-  let totalRecords = 0;
-  while (hasNextPage) {
-    const result = await client.graphql({
-      query: `
-        query ExtractInventory($after: String, $first: Int!) {
-          inventoryPositions(after: $after, first: $first) {
-            # ✅ Query.inventoryPositions (InventoryPositionConnection!)
-            pageInfo {
-              hasNextPage
-            }
-            edges {
-              cursor
-              node {
-                ref              # ✅ InventoryPosition.ref (String!)
-                productRef       # ✅ InventoryPosition.productRef (String!)
-                locationRef      # ✅ InventoryPosition.locationRef (String!)
-                qty              # ✅ InventoryPosition.qty (Int!)
-                status           # ✅ InventoryPosition.status (String)
-              }
-            }
-          }
-        }
-      `,
-      variables: {
-        first: 1000,
-        after: cursor,
-      },
-    });
-    const { edges, pageInfo } = result.data.inventoryPositions;
-    // 5. Write batch to Parquet
-    for (const edge of edges) {
-      await writer.appendRow({
-        ...edge.node,
-        extractedAt: new Date(),
-      });
-      totalRecords++;
-    }
-    console.log(`Written ${totalRecords} records...`);
-    hasNextPage = pageInfo.hasNextPage;
-    // ⚠️ Use last edge cursor (Fluent doesn't return pageInfo.endCursor)
-    cursor = edges.length > 0 ? edges[edges.length - 1].cursor : null;
-  }
-  // 6. Close writer
-  await writer.close();
-  console.log(`✅ Complete: ${totalRecords} records written to ${outputFile}`);
-}
-// Usage
-extractToParquet('inventory_2025-10-19.parquet').catch(console.error);
-```
----
-## Batch Writing for Performance
-### Writing Large Datasets
-```typescript
-async function writeLargeDataset(records: any[], outputFile: string, batchSize: number = 10000) {
-  const schema = new parquet.ParquetSchema({
-    ref: { type: 'UTF8' },
-    productRef: { type: 'UTF8' },
-    locationRef: { type: 'UTF8' },
-    qty: { type: 'INT32' },
-  });
-  const writer = await parquet.ParquetWriter.openFile(schema, outputFile, {
-    compression: 'SNAPPY',
-    rowGroupSize: batchSize, // Optimize row group size
-  });
-  let written = 0;
-  for (let i = 0; i < records.length; i += batchSize) {
-    const batch = records.slice(i, i + batchSize);
-    for (const record of batch) {
-      await writer.appendRow(record);
-    }
-    written += batch.length;
-    console.log(`Written ${written} / ${records.length} records`);
-  }
-  await writer.close();
-  console.log(`✅ Complete: ${written} records`);
-}
-```
----
-## S3DataSource.writeParquetContent()
-The SDK provides a convenience method to generate Parquet files as Buffers, which can then be uploaded to S3.
-### Method Signature
-```typescript
-async writeParquetContent(
-  records: ParquetDataRecord[],
-  options?: {
-    compression?: 'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI';
-    schema?: Record<string, any>;
-    rowGroupSize?: number;
-  }
-): Promise<Buffer>
-```
-### Basic Usage
-```typescript
-import { S3DataSource, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
-const logger = toStructuredLogger(createConsoleLogger(), { service: 'S3Parquet' });
-const s3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      bucket: process.env.S3_BUCKET!,
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION!,
-    },
-  },
-  logger
-);
-const records = [
-  { id: 1, name: 'Product A', qty: 100 },
-  { id: 2, name: 'Product B', qty: 200 },
-];
-// Generate Parquet Buffer
-const parquetBuffer = await s3.writeParquetContent(records, {
-  schema: {
-    id: { type: 'INT32' },
-    name: { type: 'UTF8' },
-    qty: { type: 'INT32' },
-  },
-  compression: 'SNAPPY',
-});
-// Upload to S3
-await s3.uploadFile('data/products-2025-10-30.parquet', parquetBuffer);
-console.log('✅ Parquet file written to S3');
-```
-### Complete Extraction Example
-```typescript
-import { createClient, S3DataSource, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
-async function extractToS3Parquet() {
-  const logger = toStructuredLogger(createConsoleLogger(), { service: 'Extraction' });
-  // 1. Create Fluent client
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!,
-    },
-  });
-  // 2. Setup S3
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      s3Config: {
-        bucket: process.env.S3_BUCKET!,
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        region: process.env.AWS_REGION!,
-      },
-    },
-    logger
-  );
-  // 3. Extract data with pagination
-  const allRecords = [];
-  let hasNextPage = true;
-  let cursor: string | null = null;
-  while (hasNextPage) {
-    const result = await client.graphql({
-      query: `
-        query ExtractInventory($after: String, $first: Int!) {
-          inventoryPositions(after: $after, first: $first) {
-            pageInfo { hasNextPage }
-            edges {
-              cursor
-              node {
-                ref
-                productRef
-                locationRef
-                qty
-                status
-              }
-            }
-          }
-        }
-      `,
-      variables: { first: 1000, after: cursor },
-    });
-    const { edges, pageInfo } = result.data.inventoryPositions;
-    allRecords.push(...edges.map((e) => ({
-      ...e.node,
-      extractedAt: new Date().toISOString(),
-    })));
-    hasNextPage = pageInfo.hasNextPage;
-    cursor = edges.length > 0 ? edges[edges.length - 1].cursor : null;
-  }
-  console.log(`Extracted ${allRecords.length} records`);
-  // 4. Generate Parquet Buffer and upload to S3
-  const parquetBuffer = await s3.writeParquetContent(allRecords, {
-    schema: {
-      ref: { type: 'UTF8' },
-      productRef: { type: 'UTF8' },
-      locationRef: { type: 'UTF8' },
-      qty: { type: 'INT32' },
-      status: { type: 'UTF8' },
-      extractedAt: { type: 'UTF8' },
-    },
-    compression: 'SNAPPY',
-  });
-  const s3Key = `extracts/inventory_${new Date().toISOString().split('T')[0]}.parquet`;
-  await s3.uploadFile(s3Key, parquetBuffer);
-  console.log(`✅ Complete: Data written to S3 as Parquet at ${s3Key}`);
-}
-extractToS3Parquet().catch(console.error);
-```
-### Advantages
-- ✅ **No temporary files** - Generates Buffer directly in memory
-- ✅ **Simplified workflow** - Generate Buffer, then upload
-- ✅ **Memory efficient** - No disk I/O needed for generation
-- ✅ **Flexible** - Upload to S3, SFTP, or save locally
-### Parameters
-**records** (`ParquetDataRecord[]`)
-- Array of data objects to write
-- Must match the provided schema structure
-**options.schema** (`Record<string, any>` | optional)
-- Parquet schema definition
-- If not provided, schema is auto-detected from records
-- Example: `{ id: { type: 'INT32' }, name: { type: 'UTF8' } }`
-**options.compression** (`'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI'` | optional)
-- Compression algorithm (default: `'UNCOMPRESSED'`)
-- Recommended: `'SNAPPY'` for balance of speed and compression
-**options.rowGroupSize** (`number` | optional)
-- Number of rows per row group (default: 5000)
-- Smaller = better for selective queries
-- Larger = better for full scans
-**Returns:** `Promise<Buffer>` - Parquet file content as Buffer
-**Note:** After generating the Buffer, upload it separately:
-```typescript
-const buffer = await s3.writeParquetContent(records, options);
-await s3.uploadFile('path/to/file.parquet', buffer);
-```
----
-## Key Takeaways
-- 🎯 **SDK reads, parquetjs writes** - Use both together for complete workflows
-- 🎯 **S3DataSource.writeParquetContent()** - Generates Parquet Buffer (then upload separately)
-- 🎯 **No temporary files** - Generates Buffer directly in memory
-- 🎯 **SNAPPY compression** - Best default choice for most use cases
-- 🎯 **Schema optional** - Auto-detected if not provided
-- 🎯 **Row groups** - Optimize size for query patterns (10K-50K typical)
-- 🎯 **Integrate with SDK** - Extract from Fluent → Write to Parquet
----
-## Next Steps
-Continue to [Module 5: GraphQL Extraction](./03-pattern-guides-parquet-05-graphql-extraction.md/) to learn complete extraction workflows from Fluent Commerce to Parquet.
----
-[← Previous: Module 3 - Reading Parquet](./03-pattern-guides-parquet-03-reading-parquet.md) | [Back to Guide](../parquet-readme.md) | [Next: Module 5 - GraphQL Extraction →](./03-pattern-guides-parquet-05-graphql-extraction.md/)
-## Related Documentation
-- [Module 5: GraphQL Extraction](./03-pattern-guides-parquet-05-graphql-extraction.md/) - Complete extraction workflows
-- [Data Extraction Guide](../../extraction/) - GraphQL extraction patterns
-- [parquetjs Documentation](https://github.com/ironSource/parquetjs) - Official library docs
+# Module 4: Writing Parquet Files
+[← Back to Parquet Guide](../parquet-readme.md)
+**Module 4 of 8** | **Level**: Intermediate | **Time**: 20 minutes
+---
+## Overview
+Learn how to write Parquet files using external libraries. The SDK focuses on **reading** Parquet files, but this module shows integration patterns for **writing** using `parquetjs`.
+## Learning Objectives
+By the end of this module, you will:
+- ✅ Understand why SDK doesn't include writing (and what to use instead)
+- ✅ Write Parquet files using `parquetjs` library
+- ✅ Define schemas and compression options
+- ✅ Integrate writing with SDK extraction workflows
+---
+## SDK Writing Support
+⚠️ **IMPORTANT**: The FC Connect SDK **ParquetParserService focuses on reading**, not writing.
+**Why?**
+- Reading is the primary use case for integrations (parse incoming files)
+- Writing libraries (`parquetjs`) are mature and well-maintained
+- Keeps SDK focused and lightweight
+**For Writing**: Use `parquetjs` library alongside the SDK.
+---
+## Writing with parquetjs
+### Installation
+```bash
+npm install parquetjs
+npm install @types/parquetjs --save-dev  # TypeScript types
+```
+### Basic Writing Example
+```typescript
+import * as parquet from 'parquetjs';
+// Define schema
+const schema = new parquet.ParquetSchema({
+  ref: { type: 'UTF8', optional: false },
+  productRef: { type: 'UTF8', optional: false },
+  locationRef: { type: 'UTF8', optional: false },
+  qty: { type: 'INT32', optional: false },
+  status: { type: 'UTF8', optional: true },
+  updatedOn: { type: 'TIMESTAMP_MILLIS', optional: true },
+});
+// Create writer
+const writer = await parquet.ParquetWriter.openFile(schema, 'inventory.parquet', {
+  compression: 'SNAPPY', // Fast compression
+  rowGroupSize: 10000, // Records per row group
+});
+// Write records
+const records = [
+  {
+    ref: 'IP-001',
+    productRef: 'SKU-001',
+    locationRef: 'WH-001',
+    qty: 100,
+    status: 'AVAILABLE',
+    updatedOn: new Date(),
+  },
+  {
+    ref: 'IP-002',
+    productRef: 'SKU-002',
+    locationRef: 'WH-001',
+    qty: 50,
+    status: 'RESERVED',
+    updatedOn: new Date(),
+  },
+];
+for (const record of records) {
+  await writer.appendRow(record);
+}
+// Close writer
+await writer.close();
+console.log('✅ Parquet file written successfully');
+```
+---
+## Schema Definition
+### Supported Data Types
+```typescript
+const schema = new parquet.ParquetSchema({
+  // Strings
+  stringField: { type: 'UTF8' },
+  // Integers
+  int32Field: { type: 'INT32' },
+  int64Field: { type: 'INT64' },
+  // Floating point
+  floatField: { type: 'FLOAT' },
+  doubleField: { type: 'DOUBLE' },
+  // Boolean
+  boolField: { type: 'BOOLEAN' },
+  // Dates/Times
+  dateField: { type: 'DATE' },
+  timestampField: { type: 'TIMESTAMP_MILLIS' },
+  // Optional fields
+  optionalField: { type: 'UTF8', optional: true },
+  // Arrays (repeated fields)
+  arrayField: {
+    type: 'UTF8',
+    repeated: true,
+  },
+  // Nested structures
+  nestedField: {
+    fields: {
+      subField1: { type: 'UTF8' },
+      subField2: { type: 'INT32' },
+    },
+  },
+});
+```
+### Inventory Position Schema
+```typescript
+const inventorySchema = new parquet.ParquetSchema({
+  ref: { type: 'UTF8', optional: false },
+  productRef: { type: 'UTF8', optional: false },
+  locationRef: { type: 'UTF8', optional: false },
+  qty: { type: 'INT32', optional: false },
+  status: { type: 'UTF8', optional: true },
+  type: { type: 'UTF8', optional: true },
+  createdOn: { type: 'TIMESTAMP_MILLIS', optional: true },
+  updatedOn: { type: 'TIMESTAMP_MILLIS', optional: true },
+});
+```
+---
+## Compression Options
+### Available Compression Algorithms
+```typescript
+// UNCOMPRESSED - No compression (fastest write, largest size)
+const writer1 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
+  compression: 'UNCOMPRESSED',
+});
+// SNAPPY - Fast compression (recommended for most cases)
+const writer2 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
+  compression: 'SNAPPY', // Default, good balance
+});
+// GZIP - Better compression ratio (slower)
+const writer3 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
+  compression: 'GZIP',
+  compressionLevel: 6, // 1-9, higher = better compression, slower
+});
+// BROTLI - Best compression (slowest)
+const writer4 = await parquet.ParquetWriter.openFile(schema, 'file.parquet', {
+  compression: 'BROTLI',
+  compressionLevel: 4, // 1-11
+});
+```
+### Compression Comparison
+| Algorithm        | Compression Ratio | Write Speed | Read Speed | Use When                      |
+| ---------------- | ----------------- | ----------- | ---------- | ----------------------------- |
+| **UNCOMPRESSED** | 1x                | Fastest     | Fastest    | Local testing only            |
+| **SNAPPY**       | 2-3x              | Very Fast   | Very Fast  | General purpose (recommended) |
+| **GZIP**         | 3-4x              | Moderate    | Fast       | Storage optimization          |
+| **BROTLI**       | 4-5x              | Slow        | Fast       | Maximum compression needed    |
+**Recommendation**: Use **SNAPPY** for most cases - excellent balance of speed and compression.
+---
+## Integration with SDK Extraction
+### Pattern: GraphQL → Transform → Parquet
+```typescript
+import { createClient } from '@fluentcommerce/fc-connect-sdk';
+import * as parquet from 'parquetjs';
+/**
+ * Extract inventory from Fluent Commerce and save as Parquet
+ * Business use case: Daily inventory snapshot for analytics
+ */
+async function extractToParquet(outputFile: string) {
+  // 1. Setup SDK client
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!,
+    },
+  });
+  // 2. Define Parquet schema
+  const schema = new parquet.ParquetSchema({
+    ref: { type: 'UTF8' },
+    productRef: { type: 'UTF8' },
+    locationRef: { type: 'UTF8' },
+    qty: { type: 'INT32' },
+    status: { type: 'UTF8', optional: true },
+    extractedAt: { type: 'TIMESTAMP_MILLIS' },
+  });
+  // 3. Create Parquet writer
+  const writer = await parquet.ParquetWriter.openFile(schema, outputFile, {
+    compression: 'SNAPPY',
+    rowGroupSize: 10000,
+  });
+  // 4. Extract data with pagination
+  let hasNextPage = true;
+  let cursor: string | null = null;
+  let totalRecords = 0;
+  while (hasNextPage) {
+    const result = await client.graphql({
+      query: `
+        query ExtractInventory($after: String, $first: Int!) {
+          inventoryPositions(after: $after, first: $first) {
+            # ✅ Query.inventoryPositions (InventoryPositionConnection!)
+            pageInfo {
+              hasNextPage
+            }
+            edges {
+              cursor
+              node {
+                ref              # ✅ InventoryPosition.ref (String!)
+                productRef       # ✅ InventoryPosition.productRef (String!)
+                locationRef      # ✅ InventoryPosition.locationRef (String!)
+                qty              # ✅ InventoryPosition.qty (Int!)
+                status           # ✅ InventoryPosition.status (String)
+              }
+            }
+          }
+        }
+      `,
+      variables: {
+        first: 1000,
+        after: cursor,
+      },
+    });
+    const { edges, pageInfo } = result.data.inventoryPositions;
+    // 5. Write batch to Parquet
+    for (const edge of edges) {
+      await writer.appendRow({
+        ...edge.node,
+        extractedAt: new Date(),
+      });
+      totalRecords++;
+    }
+    console.log(`Written ${totalRecords} records...`);
+    hasNextPage = pageInfo.hasNextPage;
+    // ⚠️ Use last edge cursor (Fluent doesn't return pageInfo.endCursor)
+    cursor = edges.length > 0 ? edges[edges.length - 1].cursor : null;
+  }
+  // 6. Close writer
+  await writer.close();
+  console.log(`✅ Complete: ${totalRecords} records written to ${outputFile}`);
+}
+// Usage
+extractToParquet('inventory_2025-10-19.parquet').catch(console.error);
+```
+---
+## Batch Writing for Performance
+### Writing Large Datasets
+```typescript
+async function writeLargeDataset(records: any[], outputFile: string, batchSize: number = 10000) {
+  const schema = new parquet.ParquetSchema({
+    ref: { type: 'UTF8' },
+    productRef: { type: 'UTF8' },
+    locationRef: { type: 'UTF8' },
+    qty: { type: 'INT32' },
+  });
+  const writer = await parquet.ParquetWriter.openFile(schema, outputFile, {
+    compression: 'SNAPPY',
+    rowGroupSize: batchSize, // Optimize row group size
+  });
+  let written = 0;
+  for (let i = 0; i < records.length; i += batchSize) {
+    const batch = records.slice(i, i + batchSize);
+    for (const record of batch) {
+      await writer.appendRow(record);
+    }
+    written += batch.length;
+    console.log(`Written ${written} / ${records.length} records`);
+  }
+  await writer.close();
+  console.log(`✅ Complete: ${written} records`);
+}
+```
+---
+## S3DataSource.writeParquetContent()
+The SDK provides a convenience method to generate Parquet files as Buffers, which can then be uploaded to S3.
+### Method Signature
+```typescript
+async writeParquetContent(
+  records: ParquetDataRecord[],
+  options?: {
+    compression?: 'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI';
+    schema?: Record<string, any>;
+    rowGroupSize?: number;
+  }
+): Promise<Buffer>
+```
+### Basic Usage
+```typescript
+import { S3DataSource, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
+const logger = toStructuredLogger(createConsoleLogger(), { service: 'S3Parquet' });
+const s3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      bucket: process.env.S3_BUCKET!,
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION!,
+    },
+  },
+  logger
+);
+const records = [
+  { id: 1, name: 'Product A', qty: 100 },
+  { id: 2, name: 'Product B', qty: 200 },
+];
+// Generate Parquet Buffer
+const parquetBuffer = await s3.writeParquetContent(records, {
+  schema: {
+    id: { type: 'INT32' },
+    name: { type: 'UTF8' },
+    qty: { type: 'INT32' },
+  },
+  compression: 'SNAPPY',
+});
+// Upload to S3
+await s3.uploadFile('data/products-2025-10-30.parquet', parquetBuffer);
+console.log('✅ Parquet file written to S3');
+```
+### Complete Extraction Example
+```typescript
+import { createClient, S3DataSource, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
+async function extractToS3Parquet() {
+  const logger = toStructuredLogger(createConsoleLogger(), { service: 'Extraction' });
+  // 1. Create Fluent client
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!,
+    },
+  });
+  // 2. Setup S3
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      s3Config: {
+        bucket: process.env.S3_BUCKET!,
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        region: process.env.AWS_REGION!,
+      },
+    },
+    logger
+  );
+  // 3. Extract data with pagination
+  const allRecords = [];
+  let hasNextPage = true;
+  let cursor: string | null = null;
+  while (hasNextPage) {
+    const result = await client.graphql({
+      query: `
+        query ExtractInventory($after: String, $first: Int!) {
+          inventoryPositions(after: $after, first: $first) {
+            pageInfo { hasNextPage }
+            edges {
+              cursor
+              node {
+                ref
+                productRef
+                locationRef
+                qty
+                status
+              }
+            }
+          }
+        }
+      `,
+      variables: { first: 1000, after: cursor },
+    });
+    const { edges, pageInfo } = result.data.inventoryPositions;
+    allRecords.push(...edges.map((e) => ({
+      ...e.node,
+      extractedAt: new Date().toISOString(),
+    })));
+    hasNextPage = pageInfo.hasNextPage;
+    cursor = edges.length > 0 ? edges[edges.length - 1].cursor : null;
+  }
+  console.log(`Extracted ${allRecords.length} records`);
+  // 4. Generate Parquet Buffer and upload to S3
+  const parquetBuffer = await s3.writeParquetContent(allRecords, {
+    schema: {
+      ref: { type: 'UTF8' },
+      productRef: { type: 'UTF8' },
+      locationRef: { type: 'UTF8' },
+      qty: { type: 'INT32' },
+      status: { type: 'UTF8' },
+      extractedAt: { type: 'UTF8' },
+    },
+    compression: 'SNAPPY',
+  });
+  const s3Key = `extracts/inventory_${new Date().toISOString().split('T')[0]}.parquet`;
+  await s3.uploadFile(s3Key, parquetBuffer);
+  console.log(`✅ Complete: Data written to S3 as Parquet at ${s3Key}`);
+}
+extractToS3Parquet().catch(console.error);
+```
+### Advantages
+- ✅ **No temporary files** - Generates Buffer directly in memory
+- ✅ **Simplified workflow** - Generate Buffer, then upload
+- ✅ **Memory efficient** - No disk I/O needed for generation
+- ✅ **Flexible** - Upload to S3, SFTP, or save locally
+### Parameters
+**records** (`ParquetDataRecord[]`)
+- Array of data objects to write
+- Must match the provided schema structure
+**options.schema** (`Record<string, any>` | optional)
+- Parquet schema definition
+- If not provided, schema is auto-detected from records
+- Example: `{ id: { type: 'INT32' }, name: { type: 'UTF8' } }`
+**options.compression** (`'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI'` | optional)
+- Compression algorithm (default: `'UNCOMPRESSED'`)
+- Recommended: `'SNAPPY'` for balance of speed and compression
+**options.rowGroupSize** (`number` | optional)
+- Number of rows per row group (default: 5000)
+- Smaller = better for selective queries
+- Larger = better for full scans
+**Returns:** `Promise<Buffer>` - Parquet file content as Buffer
+**Note:** After generating the Buffer, upload it separately:
+```typescript
+const buffer = await s3.writeParquetContent(records, options);
+await s3.uploadFile('path/to/file.parquet', buffer);
+```
+---
+## Key Takeaways
+- 🎯 **SDK reads, parquetjs writes** - Use both together for complete workflows
+- 🎯 **S3DataSource.writeParquetContent()** - Generates Parquet Buffer (then upload separately)
+- 🎯 **No temporary files** - Generates Buffer directly in memory
+- 🎯 **SNAPPY compression** - Best default choice for most use cases
+- 🎯 **Schema optional** - Auto-detected if not provided
+- 🎯 **Row groups** - Optimize size for query patterns (10K-50K typical)
+- 🎯 **Integrate with SDK** - Extract from Fluent → Write to Parquet
+---
+## Next Steps
+Continue to [Module 5: GraphQL Extraction](./03-pattern-guides-parquet-05-graphql-extraction.md/) to learn complete extraction workflows from Fluent Commerce to Parquet.
+---
+[← Previous: Module 3 - Reading Parquet](./03-pattern-guides-parquet-03-reading-parquet.md) | [Back to Guide](../parquet-readme.md) | [Next: Module 5 - GraphQL Extraction →](./03-pattern-guides-parquet-05-graphql-extraction.md/)
+## Related Documentation
+- [Module 5: GraphQL Extraction](./03-pattern-guides-parquet-05-graphql-extraction.md/) - Complete extraction workflows
+- [Data Extraction Guide](../../extraction/) - GraphQL extraction patterns
+- [parquetjs Documentation](https://github.com/ironSource/parquetjs) - Official library docs