@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md

@@ -1,447 +1,447 @@
-# Pattern: Pagination & Streaming
-
-**FC Connect SDK Use Case Guide**
-
-> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
-
-**Context**: Focused guide on handling large datasets with pagination and streaming
-
-**Type**: Reusable Pattern
-
-**Complexity**: Medium
-
-**Lines**: ~180 lines
-
-## When to Use This Pattern
-
-- GraphQL queries returning large result sets (1000+ records)
-- CSV/Parquet file processing (multi-GB files)
-- S3 multipart downloads
-- Large batch operations
-- Memory-constrained environments (Lambda, containers)
-- Real-time data processing with progress tracking
-
-## SDK Methods
-
-### Pagination Helpers
-
-- `detectPaginationVariables(query)` - Auto-detect pagination variables in query
-- `extractConnection(response)` - Extract Relay connection from response
-- `hasMorePages(pageInfo, direction)` - Check for next/previous page
-- `extractCursor(connection, direction)` - Get pagination cursor
-- `buildPaginationVariables(variables, cursor, direction)` - Build next page variables
-
-### Streaming APIs
-
-- `CSVParserService.parseStreaming(content, options, batchSize)` - Stream CSV records
-- `CSVParserService.parseStreaming(stream, options, batchSize)` - Stream from Node.js stream or string
-- `ParquetParserService.parseStreaming(buffer, fileName)` - Stream Parquet records
-- `S3DataSource.downloadFile(key, { encoding: 'binary' })` - Download with streaming
-
-## Pattern Examples
-
-### 1. GraphQL Auto-Pagination Loop
-
-Automatic pagination with safety limits and progress tracking
-
-```typescript
-// FC Connect SDK (latest)
-// Install: npm install @fluentcommerce/fc-connect-sdk@latest
-// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
-// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
-import { createClient } from '@fluentcommerce/fc-connect-sdk';
-
-const client = await createClient({
-  config: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-});
-
-// Auto-pagination with safety limits
-const result = await client.graphql({
-  query: `query GetOrders($first: Int!, $after: String) {
-      orders(first: $first, after: $after) {
-        edges {
-          cursor
-          node {
-            id
-            ref
-            status
-            totalPrice
-          }
-        }
-        pageInfo {
-          hasNextPage
-        }
-      }
-    }`,
-  variables: { first: 100 },
-  pagination: {
-    maxPages: 50, // Stop after 50 pages
-    maxRecords: 5000, // OR stop after 5000 records
-    timeoutMs: 60000, // OR stop after 60 seconds
-  },
-});
-
-// All pages merged into single response
-const allOrders = result.data.orders.edges.map(e => e.node);
-
-// Check pagination metadata
-console.log(result.extensions.autoPagination);
-// { totalPages: 23, totalRecords: 2300, truncated: false }
-```
-
-### 2. Manual Cursor-Based Pagination
-
-Full control over pagination with custom logic
-
-```typescript
-import { extractConnection, hasMorePages, extractCursor } from '@fluentcommerce/fc-connect-sdk';
-
-async function fetchAllProducts(client: FluentClient): Promise<any[]> {
-  const allProducts = [];
-  let cursor = null;
-  let hasMore = true;
-
-  const query = `query GetProducts($first: Int!, $after: String) {
-      products(first: $first, after: $after) {
-        edges {
-          cursor
-          node { id ref name }
-        }
-        pageInfo { hasNextPage }
-      }
-    }`;
-
-  while (hasMore) {
-    const result = await client.graphql({
-      query,
-      variables: { first: 200, after: cursor },
-      pagination: { enabled: false }, // Manual control
-    });
-
-    const connection = extractConnection(result.data);
-    if (!connection) break;
-
-    // Extract nodes
-    const products = connection.edges.map(e => e.node);
-    allProducts.push(...products);
-
-    // Check for next page
-    hasMore = hasMorePages(connection.pageInfo, 'forward');
-    cursor = extractCursor(connection, 'forward');
-
-    console.log(`Fetched ${products.length}, total: ${allProducts.length}`);
-  }
-
-  return allProducts;
-}
-```
-
-### 3. CSV Streaming (Memory-Efficient)
-
-Stream CSV records without loading entire file into memory
-
-```typescript
-import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
-import fs from 'fs';
-
-const parser = new CSVParserService();
-
-// Stream from file, process in batches of 100
-const fileStream = fs.createReadStream('large-inventory.csv');
-let totalProcessed = 0;
-
-for await (const batch of parser.parseStreamStreaming(fileStream, {}, 100)) {
-  // Process batch (100 records at a time)
-  await processBatch(batch);
-  totalProcessed += batch.length;
-  console.log(`Processed ${totalProcessed} records`);
-}
-
-async function processBatch(records: any[]) {
-  // Send to Fluent Batch API, transform, etc.
-  for (const record of records) {
-    // Process individual record
-  }
-}
-```
-
-### 4. Parquet Streaming with Cancellation
-
-Stream Parquet files with progress tracking and cancellation support
-
-```typescript
-import { ParquetParserService } from '@fluentcommerce/fc-connect-sdk';
-
-const parquetParser = new ParquetParserService(logger);
-
-// Download Parquet file as buffer
-const buffer = await s3DataSource.downloadFile('inventory.parquet', {
-  encoding: 'binary',
-});
-
-// Stream with cancellation
-const { records, stats, cancel } = await parquetParser.parseStreaming(buffer, 'inventory.parquet', {
-  batchSize: 1000,
-});
-
-// Process batches asynchronously
-let batchNumber = 0;
-try {
-  for await (const batch of records) {
-    batchNumber++;
-    console.log(`Processing batch ${batchNumber}: ${batch.length} records`);
-
-    // Process batch
-    await sendToFluentAPI(batch);
-
-    // Cancel if taking too long
-    if (batchNumber > 100) {
-      cancel();
-      break;
-    }
-  }
-} catch (error) {
-  console.error('Streaming failed:', error);
-}
-
-// Get final statistics
-const finalStats = await stats;
-console.log(`Total records: ${finalStats.totalRecords}`);
-console.log(`Processing time: ${finalStats.processingTimeMs}ms`);
-```
-
-### 5. S3 Multipart Streaming
-
-Download large S3 files with streaming to avoid memory issues
-
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const s3 = new S3DataSource(s3Config, logger);
-
-// Download as binary buffer for streaming to parser
-const fileBuffer = await s3.downloadFile('large-file.parquet', {
-  encoding: 'binary',
-});
-
-// Stream to parser (avoids double memory usage)
-const parquetParser = new ParquetParserService(logger);
-const { records } = await parquetParser.parseStreaming(fileBuffer, 'large-file.parquet');
-
-// Process records as they stream
-for await (const batch of records) {
-  // Process without accumulating in memory
-  await processRecords(batch);
-}
-```
-
-### 6. Conditional Pagination (Stop Early)
-
-Stop pagination when specific condition is met
-
-```typescript
-async function fetchUntilCondition(
-  client: FluentClient,
-  condition: (item: any) => boolean
-): Promise<any[]> {
-  const results = [];
-  let cursor = null;
-  let hasMore = true;
-
-  const query = `query GetItems($first: Int!, $after: String) {
-      items(first: $first, after: $after) {
-        edges {
-          cursor
-          node { id status createdAt }
-        }
-        pageInfo { hasNextPage }
-      }
-    }`;
-
-  while (hasMore) {
-    const result = await client.graphql({
-      query,
-      variables: { first: 100, after: cursor },
-      pagination: { enabled: false },
-    });
-
-    const items = result.data?.items;
-    if (!items) break;
-
-    // Check each item
-    for (const edge of items.edges) {
-      results.push(edge.node);
-
-      // Stop early if condition met
-      if (condition(edge.node)) {
-        console.log('Condition met, stopping pagination');
-        return results;
-      }
-    }
-
-    hasMore = items.pageInfo.hasNextPage;
-    const lastEdge = items.edges[items.edges.length - 1];
-    cursor = lastEdge?.cursor;
-  }
-
-  return results;
-}
-
-// Usage: Find first CRITICAL item
-const items = await fetchUntilCondition(client, item => item.status === 'CRITICAL');
-```
-
-### 7. Batch Processing with Progress
-
-Process large datasets in batches with progress tracking
-
-```typescript
-async function processBatchesWithProgress(client: FluentClient, batchSize: number = 100) {
-  let processedCount = 0;
-  let pageCount = 0;
-  let cursor = null;
-  let hasMore = true;
-
-  const query = `query GetInventory($first: Int!, $after: String) {
-      inventoryPositions(first: $first, after: $after) {
-        edges {
-          cursor
-          node {
-            ref
-            productRef
-            qty
-          }
-        }
-        pageInfo { hasNextPage }
-      }
-    }`;
-
-  while (hasMore) {
-    const result = await client.graphql({
-      query,
-      variables: { first: batchSize, after: cursor },
-      pagination: { enabled: false },
-    });
-
-    const positions = result.data?.inventoryPositions;
-    if (!positions) break;
-
-    const batch = positions.edges.map(e => e.node);
-    pageCount++;
-
-    // Process batch
-    console.log(`Processing batch ${pageCount} (${batch.length} items)`);
-    await updateInventory(batch);
-    processedCount += batch.length;
-
-    // Update progress
-    console.log(`Progress: ${processedCount} items processed`);
-
-    hasMore = positions.pageInfo.hasNextPage;
-    const lastEdge = positions.edges[positions.edges.length - 1];
-    cursor = lastEdge?.cursor;
-
-    // Add delay to avoid rate limiting
-    if (hasMore) {
-      await new Promise(resolve => setTimeout(resolve, 100));
-    }
-  }
-
-  console.log(`Complete: Processed ${processedCount} items in ${pageCount} batches`);
-}
-```
-
-## Common Patterns
-
-### Pattern 1: Auto-Pagination with Limits
-
-**Use Case**: Fetch all records with safety guardrails
-
-**Key**: Use `pagination` config with `maxPages`, `maxRecords`, `timeoutMs`
-
-### Pattern 2: Manual Pagination Loop
-
-**Use Case**: Full control over pagination logic
-
-**Key**: Disable auto-pagination, use helper functions for cursor management
-
-### Pattern 3: CSV/Parquet Streaming
-
-**Use Case**: Process large files without memory issues
-
-**Key**: Use `parseStreaming()` or `parseStreamStreaming()` with batching
-
-### Pattern 4: Conditional Pagination
-
-**Use Case**: Stop fetching when specific item found
-
-**Key**: Manual loop with early break condition
-
-### Pattern 5: Batch Processing
-
-**Use Case**: Process records in chunks as they arrive
-
-**Key**: Combine pagination with batch processing logic
-
-### Pattern 6: Progress Tracking
-
-**Use Case**: Monitor long-running operations
-
-**Key**: Use `onProgress` callback or manual logging
-
-### Pattern 7: Rate Limiting
-
-**Use Case**: Avoid API rate limits during pagination
-
-**Key**: Add `delayMs` between pages or manual `setTimeout()`
-
-## Performance Tips
-
-### Memory Management
-
-- **Stream instead of load**: Use streaming APIs for files >100MB
-- **Batch processing**: Process records in batches (100-1000) to allow GC
-- **Early termination**: Stop pagination early when condition met
-- **Buffer reuse**: Don't accumulate all results in memory if not needed
-
-### Pagination Optimization
-
-- **Page size tuning**:
-  - Small objects (IDs only): 500-1000 per page
-  - Medium objects: 100-250 per page
-  - Large nested objects: 25-50 per page
-- **Parallel queries**: Paginate multiple queries in parallel with `Promise.all()`
-- **Connection pooling**: Reuse HTTP connections for sequential requests
-
-### Error Handling
-
-- **Retry on failure**: Implement exponential backoff for transient errors
-- **Checkpoint progress**: Save cursor position for resumable pagination
-- **Validate responses**: Check for errors in each page before continuing
-
-### Common Pitfalls
-
-1. **Loading entire dataset in memory** - Use streaming APIs
-2. **Too large page sizes** - Causes timeouts and memory issues
-3. **No progress tracking** - Makes debugging difficult
-4. **Ignoring safety limits** - Can cause runaway queries
-5. **Not handling empty pages** - Check for null/undefined connections
-
-## Related Patterns
-
-- **Batch Processing**: Combine with pagination for large-scale operations
-- **State Management**: Track processed files to prevent duplicates
-- **Error Recovery**: Implement retry logic with pagination checkpoints
-
-## See Also
-
-- [Auto-Pagination Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
-- [Pagination Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
-- [CSV Parser Documentation](../../02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md)
-- [Parquet Parser Documentation](../../02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md)
+# Pattern: Pagination & Streaming
+
+**FC Connect SDK Use Case Guide**
+
+> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
+
+**Context**: Focused guide on handling large datasets with pagination and streaming
+
+**Type**: Reusable Pattern
+
+**Complexity**: Medium
+
+**Lines**: ~180 lines
+
+## When to Use This Pattern
+
+- GraphQL queries returning large result sets (1000+ records)
+- CSV/Parquet file processing (multi-GB files)
+- S3 multipart downloads
+- Large batch operations
+- Memory-constrained environments (Lambda, containers)
+- Real-time data processing with progress tracking
+
+## SDK Methods
+
+### Pagination Helpers
+
+- `detectPaginationVariables(query)` - Auto-detect pagination variables in query
+- `extractConnection(response)` - Extract Relay connection from response
+- `hasMorePages(pageInfo, direction)` - Check for next/previous page
+- `extractCursor(connection, direction)` - Get pagination cursor
+- `buildPaginationVariables(variables, cursor, direction)` - Build next page variables
+
+### Streaming APIs
+
+- `CSVParserService.parseStreaming(content, options, batchSize)` - Stream CSV records
+- `CSVParserService.parseStreaming(stream, options, batchSize)` - Stream from Node.js stream or string
+- `ParquetParserService.parseStreaming(buffer, fileName)` - Stream Parquet records
+- `S3DataSource.downloadFile(key, { encoding: 'binary' })` - Download with streaming
+
+## Pattern Examples
+
+### 1. GraphQL Auto-Pagination Loop
+
+Automatic pagination with safety limits and progress tracking
+
+```typescript
+// FC Connect SDK (latest)
+// Install: npm install @fluentcommerce/fc-connect-sdk@latest
+// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
+// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
+import { createClient } from '@fluentcommerce/fc-connect-sdk';
+
+const client = await createClient({
+  config: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+});
+
+// Auto-pagination with safety limits
+const result = await client.graphql({
+  query: `query GetOrders($first: Int!, $after: String) {
+      orders(first: $first, after: $after) {
+        edges {
+          cursor
+          node {
+            id
+            ref
+            status
+            totalPrice
+          }
+        }
+        pageInfo {
+          hasNextPage
+        }
+      }
+    }`,
+  variables: { first: 100 },
+  pagination: {
+    maxPages: 50, // Stop after 50 pages
+    maxRecords: 5000, // OR stop after 5000 records
+    timeoutMs: 60000, // OR stop after 60 seconds
+  },
+});
+
+// All pages merged into single response
+const allOrders = result.data.orders.edges.map(e => e.node);
+
+// Check pagination metadata
+console.log(result.extensions.autoPagination);
+// { totalPages: 23, totalRecords: 2300, truncated: false }
+```
+
+### 2. Manual Cursor-Based Pagination
+
+Full control over pagination with custom logic
+
+```typescript
+import { extractConnection, hasMorePages, extractCursor } from '@fluentcommerce/fc-connect-sdk';
+
+async function fetchAllProducts(client: FluentClient): Promise<any[]> {
+  const allProducts = [];
+  let cursor = null;
+  let hasMore = true;
+
+  const query = `query GetProducts($first: Int!, $after: String) {
+      products(first: $first, after: $after) {
+        edges {
+          cursor
+          node { id ref name }
+        }
+        pageInfo { hasNextPage }
+      }
+    }`;
+
+  while (hasMore) {
+    const result = await client.graphql({
+      query,
+      variables: { first: 200, after: cursor },
+      pagination: { enabled: false }, // Manual control
+    });
+
+    const connection = extractConnection(result.data);
+    if (!connection) break;
+
+    // Extract nodes
+    const products = connection.edges.map(e => e.node);
+    allProducts.push(...products);
+
+    // Check for next page
+    hasMore = hasMorePages(connection.pageInfo, 'forward');
+    cursor = extractCursor(connection, 'forward');
+
+    console.log(`Fetched ${products.length}, total: ${allProducts.length}`);
+  }
+
+  return allProducts;
+}
+```
+
+### 3. CSV Streaming (Memory-Efficient)
+
+Stream CSV records without loading entire file into memory
+
+```typescript
+import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
+import fs from 'fs';
+
+const parser = new CSVParserService();
+
+// Stream from file, process in batches of 100
+const fileStream = fs.createReadStream('large-inventory.csv');
+let totalProcessed = 0;
+
+for await (const batch of parser.parseStreamStreaming(fileStream, {}, 100)) {
+  // Process batch (100 records at a time)
+  await processBatch(batch);
+  totalProcessed += batch.length;
+  console.log(`Processed ${totalProcessed} records`);
+}
+
+async function processBatch(records: any[]) {
+  // Send to Fluent Batch API, transform, etc.
+  for (const record of records) {
+    // Process individual record
+  }
+}
+```
+
+### 4. Parquet Streaming with Cancellation
+
+Stream Parquet files with progress tracking and cancellation support
+
+```typescript
+import { ParquetParserService } from '@fluentcommerce/fc-connect-sdk';
+
+const parquetParser = new ParquetParserService(logger);
+
+// Download Parquet file as buffer
+const buffer = await s3DataSource.downloadFile('inventory.parquet', {
+  encoding: 'binary',
+});
+
+// Stream with cancellation
+const { records, stats, cancel } = await parquetParser.parseStreaming(buffer, 'inventory.parquet', {
+  batchSize: 1000,
+});
+
+// Process batches asynchronously
+let batchNumber = 0;
+try {
+  for await (const batch of records) {
+    batchNumber++;
+    console.log(`Processing batch ${batchNumber}: ${batch.length} records`);
+
+    // Process batch
+    await sendToFluentAPI(batch);
+
+    // Cancel if taking too long
+    if (batchNumber > 100) {
+      cancel();
+      break;
+    }
+  }
+} catch (error) {
+  console.error('Streaming failed:', error);
+}
+
+// Get final statistics
+const finalStats = await stats;
+console.log(`Total records: ${finalStats.totalRecords}`);
+console.log(`Processing time: ${finalStats.processingTimeMs}ms`);
+```
+
+### 5. S3 Multipart Streaming
+
+Download large S3 files with streaming to avoid memory issues
+
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const s3 = new S3DataSource(s3Config, logger);
+
+// Download as binary buffer for streaming to parser
+const fileBuffer = await s3.downloadFile('large-file.parquet', {
+  encoding: 'binary',
+});
+
+// Stream to parser (avoids double memory usage)
+const parquetParser = new ParquetParserService(logger);
+const { records } = await parquetParser.parseStreaming(fileBuffer, 'large-file.parquet');
+
+// Process records as they stream
+for await (const batch of records) {
+  // Process without accumulating in memory
+  await processRecords(batch);
+}
+```
+
+### 6. Conditional Pagination (Stop Early)
+
+Stop pagination when specific condition is met
+
+```typescript
+async function fetchUntilCondition(
+  client: FluentClient,
+  condition: (item: any) => boolean
+): Promise<any[]> {
+  const results = [];
+  let cursor = null;
+  let hasMore = true;
+
+  const query = `query GetItems($first: Int!, $after: String) {
+      items(first: $first, after: $after) {
+        edges {
+          cursor
+          node { id status createdAt }
+        }
+        pageInfo { hasNextPage }
+      }
+    }`;
+
+  while (hasMore) {
+    const result = await client.graphql({
+      query,
+      variables: { first: 100, after: cursor },
+      pagination: { enabled: false },
+    });
+
+    const items = result.data?.items;
+    if (!items) break;
+
+    // Check each item
+    for (const edge of items.edges) {
+      results.push(edge.node);
+
+      // Stop early if condition met
+      if (condition(edge.node)) {
+        console.log('Condition met, stopping pagination');
+        return results;
+      }
+    }
+
+    hasMore = items.pageInfo.hasNextPage;
+    const lastEdge = items.edges[items.edges.length - 1];
+    cursor = lastEdge?.cursor;
+  }
+
+  return results;
+}
+
+// Usage: Find first CRITICAL item
+const items = await fetchUntilCondition(client, item => item.status === 'CRITICAL');
+```
+
+### 7. Batch Processing with Progress
+
+Process large datasets in batches with progress tracking
+
+```typescript
+async function processBatchesWithProgress(client: FluentClient, batchSize: number = 100) {
+  let processedCount = 0;
+  let pageCount = 0;
+  let cursor = null;
+  let hasMore = true;
+
+  const query = `query GetInventory($first: Int!, $after: String) {
+      inventoryPositions(first: $first, after: $after) {
+        edges {
+          cursor
+          node {
+            ref
+            productRef
+            qty
+          }
+        }
+        pageInfo { hasNextPage }
+      }
+    }`;
+
+  while (hasMore) {
+    const result = await client.graphql({
+      query,
+      variables: { first: batchSize, after: cursor },
+      pagination: { enabled: false },
+    });
+
+    const positions = result.data?.inventoryPositions;
+    if (!positions) break;
+
+    const batch = positions.edges.map(e => e.node);
+    pageCount++;
+
+    // Process batch
+    console.log(`Processing batch ${pageCount} (${batch.length} items)`);
+    await updateInventory(batch);
+    processedCount += batch.length;
+
+    // Update progress
+    console.log(`Progress: ${processedCount} items processed`);
+
+    hasMore = positions.pageInfo.hasNextPage;
+    const lastEdge = positions.edges[positions.edges.length - 1];
+    cursor = lastEdge?.cursor;
+
+    // Add delay to avoid rate limiting
+    if (hasMore) {
+      await new Promise(resolve => setTimeout(resolve, 100));
+    }
+  }
+
+  console.log(`Complete: Processed ${processedCount} items in ${pageCount} batches`);
+}
+```
+
+## Common Patterns
+
+### Pattern 1: Auto-Pagination with Limits
+
+**Use Case**: Fetch all records with safety guardrails
+
+**Key**: Use `pagination` config with `maxPages`, `maxRecords`, `timeoutMs`
+
+### Pattern 2: Manual Pagination Loop
+
+**Use Case**: Full control over pagination logic
+
+**Key**: Disable auto-pagination, use helper functions for cursor management
+
+### Pattern 3: CSV/Parquet Streaming
+
+**Use Case**: Process large files without memory issues
+
+**Key**: Use `parseStreaming()` or `parseStreamStreaming()` with batching
+
+### Pattern 4: Conditional Pagination
+
+**Use Case**: Stop fetching when specific item found
+
+**Key**: Manual loop with early break condition
+
+### Pattern 5: Batch Processing
+
+**Use Case**: Process records in chunks as they arrive
+
+**Key**: Combine pagination with batch processing logic
+
+### Pattern 6: Progress Tracking
+
+**Use Case**: Monitor long-running operations
+
+**Key**: Use `onProgress` callback or manual logging
+
+### Pattern 7: Rate Limiting
+
+**Use Case**: Avoid API rate limits during pagination
+
+**Key**: Add `delayMs` between pages or manual `setTimeout()`
+
+## Performance Tips
+
+### Memory Management
+
+- **Stream instead of load**: Use streaming APIs for files >100MB
+- **Batch processing**: Process records in batches (100-1000) to allow GC
+- **Early termination**: Stop pagination early when condition met
+- **Buffer reuse**: Don't accumulate all results in memory if not needed
+
+### Pagination Optimization
+
+- **Page size tuning**:
+  - Small objects (IDs only): 500-1000 per page
+  - Medium objects: 100-250 per page
+  - Large nested objects: 25-50 per page
+- **Parallel queries**: Paginate multiple queries in parallel with `Promise.all()`
+- **Connection pooling**: Reuse HTTP connections for sequential requests
+
+### Error Handling
+
+- **Retry on failure**: Implement exponential backoff for transient errors
+- **Checkpoint progress**: Save cursor position for resumable pagination
+- **Validate responses**: Check for errors in each page before continuing
+
+### Common Pitfalls
+
+1. **Loading entire dataset in memory** - Use streaming APIs
+2. **Too large page sizes** - Causes timeouts and memory issues
+3. **No progress tracking** - Makes debugging difficult
+4. **Ignoring safety limits** - Can cause runaway queries
+5. **Not handling empty pages** - Check for null/undefined connections
+
+## Related Patterns
+
+- **Batch Processing**: Combine with pagination for large-scale operations
+- **State Management**: Track processed files to prevent duplicates
+- **Error Recovery**: Implement retry logic with pagination checkpoints
+
+## See Also
+
+- [Auto-Pagination Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
+- [Pagination Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
+- [CSV Parser Documentation](../../02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md)
+- [Parquet Parser Documentation](../../02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md)