@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md

@@ -1,1140 +1,1140 @@
-# Standalone: Fluent GraphQL → S3 Parquet Export
-
-**FC Connect SDK Use Case Guide**
-
-> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
-
-**Context**: Node.js script that extracts inventory/orders from Fluent Commerce via GraphQL and exports to S3 as Parquet files
-
-**Complexity**: Medium
-
-**Runtime**: Node.js ≥18 / Deno
-
-**Estimated Lines**: ~500 lines
-
-## What You'll Build
-
-- Standalone Node.js/Deno extraction script
-- OAuth2 authentication with Fluent Commerce
-- GraphQL query execution with auto-pagination
-- Data transformation with UniversalMapper
-- Parquet file generation
-- S3 upload with metadata
-- Incremental extraction (last updated timestamp tracking)
-- Error handling and performance metrics
-- Scheduled execution support
-
-## SDK Methods Used
-
-- `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client creation
-- `client.graphql({ query, variables, pagination })` - Execute GraphQL query with auto-pagination
-- `UniversalMapper(extractionConfig)` - Transform data for export format
-- `ParquetParserService.writeParquetContent(data, options)` - Generate Parquet buffer
-- `S3DataSource.uploadFile(key, buffer, options)` - Upload to S3
-- `StateService` - Track incremental extraction state
-
-## Complete Working Code
-
-### 1. Project Setup
-
-```bash
-# Initialize project
-mkdir fluent-graphql-export
-cd fluent-graphql-export
-npm init -y
-
-# Install dependencies
-npm install @fluentcommerce/fc-connect-sdk
-npm install dotenv
-
-# Create directory structure
-mkdir -p src config logs state
-```
-
-### 2. Environment Configuration
-
-Create `.env`:
-
-```bash
-# Fluent Commerce API
-FLUENT_BASE_URL=https://your-account.api.fluentretail.com
-FLUENT_CLIENT_ID=your-oauth-client-id
-FLUENT_CLIENT_SECRET=your-oauth-client-secret
-FLUENT_RETAILER_ID=your-retailer-id
-
-# AWS S3 (Target)
-TARGET_AWS_ACCESS_KEY_ID=your-aws-access-key
-TARGET_AWS_SECRET_ACCESS_KEY=your-aws-secret-key
-TARGET_AWS_REGION=us-east-1
-TARGET_S3_BUCKET=fluent-exports
-
-# Export Configuration
-EXPORT_ENTITY_TYPE=inventory  # or 'orders', 'products'
-EXPORT_PAGE_SIZE=100
-EXPORT_MAX_RECORDS=10000
-EXPORT_INCREMENTAL=true
-STATE_FILE=./state/last-export.json
-LOG_LEVEL=info
-```
-
-### 3. Extraction Configuration
-
-Create `config/extraction-mappings.json`:
-
-```json
-{
-  "inventory": {
-    "fields": {
-      "sku": { "source": "productRef", "resolver": "sdk.uppercase" },
-      "location_code": { "source": "locationRef", "resolver": "sdk.toString" },
-      "quantity": { "source": "qty", "resolver": "sdk.parseInt" },
-      "available_qty": { "source": "availableQty", "resolver": "sdk.parseInt" },
-      "reserved_qty": { "source": "reservedQty", "resolver": "sdk.parseInt" },
-      "status": { "source": "status", "resolver": "sdk.lowercase" },
-      "last_updated": { "source": "updatedOn", "resolver": "sdk.formatDate" },
-      "extracted_at": {
-        "source": null,
-        "resolver": "custom.timestamp",
-        "defaultValue": "{{ now }}"
-      }
-    }
-  },
-  "orders": {
-    "fields": {
-      "order_id": { "source": "ref", "resolver": "sdk.toString" },
-      "order_status": { "source": "status", "resolver": "sdk.lowercase" },
-      "total_amount": { "source": "totalPrice", "resolver": "sdk.parseFloat" },
-      "tax_amount": { "source": "totalTaxPrice", "resolver": "sdk.parseFloat" },
-      "customer_email": { "source": "customer.email", "resolver": "sdk.toString" },
-      "customer_name": {
-        "source": null,
-        "resolver": "custom.fullName"
-      },
-      "created_date": { "source": "createdOn", "resolver": "sdk.formatDate" },
-      "item_count": {
-        "source": "items.edges",
-        "resolver": "custom.arrayLength"
-      }
-    }
-  }
-}
-```
-
-### 4. GraphQL Queries
-
-Create `config/queries.json`:
-
-```json
-{
-  "inventory": {
-    "query": "query GetInventory($retailerId: ID!, $first: Int!, $after: String, $updatedFrom: DateTime) {\n  inventoryPositions(\n    retailerId: $retailerId\n    first: $first\n    after: $after\n    updatedOn: { from: $updatedFrom }\n  ) {\n    edges {\n      node {\n        id\n        ref\n        productRef\n        locationRef\n        qty\n        availableQty\n        reservedQty\n        status\n        createdOn\n        updatedOn\n      }\n      cursor\n    }\n    pageInfo {\n      hasNextPage\n    }\n  }\n}",
-    "connectionPath": "inventoryPositions"
-  },
-  "orders": {
-    "query": "query GetOrders($retailerId: ID!, $first: Int!, $after: String, $createdFrom: DateTime) {\n  orders(\n    retailerId: $retailerId\n    first: $first\n    after: $after\n    createdOn: { from: $createdFrom }\n  ) {\n    edges {\n      node {\n        id\n        ref\n        type\n        status\n        createdOn\n        totalPrice\n        totalTaxPrice\n        customer {\n          ref\n          email\n          firstName\n          lastName\n        }\n        items {\n          edges {\n            node {\n              ref\n              productRef\n              quantity\n              price\n            }\n          }\n        }\n      }\n      cursor\n    }\n    pageInfo {\n      hasNextPage\n    }\n  }\n}",
-    "connectionPath": "orders"
-  }
-}
-```
-
-### 5. Main Extraction Script
-
-Create `src/export-to-s3.ts`:
-
-> **⚠️ RUNTIME COMPATIBILITY NOTE:**
-> This template includes `import { Buffer } from 'node:buffer';` for Deno compatibility.
-> While Buffer is globally available in Node.js, including this import ensures the code works
-> across all runtimes (Node.js, Deno, Versori) without modification.
-
-```typescript
-import * as dotenv from 'dotenv';
-import * as fs from 'fs/promises';
-import * as path from 'path';
-import { Buffer } from 'node:buffer'; // Required for Deno compatibility
-import {
-  createClient,
-  FluentClient,
-  UniversalMapper,
-  ParquetParserService,
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Load environment variables
-dotenv.config();
-
-/**
- * State management for incremental extraction
- */
-interface ExportState {
-  lastExportTimestamp: string;
-  lastEntityType: string;
-  totalRecordsExported: number;
-  lastExportDuration: number;
-}
-
-class StateManager {
-  constructor(private stateFilePath: string) {}
-
-  async loadState(): Promise<ExportState | null> {
-    try {
-      const data = await fs.readFile(this.stateFilePath, 'utf-8');
-      return JSON.parse(data);
-    } catch {
-      return null;
-    }
-  }
-
-  async saveState(state: ExportState): Promise<void> {
-    const dir = path.dirname(this.stateFilePath);
-    await fs.mkdir(dir, { recursive: true });
-    await fs.writeFile(this.stateFilePath, JSON.stringify(state, null, 2));
-  }
-}
-
-/**
- * Export metrics tracking
- */
-interface ExportMetrics {
-  startTime: number;
-  endTime: number;
-  totalRecords: number;
-  totalPages: number;
-  queryTimeMs: number;
-  transformTimeMs: number;
-  parquetWriteTimeMs: number;
-  s3UploadTimeMs: number;
-  fileSizeBytes: number;
-  recordsPerSecond: number;
-  errors: string[];
-}
-
-/**
- * GraphQL to S3 Parquet Exporter
- */
-class GraphQLParquetExporter {
-  private client: FluentClient;
-  private logger: LoggingService;
-  private stateManager: StateManager;
-  private mapper: UniversalMapper;
-  private parquetParser: ParquetParserService;
-  private s3: S3DataSource;
-  private queries: Record<string, any>;
-  private mappings: Record<string, any>;
-
-  constructor() {
-    this.logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: process.env.LOG_LEVEL || 'info'
-});
-    this.stateManager = new StateManager(process.env.STATE_FILE || './state/last-export.json');
-  }
-
-  async initialize(): Promise<void> {
-    this.logger.info('Initializing GraphQL Parquet Exporter');
-
-    // Create Fluent client with OAuth2
-    this.client = await createClient({
-      config: {
-        baseUrl: process.env.FLUENT_BASE_URL!,
-        clientId: process.env.FLUENT_CLIENT_ID!,
-        clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-        retailerId: process.env.FLUENT_RETAILER_ID!,
-      },
-    });
-
-    // Load queries configuration
-    this.queries = JSON.parse(await fs.readFile('config/queries.json', 'utf-8'));
-
-    // Load extraction mappings
-    this.mappings = JSON.parse(await fs.readFile('config/extraction-mappings.json', 'utf-8'));
-
-    // Initialize Parquet parser
-    this.parquetParser = new ParquetParserService(this.logger, {
-      batchSize: 5000,
-      enableStreaming: false,
-    });
-
-    // Initialize S3 data source for target
-    this.s3 = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        connectionId: 'target-s3',
-        name: 'Export Target S3',
-        s3Config: {
-          bucket: process.env.TARGET_S3_BUCKET!,
-          region: process.env.TARGET_AWS_REGION!,
-          accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
-          secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
-        },
-      },
-      this.logger
-    );
-
-    this.logger.info('Initialization complete');
-  }
-
-  /**
-   * Execute extraction workflow
-   */
-  async export(
-    entityType: string = process.env.EXPORT_ENTITY_TYPE || 'inventory'
-  ): Promise<ExportMetrics> {
-    const metrics: ExportMetrics = {
-      startTime: Date.now(),
-      endTime: 0,
-      totalRecords: 0,
-      totalPages: 0,
-      queryTimeMs: 0,
-      transformTimeMs: 0,
-      parquetWriteTimeMs: 0,
-      s3UploadTimeMs: 0,
-      fileSizeBytes: 0,
-      recordsPerSecond: 0,
-      errors: [],
-    };
-
-    try {
-      this.logger.info(`Starting export for entity type: ${entityType}`);
-
-      // STEP 1: Determine incremental extraction timestamp
-      const fromTimestamp = await this.getIncrementalTimestamp(entityType);
-
-      // STEP 2: Execute GraphQL query with auto-pagination
-      const queryStart = Date.now();
-      const rawData = await this.executeQuery(entityType, fromTimestamp, metrics);
-      metrics.queryTimeMs = Date.now() - queryStart;
-      metrics.totalRecords = rawData.length;
-      this.logger.info(`Extracted ${rawData.length} records in ${metrics.queryTimeMs}ms`);
-
-      if (rawData.length === 0) {
-        this.logger.info('No new records to export');
-        metrics.endTime = Date.now();
-        return metrics;
-      }
-
-      // STEP 3: Transform data for export
-      const transformStart = Date.now();
-      const transformedData = await this.transformData(entityType, rawData);
-      metrics.transformTimeMs = Date.now() - transformStart;
-      this.logger.info(
-        `Transformed ${transformedData.length} records in ${metrics.transformTimeMs}ms`
-      );
-
-      // STEP 4: Generate Parquet file
-      const parquetStart = Date.now();
-      const parquetBuffer = await this.generateParquet(transformedData);
-      metrics.parquetWriteTimeMs = Date.now() - parquetStart;
-      metrics.fileSizeBytes = parquetBuffer.length;
-      this.logger.info(
-        `Generated Parquet file: ${parquetBuffer.length} bytes in ${metrics.parquetWriteTimeMs}ms`
-      );
-
-      // STEP 5: Upload to S3
-      const uploadStart = Date.now();
-      const s3Key = await this.uploadToS3(entityType, parquetBuffer, metrics);
-      metrics.s3UploadTimeMs = Date.now() - uploadStart;
-      this.logger.info(`Uploaded to S3: ${s3Key} in ${metrics.s3UploadTimeMs}ms`);
-
-      // STEP 6: Update state
-      await this.updateState(entityType, metrics);
-
-      metrics.endTime = Date.now();
-      metrics.recordsPerSecond = Math.round(
-        (metrics.totalRecords / (metrics.endTime - metrics.startTime)) * 1000
-      );
-
-      this.logger.info('Export completed successfully', {
-        entityType,
-        totalRecords: metrics.totalRecords,
-        totalPages: metrics.totalPages,
-        duration: metrics.endTime - metrics.startTime,
-        recordsPerSecond: metrics.recordsPerSecond,
-        s3Key,
-      });
-
-      return metrics;
-    } catch (error) {
-      metrics.errors.push((error as Error).message);
-      this.logger.error('Export failed', error as Error, { entityType });
-      throw error;
-    }
-  }
-
-  /**
-   * Get incremental extraction timestamp
-   */
-  private async getIncrementalTimestamp(entityType: string): Promise<string | undefined> {
-    if (process.env.EXPORT_INCREMENTAL !== 'true') {
-      this.logger.info('Incremental extraction disabled, performing full export');
-      return undefined;
-    }
-
-    const state = await this.stateManager.loadState();
-    if (!state || state.lastEntityType !== entityType) {
-      this.logger.info('No previous state found, performing full export');
-      return undefined;
-    }
-
-    this.logger.info(`Incremental extraction from: ${state.lastExportTimestamp}`);
-    return state.lastExportTimestamp;
-  }
-
-  /**
-   * Execute GraphQL query with auto-pagination
-   */
-  private async executeQuery(
-    entityType: string,
-    fromTimestamp: string | undefined,
-    metrics: ExportMetrics
-  ): Promise<any[]> {
-    const queryConfig = this.queries[entityType];
-    if (!queryConfig) {
-      throw new Error(`No query configuration found for entity type: ${entityType}`);
-    }
-
-    const variables: any = {
-      retailerId: process.env.FLUENT_RETAILER_ID!,
-      first: parseInt(process.env.EXPORT_PAGE_SIZE || '100'),
-    };
-
-    // Add incremental filter if timestamp provided
-    if (fromTimestamp) {
-      if (entityType === 'inventory') {
-        variables.updatedFrom = fromTimestamp;
-      } else if (entityType === 'orders') {
-        variables.createdFrom = fromTimestamp;
-      }
-    }
-
-    // Execute query with auto-pagination
-    const result = await this.client.graphql({
-      query: queryConfig.query,
-      variables,
-      pagination: {
-        maxRecords: parseInt(process.env.EXPORT_MAX_RECORDS || '10000'),
-      },
-    });
-
-    // Extract records from connection
-    const connectionPath = queryConfig.connectionPath;
-    const connection = result.data[connectionPath];
-    if (!connection?.edges) {
-      return [];
-    }
-
-    return connection.edges.map((edge: any) => edge.node);
-  }
-
-  /**
-   * Transform data using UniversalMapper
-   */
-  private async transformData(entityType: string, rawData: any[]): Promise<any[]> {
-    const mappingConfig = this.mappings[entityType];
-    if (!mappingConfig) {
-      throw new Error(`No mapping configuration found for entity type: ${entityType}`);
-    }
-
-    // Initialize mapper with custom resolvers
-    this.mapper = new UniversalMapper(mappingConfig, {
-      customResolvers: {
-        'custom.timestamp': () => new Date().toISOString(),
-        'custom.fullName': (value: any, sourceData: any) => {
-          const firstName = this.getNestedValue(sourceData, 'customer.firstName') || '';
-          const lastName = this.getNestedValue(sourceData, 'customer.lastName') || '';
-          return `${firstName} ${lastName}`.trim();
-        },
-        'custom.arrayLength': (value: any) => {
-          return Array.isArray(value) ? value.length : 0;
-        },
-      },
-    });
-
-    const transformed = [];
-    for (const record of rawData) {
-      const result = await this.mapper.map(record);
-      if (!result.success) {
-        this.logger.warn('Mapping errors encountered', {
-          record: record.id || record.ref,
-          errors: result.errors,
-        });
-        continue;
-      }
-      transformed.push(result.data);
-    }
-
-    return transformed;
-  }
-
-  /**
-   * Generate Parquet file buffer
-   */
-  private async generateParquet(data: any[]): Promise<Buffer> {
-    return await this.s3.writeParquetContent(data, {
-      compression: 'SNAPPY',
-      rowGroupSize: 10000,
-    });
-  }
-
-  /**
-   * Upload Parquet file to S3
-   */
-  private async uploadToS3(
-    entityType: string,
-    buffer: Buffer,
-    metrics: ExportMetrics
-  ): Promise<string> {
-    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-    const key = `exports/${entityType}/${new Date().toISOString().split('T')[0]}/${entityType}_${timestamp}.parquet`;
-
-    await this.s3.uploadFile(key, buffer, {
-      contentType: 'application/octet-stream',
-      metadata: {
-        'entity-type': entityType,
-        'record-count': metrics.totalRecords.toString(),
-        'export-timestamp': new Date().toISOString(),
-        incremental: process.env.EXPORT_INCREMENTAL || 'false',
-      },
-    });
-
-    return key;
-  }
-
-  /**
-   * Update export state
-   */
-  private async updateState(entityType: string, metrics: ExportMetrics): Promise<void> {
-    const state: ExportState = {
-      lastExportTimestamp: new Date().toISOString(),
-      lastEntityType: entityType,
-      totalRecordsExported: metrics.totalRecords,
-      lastExportDuration: metrics.endTime - metrics.startTime,
-    };
-
-    await this.stateManager.saveState(state);
-  }
-
-  /**
-   * Get nested value from object
-   */
-  private getNestedValue(obj: any, path: string): any {
-    return path.split('.').reduce((current, key) => {
-      return current?.[key];
-    }, obj);
-  }
-
-  /**
-   * Export metrics summary
-   */
-  printMetrics(metrics: ExportMetrics): void {
-    const duration = metrics.endTime - metrics.startTime;
-
-    console.log('\n=== Export Metrics ===');
-    console.log(`Total Duration: ${duration}ms`);
-    console.log(`Total Records: ${metrics.totalRecords}`);
-    console.log(`Total Pages: ${metrics.totalPages}`);
-    console.log(`Records/Second: ${metrics.recordsPerSecond}`);
-    console.log(`\nBreakdown:`);
-    console.log(
-      `  Query Time: ${metrics.queryTimeMs}ms (${Math.round((metrics.queryTimeMs / duration) * 100)}%)`
-    );
-    console.log(
-      `  Transform Time: ${metrics.transformTimeMs}ms (${Math.round((metrics.transformTimeMs / duration) * 100)}%)`
-    );
-    console.log(
-      `  Parquet Write: ${metrics.parquetWriteTimeMs}ms (${Math.round((metrics.parquetWriteTimeMs / duration) * 100)}%)`
-    );
-    console.log(
-      `  S3 Upload: ${metrics.s3UploadTimeMs}ms (${Math.round((metrics.s3UploadTimeMs / duration) * 100)}%)`
-    );
-    console.log(`\nFile Size: ${Math.round(metrics.fileSizeBytes / 1024)} KB`);
-
-    if (metrics.errors.length > 0) {
-      console.log(`\nErrors: ${metrics.errors.length}`);
-      metrics.errors.forEach((error, idx) => {
-        console.log(`  ${idx + 1}. ${error}`);
-      });
-    }
-
-    console.log('====================\n');
-  }
-}
-
-/**
- * Main execution function
- */
-async function main() {
-  const exporter = new GraphQLParquetExporter();
-
-  try {
-    await exporter.initialize();
-
-    const entityType = process.argv[2] || process.env.EXPORT_ENTITY_TYPE || 'inventory';
-    const metrics = await exporter.export(entityType);
-
-    exporter.printMetrics(metrics);
-    process.exit(0);
-  } catch (error) {
-    console.error('Export failed:', error);
-    process.exit(1);
-  }
-}
-
-// Run if executed directly
-if (require.main === module) {
-  main();
-}
-
-export { GraphQLParquetExporter };
-```
-
-## Key Patterns Explained
-
-### Pattern 1: GraphQL Query with Pagination Variables
-
-```typescript
-// Define query with pagination variables
-const query = `
-  query GetInventory($first: Int!, $after: String, $updatedFrom: DateTime) {
-    inventoryPositions(
-      first: $first
-      after: $after
-      updatedOn: { from: $updatedFrom }
-    ) {
-      edges {
-        node { id ref productRef qty }
-        cursor
-      }
-      pageInfo { hasNextPage }
-    }
-  }
-`;
-
-// SDK auto-detects $first and $after for auto-pagination
-const result = await client.graphql({
-  query,
-  variables: { first: 100, updatedFrom: '2024-01-01T00:00:00Z' },
-  pagination: { maxRecords: 10000 },
-});
-```
-
-**Why This Works:**
-
-- SDK detects `$first` and `$after` variables
-- Automatically fetches all pages until `hasNextPage = false`
-- Merges results into single response
-- Progress callbacks track pagination status
-
-### Pattern 2: Auto-Pagination Loop
-
-```typescript
-// Manual pagination (OLD WAY)
-let allRecords = [];
-let hasMore = true;
-let cursor = null;
-
-while (hasMore) {
-  const result = await client.graphql({
-    query,
-    variables: { first: 100, after: cursor },
-  });
-  allRecords.push(...result.data.inventoryPositions.edges);
-  hasMore = result.data.inventoryPositions.pageInfo.hasNextPage;
-  cursor = result.data.inventoryPositions.edges[edges.length - 1]?.cursor;
-}
-
-// Auto-pagination (NEW WAY)
-const result = await client.graphql({
-  query,
-  variables: { first: 100 },
-  pagination: {
-    maxRecords: 10000,
-  },
-});
-
-const allRecords = result.data.inventoryPositions.edges;
-```
-
-**Benefits:**
-
-- 90% less code
-- Built-in safety limits
-- Progress tracking
-- Error handling
-- Deduplication
-
-### Pattern 3: Data Transformation for Export
-
-```typescript
-// Configure extraction mapping
-const mappingConfig = {
-  fields: {
-    // Direct field mapping
-    sku: { source: 'productRef', resolver: 'sdk.uppercase' },
-
-    // Nested field access
-    customer_email: { source: 'customer.email', resolver: 'sdk.toString' },
-
-    // Type conversion
-    quantity: { source: 'qty', resolver: 'sdk.parseInt' },
-
-    // Date formatting
-    last_updated: { source: 'updatedOn', resolver: 'sdk.formatDate' },
-
-    // Custom resolver
-    full_name: {
-      source: null,
-      resolver: 'custom.fullName',
-    },
-  },
-};
-
-// Custom resolvers
-const mapper = new UniversalMapper(mappingConfig, {
-  customResolvers: {
-    'custom.fullName': (value, sourceData, config, helpers) => {
-      const firstName = sourceData.customer?.firstName || '';
-      const lastName = sourceData.customer?.lastName || '';
-      return `${firstName} ${lastName}`.trim();
-    },
-  },
-});
-
-// Transform
-const result = await mapper.map(record);
-```
-
-### Pattern 4: Parquet File Generation
-
-```typescript
-// Use S3DataSource's built-in Parquet writer
-const parquetBuffer = await s3.writeParquetContent(transformedData, {
-  compression: 'SNAPPY', // Fast compression
-  rowGroupSize: 10000, // Optimize for analytics
-});
-
-// Alternative: Use ParquetParserService directly
-const parser = new ParquetParserService(logger);
-const buffer = await parser.write(transformedData, 'output.parquet', {
-  schema: {
-    sku: { type: 'UTF8' },
-    quantity: { type: 'INT64' },
-    last_updated: { type: 'TIMESTAMP_MILLIS' },
-  },
-});
-```
-
-**Compression Options:**
-
-- `UNCOMPRESSED` - Fastest write, largest file
-- `SNAPPY` - Good balance (recommended)
-- `GZIP` - Best compression, slower
-- `BROTLI` - Best compression ratio
-
-### Pattern 5: Incremental Extraction
-
-```typescript
-// State management
-interface ExportState {
-  lastExportTimestamp: string;
-  lastEntityType: string;
-  totalRecordsExported: number;
-}
-
-// Load previous state
-const state = await loadState();
-const fromTimestamp = state?.lastExportTimestamp;
-
-// Query with incremental filter
-const result = await client.graphql({
-  query: `
-    query GetInventory($first: Int!, $after: String, $updatedFrom: DateTime) {
-      inventoryPositions(
-        first: $first
-        after: $after
-        updatedOn: { from: $updatedFrom }  # Incremental filter
-      ) {
-        edges { node { id ref qty updatedOn } }
-      }
-    }
-  `,
-  variables: {
-    first: 100,
-    updatedFrom: fromTimestamp, // Only records updated since last export
-  },
-});
-
-// Save new state
-await saveState({
-  lastExportTimestamp: new Date().toISOString(),
-  lastEntityType: entityType,
-  totalRecordsExported: result.data.inventoryPositions.edges.length,
-});
-```
-
-## Deployment Options
-
-### Local Execution
-
-```bash
-# Run once
-npm run export
-
-# Export specific entity
-npm run export orders
-
-# With custom config
-EXPORT_ENTITY_TYPE=products EXPORT_MAX_RECORDS=50000 npm run export
-```
-
-### Scheduled Export (Daily)
-
-**Option 1: Cron (Linux/Mac)**
-
-```bash
-# Edit crontab
-crontab -e
-
-# Add daily export at 2 AM
-0 2 * * * cd /path/to/fluent-graphql-export && npm run export >> logs/export.log 2>&1
-```
-
-**Option 2: Windows Task Scheduler**
-
-```powershell
-# Create scheduled task
-$action = New-ScheduledTaskAction -Execute "node" -Argument "src/export-to-s3.ts"
-$trigger = New-ScheduledTaskTrigger -Daily -At 2am
-Register-ScheduledTask -TaskName "FluentExport" -Action $action -Trigger $trigger
-```
-
-**Option 3: Node-cron (Cross-platform)**
-
-Create `src/scheduler.ts`:
-
-```typescript
-import cron from 'node-cron';
-import { GraphQLParquetExporter } from './export-to-s3';
-
-const exporter = new GraphQLParquetExporter();
-
-// Run daily at 2 AM
-cron.schedule('0 2 * * *', async () => {
-  console.log('Starting scheduled export...');
-  try {
-    await exporter.initialize();
-    await exporter.export('inventory');
-    console.log('Scheduled export completed');
-  } catch (error) {
-    console.error('Scheduled export failed:', error);
-  }
-});
-
-console.log('Scheduler started');
-```
-
-### AWS Lambda Deployment
-
-Create `lambda/handler.ts`:
-
-```typescript
-import { GraphQLParquetExporter } from '../src/export-to-s3';
-
-export const handler = async (event: any) => {
-  const exporter = new GraphQLParquetExporter();
-
-  try {
-    await exporter.initialize();
-    const entityType = event.entityType || 'inventory';
-    const metrics = await exporter.export(entityType);
-
-    return {
-      statusCode: 200,
-      body: JSON.stringify({
-        message: 'Export completed successfully',
-        metrics,
-      }),
-    };
-  } catch (error) {
-    return {
-      statusCode: 500,
-      body: JSON.stringify({
-        message: 'Export failed',
-        error: (error as Error).message,
-      }),
-    };
-  }
-};
-```
-
-**EventBridge Schedule:**
-
-```json
-{
-  "scheduleExpression": "cron(0 2 * * ? *)",
-  "target": {
-    "arn": "arn:aws:lambda:us-east-1:123456789:function:fluent-export",
-    "input": "{\"entityType\": \"inventory\"}"
-  }
-}
-```
-
-## Testing
-
-### Test Extraction Locally
-
-```bash
-# Test with small dataset
-EXPORT_MAX_RECORDS=10 EXPORT_INCREMENTAL=false npm run export
-
-# Test incremental extraction
-EXPORT_INCREMENTAL=true npm run export
-
-# Verify S3 upload
-aws s3 ls s3://fluent-exports/exports/inventory/
-
-# Download and validate Parquet
-aws s3 cp s3://fluent-exports/exports/inventory/latest.parquet test.parquet
-python -c "import pandas as pd; print(pd.read_parquet('test.parquet'))"
-```
-
-### Validation Script
-
-Create `scripts/validate-export.sh`:
-
-```bash
-#!/bin/bash
-echo "Validating export..."
-
-# Check state file
-if [ -f "./state/last-export.json" ]; then
-  echo "✓ State file exists"
-  cat ./state/last-export.json | jq .
-else
-  echo "✗ State file missing"
-  exit 1
-fi
-
-# Check S3 upload
-LATEST_FILE=$(aws s3 ls s3://fluent-exports/exports/inventory/ --recursive | sort | tail -n 1 | awk '{print $4}')
-if [ -z "$LATEST_FILE" ]; then
-  echo "✗ No files found in S3"
-  exit 1
-fi
-echo "✓ Latest export: $LATEST_FILE"
-
-# Get file size
-FILE_SIZE=$(aws s3 ls s3://fluent-exports/$LATEST_FILE | awk '{print $3}')
-echo "✓ File size: $FILE_SIZE bytes"
-
-# Check record count
-RECORD_COUNT=$(cat ./state/last-export.json | jq -r .totalRecordsExported)
-echo "✓ Records exported: $RECORD_COUNT"
-
-echo "Validation complete!"
-```
-
-## Common Issues
-
-### Issue 1: Auto-Pagination Not Triggering
-
-**Symptoms:**
-
-- Only first page of results returned
-- `autoPagination` metadata missing
-
-**Cause:**
-
-- Query missing `$first` or `$after` variables
-- Variables not passed in query execution
-
-**Solution:**
-
-```typescript
-// ✗ Wrong - no pagination variables
-const query = `query { inventoryPositions(first: 100) { edges { node { id } } } }`;
-
-// ✓ Correct - pagination variables defined
-const query = `query GetInventory($first: Int!, $after: String) {
-  inventoryPositions(first: $first, after: $after) {
-    edges { node { id } }
-    pageInfo { hasNextPage }
-  }
-}`;
-
-const result = await client.graphql({
-  query,
-  variables: { first: 100 }, // SDK adds $after automatically
-});
-```
-
-### Issue 2: Parquet File Too Large
-
-**Symptoms:**
-
-- Out of memory errors
-- Slow S3 upload
-- Lambda timeout
-
-**Solution:**
-
-```typescript
-// Use compression
-const buffer = await s3.writeParquetContent(data, {
-  compression: 'SNAPPY', // or 'GZIP' for better compression
-  rowGroupSize: 5000, // Smaller row groups
-});
-
-// Or split into multiple files
-const BATCH_SIZE = 10000;
-for (let i = 0; i < data.length; i += BATCH_SIZE) {
-  const batch = data.slice(i, i + BATCH_SIZE);
-  const buffer = await s3.writeParquetContent(batch);
-  await s3.uploadFile(`exports/part-${i}.parquet`, buffer);
-}
-```
-
-### Issue 3: Incremental Extraction Missing Records
-
-**Symptoms:**
-
-- Records not appearing in export
-- Duplicate records in subsequent exports
-
-**Cause:**
-
-- Incorrect timestamp filtering
-- Clock skew between systems
-
-**Solution:**
-
-```typescript
-// Add buffer to timestamp (5 minutes)
-const lastTimestamp = new Date(state.lastExportTimestamp);
-const fromTimestamp = new Date(lastTimestamp.getTime() - 5 * 60 * 1000).toISOString();
-
-// Use overlap detection
-const seenRecords = new Set(previousExport.map(r => r.id));
-const newRecords = currentExport.filter(r => !seenRecords.has(r.id));
-```
-
-### Issue 4: GraphQL Query Timeout
-
-**Symptoms:**
-
-- Query fails after long wait
-- Partial results returned
-
-**Solution:**
-
-```typescript
-// Use pagination safety limits
-const result = await client.graphql({
-  query,
-  variables: { first: 100 },
-  pagination: {
-    maxPages: 100, // Stop after 100 pages
-    maxRecords: 10000, // Stop after 10k records
-    timeoutMs: 300000, // Stop after 5 minutes
-  },
-});
-
-// Check for truncation
-if (result.extensions.autoPagination.truncated) {
-  console.log('Results truncated:', result.extensions.autoPagination.truncationReason);
-  // Handle partial export...
-}
-```
-
-### Issue 5: UniversalMapper Transform Failures
-
-**Symptoms:**
-
-- Some records missing from export
-- Warning logs about mapping errors
-
-**Solution:**
-
-```typescript
-// Add error handling to mapper
-const results = [];
-for (const record of rawData) {
-  const result = await mapper.map(record);
-  if (!result.success) {
-    logger.warn('Mapping failed', {
-      record: record.id,
-      errors: result.errors,
-    });
-    // Include partial data or skip
-    if (result.data) {
-      results.push({ ...result.data, _errors: result.errors });
-    }
-    continue;
-  }
-  results.push(result.data);
-}
-
-// Or use default values
-const mappingConfig = {
-  fields: {
-    qty: {
-      source: 'qty',
-      resolver: 'sdk.parseInt',
-      defaultValue: 0, // Use 0 if parsing fails
-    },
-  },
-};
-```
-
-## Related Guides
-
-- [Auto-Pagination Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Deep dive into automatic pagination
-- [Universal Mapping Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Field mapping and transformations
-- [Extraction Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - General extraction patterns
-- [Connector Scenarios Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - More complete examples
-- [Standalone CSV Ingestion](./s3-csv-batch-api.md) - Ingestion counterpart
-- [SFTP to Batch API](../../02-CORE-GUIDES/ingestion/ingestion-readme.md) - SFTP-based ingestion (see Versori templates)
-
-## Performance Optimization Tips
-
-1. **Batch Size Tuning:**
-   - Small records: 200-500 per page
-   - Large records: 50-100 per page
-   - Monitor query time vs network overhead
-
-2. **Parquet Optimization:**
-   - Use SNAPPY compression for balance
-   - Adjust rowGroupSize based on query patterns
-   - Use columnar encoding for analytics
-
-3. **Incremental Extraction:**
-   - Always use timestamp filtering when possible
-   - Add overlap window to prevent gaps
-   - Track state per entity type
-
-4. **Memory Management:**
-   - Process large exports in batches
-   - Clear data after each batch upload
-   - Use streaming for very large datasets
-
-5. **Error Recovery:**
-   - Implement retry logic with exponential backoff
-   - Save checkpoints for long-running exports
-   - Keep partial results for debugging
-
-## Production Checklist
-
-- [ ] Environment variables configured
-- [ ] S3 bucket created with proper permissions
-- [ ] OAuth2 credentials tested
-- [ ] GraphQL queries validated
-- [ ] Extraction mappings tested with sample data
-- [ ] Incremental extraction state directory created
-- [ ] Logging configured (file + console)
-- [ ] Error alerting configured
-- [ ] Scheduled execution tested
-- [ ] S3 export verified in analytics tools
-- [ ] Monitoring dashboards created
-- [ ] Backup strategy for state files
-- [ ] Documentation for operators
+# Standalone: Fluent GraphQL → S3 Parquet Export
+
+**FC Connect SDK Use Case Guide**
+
+> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
+
+**Context**: Node.js script that extracts inventory/orders from Fluent Commerce via GraphQL and exports to S3 as Parquet files
+
+**Complexity**: Medium
+
+**Runtime**: Node.js ≥18 / Deno
+
+**Estimated Lines**: ~500 lines
+
+## What You'll Build
+
+- Standalone Node.js/Deno extraction script
+- OAuth2 authentication with Fluent Commerce
+- GraphQL query execution with auto-pagination
+- Data transformation with UniversalMapper
+- Parquet file generation
+- S3 upload with metadata
+- Incremental extraction (last updated timestamp tracking)
+- Error handling and performance metrics
+- Scheduled execution support
+
+## SDK Methods Used
+
+- `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client creation
+- `client.graphql({ query, variables, pagination })` - Execute GraphQL query with auto-pagination
+- `UniversalMapper(extractionConfig)` - Transform data for export format
+- `ParquetParserService.writeParquetContent(data, options)` - Generate Parquet buffer
+- `S3DataSource.uploadFile(key, buffer, options)` - Upload to S3
+- `StateService` - Track incremental extraction state
+
+## Complete Working Code
+
+### 1. Project Setup
+
+```bash
+# Initialize project
+mkdir fluent-graphql-export
+cd fluent-graphql-export
+npm init -y
+
+# Install dependencies
+npm install @fluentcommerce/fc-connect-sdk
+npm install dotenv
+
+# Create directory structure
+mkdir -p src config logs state
+```
+
+### 2. Environment Configuration
+
+Create `.env`:
+
+```bash
+# Fluent Commerce API
+FLUENT_BASE_URL=https://your-account.api.fluentretail.com
+FLUENT_CLIENT_ID=your-oauth-client-id
+FLUENT_CLIENT_SECRET=your-oauth-client-secret
+FLUENT_RETAILER_ID=your-retailer-id
+
+# AWS S3 (Target)
+TARGET_AWS_ACCESS_KEY_ID=your-aws-access-key
+TARGET_AWS_SECRET_ACCESS_KEY=your-aws-secret-key
+TARGET_AWS_REGION=us-east-1
+TARGET_S3_BUCKET=fluent-exports
+
+# Export Configuration
+EXPORT_ENTITY_TYPE=inventory  # or 'orders', 'products'
+EXPORT_PAGE_SIZE=100
+EXPORT_MAX_RECORDS=10000
+EXPORT_INCREMENTAL=true
+STATE_FILE=./state/last-export.json
+LOG_LEVEL=info
+```
+
+### 3. Extraction Configuration
+
+Create `config/extraction-mappings.json`:
+
+```json
+{
+  "inventory": {
+    "fields": {
+      "sku": { "source": "productRef", "resolver": "sdk.uppercase" },
+      "location_code": { "source": "locationRef", "resolver": "sdk.toString" },
+      "quantity": { "source": "qty", "resolver": "sdk.parseInt" },
+      "available_qty": { "source": "availableQty", "resolver": "sdk.parseInt" },
+      "reserved_qty": { "source": "reservedQty", "resolver": "sdk.parseInt" },
+      "status": { "source": "status", "resolver": "sdk.lowercase" },
+      "last_updated": { "source": "updatedOn", "resolver": "sdk.formatDate" },
+      "extracted_at": {
+        "source": null,
+        "resolver": "custom.timestamp",
+        "defaultValue": "{{ now }}"
+      }
+    }
+  },
+  "orders": {
+    "fields": {
+      "order_id": { "source": "ref", "resolver": "sdk.toString" },
+      "order_status": { "source": "status", "resolver": "sdk.lowercase" },
+      "total_amount": { "source": "totalPrice", "resolver": "sdk.parseFloat" },
+      "tax_amount": { "source": "totalTaxPrice", "resolver": "sdk.parseFloat" },
+      "customer_email": { "source": "customer.email", "resolver": "sdk.toString" },
+      "customer_name": {
+        "source": null,
+        "resolver": "custom.fullName"
+      },
+      "created_date": { "source": "createdOn", "resolver": "sdk.formatDate" },
+      "item_count": {
+        "source": "items.edges",
+        "resolver": "custom.arrayLength"
+      }
+    }
+  }
+}
+```
+
+### 4. GraphQL Queries
+
+Create `config/queries.json`:
+
+```json
+{
+  "inventory": {
+    "query": "query GetInventory($retailerId: ID!, $first: Int!, $after: String, $updatedFrom: DateTime) {\n  inventoryPositions(\n    retailerId: $retailerId\n    first: $first\n    after: $after\n    updatedOn: { from: $updatedFrom }\n  ) {\n    edges {\n      node {\n        id\n        ref\n        productRef\n        locationRef\n        qty\n        availableQty\n        reservedQty\n        status\n        createdOn\n        updatedOn\n      }\n      cursor\n    }\n    pageInfo {\n      hasNextPage\n    }\n  }\n}",
+    "connectionPath": "inventoryPositions"
+  },
+  "orders": {
+    "query": "query GetOrders($retailerId: ID!, $first: Int!, $after: String, $createdFrom: DateTime) {\n  orders(\n    retailerId: $retailerId\n    first: $first\n    after: $after\n    createdOn: { from: $createdFrom }\n  ) {\n    edges {\n      node {\n        id\n        ref\n        type\n        status\n        createdOn\n        totalPrice\n        totalTaxPrice\n        customer {\n          ref\n          email\n          firstName\n          lastName\n        }\n        items {\n          edges {\n            node {\n              ref\n              productRef\n              quantity\n              price\n            }\n          }\n        }\n      }\n      cursor\n    }\n    pageInfo {\n      hasNextPage\n    }\n  }\n}",
+    "connectionPath": "orders"
+  }
+}
+```
+
+### 5. Main Extraction Script
+
+Create `src/export-to-s3.ts`:
+
+> **⚠️ RUNTIME COMPATIBILITY NOTE:**
+> This template includes `import { Buffer } from 'node:buffer';` for Deno compatibility.
+> While Buffer is globally available in Node.js, including this import ensures the code works
+> across all runtimes (Node.js, Deno, Versori) without modification.
+
+```typescript
+import * as dotenv from 'dotenv';
+import * as fs from 'fs/promises';
+import * as path from 'path';
+import { Buffer } from 'node:buffer'; // Required for Deno compatibility
+import {
+  createClient,
+  FluentClient,
+  UniversalMapper,
+  ParquetParserService,
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Load environment variables
+dotenv.config();
+
+/**
+ * State management for incremental extraction
+ */
+interface ExportState {
+  lastExportTimestamp: string;
+  lastEntityType: string;
+  totalRecordsExported: number;
+  lastExportDuration: number;
+}
+
+class StateManager {
+  constructor(private stateFilePath: string) {}
+
+  async loadState(): Promise<ExportState | null> {
+    try {
+      const data = await fs.readFile(this.stateFilePath, 'utf-8');
+      return JSON.parse(data);
+    } catch {
+      return null;
+    }
+  }
+
+  async saveState(state: ExportState): Promise<void> {
+    const dir = path.dirname(this.stateFilePath);
+    await fs.mkdir(dir, { recursive: true });
+    await fs.writeFile(this.stateFilePath, JSON.stringify(state, null, 2));
+  }
+}
+
+/**
+ * Export metrics tracking
+ */
+interface ExportMetrics {
+  startTime: number;
+  endTime: number;
+  totalRecords: number;
+  totalPages: number;
+  queryTimeMs: number;
+  transformTimeMs: number;
+  parquetWriteTimeMs: number;
+  s3UploadTimeMs: number;
+  fileSizeBytes: number;
+  recordsPerSecond: number;
+  errors: string[];
+}
+
+/**
+ * GraphQL to S3 Parquet Exporter
+ */
+class GraphQLParquetExporter {
+  private client: FluentClient;
+  private logger: LoggingService;
+  private stateManager: StateManager;
+  private mapper: UniversalMapper;
+  private parquetParser: ParquetParserService;
+  private s3: S3DataSource;
+  private queries: Record<string, any>;
+  private mappings: Record<string, any>;
+
+  constructor() {
+    this.logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: process.env.LOG_LEVEL || 'info'
+});
+    this.stateManager = new StateManager(process.env.STATE_FILE || './state/last-export.json');
+  }
+
+  async initialize(): Promise<void> {
+    this.logger.info('Initializing GraphQL Parquet Exporter');
+
+    // Create Fluent client with OAuth2
+    this.client = await createClient({
+      config: {
+        baseUrl: process.env.FLUENT_BASE_URL!,
+        clientId: process.env.FLUENT_CLIENT_ID!,
+        clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+        retailerId: process.env.FLUENT_RETAILER_ID!,
+      },
+    });
+
+    // Load queries configuration
+    this.queries = JSON.parse(await fs.readFile('config/queries.json', 'utf-8'));
+
+    // Load extraction mappings
+    this.mappings = JSON.parse(await fs.readFile('config/extraction-mappings.json', 'utf-8'));
+
+    // Initialize Parquet parser
+    this.parquetParser = new ParquetParserService(this.logger, {
+      batchSize: 5000,
+      enableStreaming: false,
+    });
+
+    // Initialize S3 data source for target
+    this.s3 = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        connectionId: 'target-s3',
+        name: 'Export Target S3',
+        s3Config: {
+          bucket: process.env.TARGET_S3_BUCKET!,
+          region: process.env.TARGET_AWS_REGION!,
+          accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
+          secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
+        },
+      },
+      this.logger
+    );
+
+    this.logger.info('Initialization complete');
+  }
+
+  /**
+   * Execute extraction workflow
+   */
+  async export(
+    entityType: string = process.env.EXPORT_ENTITY_TYPE || 'inventory'
+  ): Promise<ExportMetrics> {
+    const metrics: ExportMetrics = {
+      startTime: Date.now(),
+      endTime: 0,
+      totalRecords: 0,
+      totalPages: 0,
+      queryTimeMs: 0,
+      transformTimeMs: 0,
+      parquetWriteTimeMs: 0,
+      s3UploadTimeMs: 0,
+      fileSizeBytes: 0,
+      recordsPerSecond: 0,
+      errors: [],
+    };
+
+    try {
+      this.logger.info(`Starting export for entity type: ${entityType}`);
+
+      // STEP 1: Determine incremental extraction timestamp
+      const fromTimestamp = await this.getIncrementalTimestamp(entityType);
+
+      // STEP 2: Execute GraphQL query with auto-pagination
+      const queryStart = Date.now();
+      const rawData = await this.executeQuery(entityType, fromTimestamp, metrics);
+      metrics.queryTimeMs = Date.now() - queryStart;
+      metrics.totalRecords = rawData.length;
+      this.logger.info(`Extracted ${rawData.length} records in ${metrics.queryTimeMs}ms`);
+
+      if (rawData.length === 0) {
+        this.logger.info('No new records to export');
+        metrics.endTime = Date.now();
+        return metrics;
+      }
+
+      // STEP 3: Transform data for export
+      const transformStart = Date.now();
+      const transformedData = await this.transformData(entityType, rawData);
+      metrics.transformTimeMs = Date.now() - transformStart;
+      this.logger.info(
+        `Transformed ${transformedData.length} records in ${metrics.transformTimeMs}ms`
+      );
+
+      // STEP 4: Generate Parquet file
+      const parquetStart = Date.now();
+      const parquetBuffer = await this.generateParquet(transformedData);
+      metrics.parquetWriteTimeMs = Date.now() - parquetStart;
+      metrics.fileSizeBytes = parquetBuffer.length;
+      this.logger.info(
+        `Generated Parquet file: ${parquetBuffer.length} bytes in ${metrics.parquetWriteTimeMs}ms`
+      );
+
+      // STEP 5: Upload to S3
+      const uploadStart = Date.now();
+      const s3Key = await this.uploadToS3(entityType, parquetBuffer, metrics);
+      metrics.s3UploadTimeMs = Date.now() - uploadStart;
+      this.logger.info(`Uploaded to S3: ${s3Key} in ${metrics.s3UploadTimeMs}ms`);
+
+      // STEP 6: Update state
+      await this.updateState(entityType, metrics);
+
+      metrics.endTime = Date.now();
+      metrics.recordsPerSecond = Math.round(
+        (metrics.totalRecords / (metrics.endTime - metrics.startTime)) * 1000
+      );
+
+      this.logger.info('Export completed successfully', {
+        entityType,
+        totalRecords: metrics.totalRecords,
+        totalPages: metrics.totalPages,
+        duration: metrics.endTime - metrics.startTime,
+        recordsPerSecond: metrics.recordsPerSecond,
+        s3Key,
+      });
+
+      return metrics;
+    } catch (error) {
+      metrics.errors.push((error as Error).message);
+      this.logger.error('Export failed', error as Error, { entityType });
+      throw error;
+    }
+  }
+
+  /**
+   * Get incremental extraction timestamp
+   */
+  private async getIncrementalTimestamp(entityType: string): Promise<string | undefined> {
+    if (process.env.EXPORT_INCREMENTAL !== 'true') {
+      this.logger.info('Incremental extraction disabled, performing full export');
+      return undefined;
+    }
+
+    const state = await this.stateManager.loadState();
+    if (!state || state.lastEntityType !== entityType) {
+      this.logger.info('No previous state found, performing full export');
+      return undefined;
+    }
+
+    this.logger.info(`Incremental extraction from: ${state.lastExportTimestamp}`);
+    return state.lastExportTimestamp;
+  }
+
+  /**
+   * Execute GraphQL query with auto-pagination
+   */
+  private async executeQuery(
+    entityType: string,
+    fromTimestamp: string | undefined,
+    metrics: ExportMetrics
+  ): Promise<any[]> {
+    const queryConfig = this.queries[entityType];
+    if (!queryConfig) {
+      throw new Error(`No query configuration found for entity type: ${entityType}`);
+    }
+
+    const variables: any = {
+      retailerId: process.env.FLUENT_RETAILER_ID!,
+      first: parseInt(process.env.EXPORT_PAGE_SIZE || '100'),
+    };
+
+    // Add incremental filter if timestamp provided
+    if (fromTimestamp) {
+      if (entityType === 'inventory') {
+        variables.updatedFrom = fromTimestamp;
+      } else if (entityType === 'orders') {
+        variables.createdFrom = fromTimestamp;
+      }
+    }
+
+    // Execute query with auto-pagination
+    const result = await this.client.graphql({
+      query: queryConfig.query,
+      variables,
+      pagination: {
+        maxRecords: parseInt(process.env.EXPORT_MAX_RECORDS || '10000'),
+      },
+    });
+
+    // Extract records from connection
+    const connectionPath = queryConfig.connectionPath;
+    const connection = result.data[connectionPath];
+    if (!connection?.edges) {
+      return [];
+    }
+
+    return connection.edges.map((edge: any) => edge.node);
+  }
+
+  /**
+   * Transform data using UniversalMapper
+   */
+  private async transformData(entityType: string, rawData: any[]): Promise<any[]> {
+    const mappingConfig = this.mappings[entityType];
+    if (!mappingConfig) {
+      throw new Error(`No mapping configuration found for entity type: ${entityType}`);
+    }
+
+    // Initialize mapper with custom resolvers
+    this.mapper = new UniversalMapper(mappingConfig, {
+      customResolvers: {
+        'custom.timestamp': () => new Date().toISOString(),
+        'custom.fullName': (value: any, sourceData: any) => {
+          const firstName = this.getNestedValue(sourceData, 'customer.firstName') || '';
+          const lastName = this.getNestedValue(sourceData, 'customer.lastName') || '';
+          return `${firstName} ${lastName}`.trim();
+        },
+        'custom.arrayLength': (value: any) => {
+          return Array.isArray(value) ? value.length : 0;
+        },
+      },
+    });
+
+    const transformed = [];
+    for (const record of rawData) {
+      const result = await this.mapper.map(record);
+      if (!result.success) {
+        this.logger.warn('Mapping errors encountered', {
+          record: record.id || record.ref,
+          errors: result.errors,
+        });
+        continue;
+      }
+      transformed.push(result.data);
+    }
+
+    return transformed;
+  }
+
+  /**
+   * Generate Parquet file buffer
+   */
+  private async generateParquet(data: any[]): Promise<Buffer> {
+    return await this.s3.writeParquetContent(data, {
+      compression: 'SNAPPY',
+      rowGroupSize: 10000,
+    });
+  }
+
+  /**
+   * Upload Parquet file to S3
+   */
+  private async uploadToS3(
+    entityType: string,
+    buffer: Buffer,
+    metrics: ExportMetrics
+  ): Promise<string> {
+    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+    const key = `exports/${entityType}/${new Date().toISOString().split('T')[0]}/${entityType}_${timestamp}.parquet`;
+
+    await this.s3.uploadFile(key, buffer, {
+      contentType: 'application/octet-stream',
+      metadata: {
+        'entity-type': entityType,
+        'record-count': metrics.totalRecords.toString(),
+        'export-timestamp': new Date().toISOString(),
+        incremental: process.env.EXPORT_INCREMENTAL || 'false',
+      },
+    });
+
+    return key;
+  }
+
+  /**
+   * Update export state
+   */
+  private async updateState(entityType: string, metrics: ExportMetrics): Promise<void> {
+    const state: ExportState = {
+      lastExportTimestamp: new Date().toISOString(),
+      lastEntityType: entityType,
+      totalRecordsExported: metrics.totalRecords,
+      lastExportDuration: metrics.endTime - metrics.startTime,
+    };
+
+    await this.stateManager.saveState(state);
+  }
+
+  /**
+   * Get nested value from object
+   */
+  private getNestedValue(obj: any, path: string): any {
+    return path.split('.').reduce((current, key) => {
+      return current?.[key];
+    }, obj);
+  }
+
+  /**
+   * Export metrics summary
+   */
+  printMetrics(metrics: ExportMetrics): void {
+    const duration = metrics.endTime - metrics.startTime;
+
+    console.log('\n=== Export Metrics ===');
+    console.log(`Total Duration: ${duration}ms`);
+    console.log(`Total Records: ${metrics.totalRecords}`);
+    console.log(`Total Pages: ${metrics.totalPages}`);
+    console.log(`Records/Second: ${metrics.recordsPerSecond}`);
+    console.log(`\nBreakdown:`);
+    console.log(
+      `  Query Time: ${metrics.queryTimeMs}ms (${Math.round((metrics.queryTimeMs / duration) * 100)}%)`
+    );
+    console.log(
+      `  Transform Time: ${metrics.transformTimeMs}ms (${Math.round((metrics.transformTimeMs / duration) * 100)}%)`
+    );
+    console.log(
+      `  Parquet Write: ${metrics.parquetWriteTimeMs}ms (${Math.round((metrics.parquetWriteTimeMs / duration) * 100)}%)`
+    );
+    console.log(
+      `  S3 Upload: ${metrics.s3UploadTimeMs}ms (${Math.round((metrics.s3UploadTimeMs / duration) * 100)}%)`
+    );
+    console.log(`\nFile Size: ${Math.round(metrics.fileSizeBytes / 1024)} KB`);
+
+    if (metrics.errors.length > 0) {
+      console.log(`\nErrors: ${metrics.errors.length}`);
+      metrics.errors.forEach((error, idx) => {
+        console.log(`  ${idx + 1}. ${error}`);
+      });
+    }
+
+    console.log('====================\n');
+  }
+}
+
+/**
+ * Main execution function
+ */
+async function main() {
+  const exporter = new GraphQLParquetExporter();
+
+  try {
+    await exporter.initialize();
+
+    const entityType = process.argv[2] || process.env.EXPORT_ENTITY_TYPE || 'inventory';
+    const metrics = await exporter.export(entityType);
+
+    exporter.printMetrics(metrics);
+    process.exit(0);
+  } catch (error) {
+    console.error('Export failed:', error);
+    process.exit(1);
+  }
+}
+
+// Run if executed directly
+if (require.main === module) {
+  main();
+}
+
+export { GraphQLParquetExporter };
+```
+
+## Key Patterns Explained
+
+### Pattern 1: GraphQL Query with Pagination Variables
+
+```typescript
+// Define query with pagination variables
+const query = `
+  query GetInventory($first: Int!, $after: String, $updatedFrom: DateTime) {
+    inventoryPositions(
+      first: $first
+      after: $after
+      updatedOn: { from: $updatedFrom }
+    ) {
+      edges {
+        node { id ref productRef qty }
+        cursor
+      }
+      pageInfo { hasNextPage }
+    }
+  }
+`;
+
+// SDK auto-detects $first and $after for auto-pagination
+const result = await client.graphql({
+  query,
+  variables: { first: 100, updatedFrom: '2024-01-01T00:00:00Z' },
+  pagination: { maxRecords: 10000 },
+});
+```
+
+**Why This Works:**
+
+- SDK detects `$first` and `$after` variables
+- Automatically fetches all pages until `hasNextPage = false`
+- Merges results into single response
+- Progress callbacks track pagination status
+
+### Pattern 2: Auto-Pagination Loop
+
+```typescript
+// Manual pagination (OLD WAY)
+let allRecords = [];
+let hasMore = true;
+let cursor = null;
+
+while (hasMore) {
+  const result = await client.graphql({
+    query,
+    variables: { first: 100, after: cursor },
+  });
+  allRecords.push(...result.data.inventoryPositions.edges);
+  hasMore = result.data.inventoryPositions.pageInfo.hasNextPage;
+  cursor = result.data.inventoryPositions.edges[edges.length - 1]?.cursor;
+}
+
+// Auto-pagination (NEW WAY)
+const result = await client.graphql({
+  query,
+  variables: { first: 100 },
+  pagination: {
+    maxRecords: 10000,
+  },
+});
+
+const allRecords = result.data.inventoryPositions.edges;
+```
+
+**Benefits:**
+
+- 90% less code
+- Built-in safety limits
+- Progress tracking
+- Error handling
+- Deduplication
+
+### Pattern 3: Data Transformation for Export
+
+```typescript
+// Configure extraction mapping
+const mappingConfig = {
+  fields: {
+    // Direct field mapping
+    sku: { source: 'productRef', resolver: 'sdk.uppercase' },
+
+    // Nested field access
+    customer_email: { source: 'customer.email', resolver: 'sdk.toString' },
+
+    // Type conversion
+    quantity: { source: 'qty', resolver: 'sdk.parseInt' },
+
+    // Date formatting
+    last_updated: { source: 'updatedOn', resolver: 'sdk.formatDate' },
+
+    // Custom resolver
+    full_name: {
+      source: null,
+      resolver: 'custom.fullName',
+    },
+  },
+};
+
+// Custom resolvers
+const mapper = new UniversalMapper(mappingConfig, {
+  customResolvers: {
+    'custom.fullName': (value, sourceData, config, helpers) => {
+      const firstName = sourceData.customer?.firstName || '';
+      const lastName = sourceData.customer?.lastName || '';
+      return `${firstName} ${lastName}`.trim();
+    },
+  },
+});
+
+// Transform
+const result = await mapper.map(record);
+```
+
+### Pattern 4: Parquet File Generation
+
+```typescript
+// Use S3DataSource's built-in Parquet writer
+const parquetBuffer = await s3.writeParquetContent(transformedData, {
+  compression: 'SNAPPY', // Fast compression
+  rowGroupSize: 10000, // Optimize for analytics
+});
+
+// Alternative: Use ParquetParserService directly
+const parser = new ParquetParserService(logger);
+const buffer = await parser.write(transformedData, 'output.parquet', {
+  schema: {
+    sku: { type: 'UTF8' },
+    quantity: { type: 'INT64' },
+    last_updated: { type: 'TIMESTAMP_MILLIS' },
+  },
+});
+```
+
+**Compression Options:**
+
+- `UNCOMPRESSED` - Fastest write, largest file
+- `SNAPPY` - Good balance (recommended)
+- `GZIP` - Best compression, slower
+- `BROTLI` - Best compression ratio
+
+### Pattern 5: Incremental Extraction
+
+```typescript
+// State management
+interface ExportState {
+  lastExportTimestamp: string;
+  lastEntityType: string;
+  totalRecordsExported: number;
+}
+
+// Load previous state
+const state = await loadState();
+const fromTimestamp = state?.lastExportTimestamp;
+
+// Query with incremental filter
+const result = await client.graphql({
+  query: `
+    query GetInventory($first: Int!, $after: String, $updatedFrom: DateTime) {
+      inventoryPositions(
+        first: $first
+        after: $after
+        updatedOn: { from: $updatedFrom }  # Incremental filter
+      ) {
+        edges { node { id ref qty updatedOn } }
+      }
+    }
+  `,
+  variables: {
+    first: 100,
+    updatedFrom: fromTimestamp, // Only records updated since last export
+  },
+});
+
+// Save new state
+await saveState({
+  lastExportTimestamp: new Date().toISOString(),
+  lastEntityType: entityType,
+  totalRecordsExported: result.data.inventoryPositions.edges.length,
+});
+```
+
+## Deployment Options
+
+### Local Execution
+
+```bash
+# Run once
+npm run export
+
+# Export specific entity
+npm run export orders
+
+# With custom config
+EXPORT_ENTITY_TYPE=products EXPORT_MAX_RECORDS=50000 npm run export
+```
+
+### Scheduled Export (Daily)
+
+**Option 1: Cron (Linux/Mac)**
+
+```bash
+# Edit crontab
+crontab -e
+
+# Add daily export at 2 AM
+0 2 * * * cd /path/to/fluent-graphql-export && npm run export >> logs/export.log 2>&1
+```
+
+**Option 2: Windows Task Scheduler**
+
+```powershell
+# Create scheduled task
+$action = New-ScheduledTaskAction -Execute "node" -Argument "src/export-to-s3.ts"
+$trigger = New-ScheduledTaskTrigger -Daily -At 2am
+Register-ScheduledTask -TaskName "FluentExport" -Action $action -Trigger $trigger
+```
+
+**Option 3: Node-cron (Cross-platform)**
+
+Create `src/scheduler.ts`:
+
+```typescript
+import cron from 'node-cron';
+import { GraphQLParquetExporter } from './export-to-s3';
+
+const exporter = new GraphQLParquetExporter();
+
+// Run daily at 2 AM
+cron.schedule('0 2 * * *', async () => {
+  console.log('Starting scheduled export...');
+  try {
+    await exporter.initialize();
+    await exporter.export('inventory');
+    console.log('Scheduled export completed');
+  } catch (error) {
+    console.error('Scheduled export failed:', error);
+  }
+});
+
+console.log('Scheduler started');
+```
+
+### AWS Lambda Deployment
+
+Create `lambda/handler.ts`:
+
+```typescript
+import { GraphQLParquetExporter } from '../src/export-to-s3';
+
+export const handler = async (event: any) => {
+  const exporter = new GraphQLParquetExporter();
+
+  try {
+    await exporter.initialize();
+    const entityType = event.entityType || 'inventory';
+    const metrics = await exporter.export(entityType);
+
+    return {
+      statusCode: 200,
+      body: JSON.stringify({
+        message: 'Export completed successfully',
+        metrics,
+      }),
+    };
+  } catch (error) {
+    return {
+      statusCode: 500,
+      body: JSON.stringify({
+        message: 'Export failed',
+        error: (error as Error).message,
+      }),
+    };
+  }
+};
+```
+
+**EventBridge Schedule:**
+
+```json
+{
+  "scheduleExpression": "cron(0 2 * * ? *)",
+  "target": {
+    "arn": "arn:aws:lambda:us-east-1:123456789:function:fluent-export",
+    "input": "{\"entityType\": \"inventory\"}"
+  }
+}
+```
+
+## Testing
+
+### Test Extraction Locally
+
+```bash
+# Test with small dataset
+EXPORT_MAX_RECORDS=10 EXPORT_INCREMENTAL=false npm run export
+
+# Test incremental extraction
+EXPORT_INCREMENTAL=true npm run export
+
+# Verify S3 upload
+aws s3 ls s3://fluent-exports/exports/inventory/
+
+# Download and validate Parquet
+aws s3 cp s3://fluent-exports/exports/inventory/latest.parquet test.parquet
+python -c "import pandas as pd; print(pd.read_parquet('test.parquet'))"
+```
+
+### Validation Script
+
+Create `scripts/validate-export.sh`:
+
+```bash
+#!/bin/bash
+echo "Validating export..."
+
+# Check state file
+if [ -f "./state/last-export.json" ]; then
+  echo "✓ State file exists"
+  cat ./state/last-export.json | jq .
+else
+  echo "✗ State file missing"
+  exit 1
+fi
+
+# Check S3 upload
+LATEST_FILE=$(aws s3 ls s3://fluent-exports/exports/inventory/ --recursive | sort | tail -n 1 | awk '{print $4}')
+if [ -z "$LATEST_FILE" ]; then
+  echo "✗ No files found in S3"
+  exit 1
+fi
+echo "✓ Latest export: $LATEST_FILE"
+
+# Get file size
+FILE_SIZE=$(aws s3 ls s3://fluent-exports/$LATEST_FILE | awk '{print $3}')
+echo "✓ File size: $FILE_SIZE bytes"
+
+# Check record count
+RECORD_COUNT=$(cat ./state/last-export.json | jq -r .totalRecordsExported)
+echo "✓ Records exported: $RECORD_COUNT"
+
+echo "Validation complete!"
+```
+
+## Common Issues
+
+### Issue 1: Auto-Pagination Not Triggering
+
+**Symptoms:**
+
+- Only first page of results returned
+- `autoPagination` metadata missing
+
+**Cause:**
+
+- Query missing `$first` or `$after` variables
+- Variables not passed in query execution
+
+**Solution:**
+
+```typescript
+// ✗ Wrong - no pagination variables
+const query = `query { inventoryPositions(first: 100) { edges { node { id } } } }`;
+
+// ✓ Correct - pagination variables defined
+const query = `query GetInventory($first: Int!, $after: String) {
+  inventoryPositions(first: $first, after: $after) {
+    edges { node { id } }
+    pageInfo { hasNextPage }
+  }
+}`;
+
+const result = await client.graphql({
+  query,
+  variables: { first: 100 }, // SDK adds $after automatically
+});
+```
+
+### Issue 2: Parquet File Too Large
+
+**Symptoms:**
+
+- Out of memory errors
+- Slow S3 upload
+- Lambda timeout
+
+**Solution:**
+
+```typescript
+// Use compression
+const buffer = await s3.writeParquetContent(data, {
+  compression: 'SNAPPY', // or 'GZIP' for better compression
+  rowGroupSize: 5000, // Smaller row groups
+});
+
+// Or split into multiple files
+const BATCH_SIZE = 10000;
+for (let i = 0; i < data.length; i += BATCH_SIZE) {
+  const batch = data.slice(i, i + BATCH_SIZE);
+  const buffer = await s3.writeParquetContent(batch);
+  await s3.uploadFile(`exports/part-${i}.parquet`, buffer);
+}
+```
+
+### Issue 3: Incremental Extraction Missing Records
+
+**Symptoms:**
+
+- Records not appearing in export
+- Duplicate records in subsequent exports
+
+**Cause:**
+
+- Incorrect timestamp filtering
+- Clock skew between systems
+
+**Solution:**
+
+```typescript
+// Add buffer to timestamp (5 minutes)
+const lastTimestamp = new Date(state.lastExportTimestamp);
+const fromTimestamp = new Date(lastTimestamp.getTime() - 5 * 60 * 1000).toISOString();
+
+// Use overlap detection
+const seenRecords = new Set(previousExport.map(r => r.id));
+const newRecords = currentExport.filter(r => !seenRecords.has(r.id));
+```
+
+### Issue 4: GraphQL Query Timeout
+
+**Symptoms:**
+
+- Query fails after long wait
+- Partial results returned
+
+**Solution:**
+
+```typescript
+// Use pagination safety limits
+const result = await client.graphql({
+  query,
+  variables: { first: 100 },
+  pagination: {
+    maxPages: 100, // Stop after 100 pages
+    maxRecords: 10000, // Stop after 10k records
+    timeoutMs: 300000, // Stop after 5 minutes
+  },
+});
+
+// Check for truncation
+if (result.extensions.autoPagination.truncated) {
+  console.log('Results truncated:', result.extensions.autoPagination.truncationReason);
+  // Handle partial export...
+}
+```
+
+### Issue 5: UniversalMapper Transform Failures
+
+**Symptoms:**
+
+- Some records missing from export
+- Warning logs about mapping errors
+
+**Solution:**
+
+```typescript
+// Add error handling to mapper
+const results = [];
+for (const record of rawData) {
+  const result = await mapper.map(record);
+  if (!result.success) {
+    logger.warn('Mapping failed', {
+      record: record.id,
+      errors: result.errors,
+    });
+    // Include partial data or skip
+    if (result.data) {
+      results.push({ ...result.data, _errors: result.errors });
+    }
+    continue;
+  }
+  results.push(result.data);
+}
+
+// Or use default values
+const mappingConfig = {
+  fields: {
+    qty: {
+      source: 'qty',
+      resolver: 'sdk.parseInt',
+      defaultValue: 0, // Use 0 if parsing fails
+    },
+  },
+};
+```
+
+## Related Guides
+
+- [Auto-Pagination Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Deep dive into automatic pagination
+- [Universal Mapping Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Field mapping and transformations
+- [Extraction Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - General extraction patterns
+- [Connector Scenarios Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - More complete examples
+- [Standalone CSV Ingestion](./s3-csv-batch-api.md) - Ingestion counterpart
+- [SFTP to Batch API](../../02-CORE-GUIDES/ingestion/ingestion-readme.md) - SFTP-based ingestion (see Versori templates)
+
+## Performance Optimization Tips
+
+1. **Batch Size Tuning:**
+   - Small records: 200-500 per page
+   - Large records: 50-100 per page
+   - Monitor query time vs network overhead
+
+2. **Parquet Optimization:**
+   - Use SNAPPY compression for balance
+   - Adjust rowGroupSize based on query patterns
+   - Use columnar encoding for analytics
+
+3. **Incremental Extraction:**
+   - Always use timestamp filtering when possible
+   - Add overlap window to prevent gaps
+   - Track state per entity type
+
+4. **Memory Management:**
+   - Process large exports in batches
+   - Clear data after each batch upload
+   - Use streaming for very large datasets
+
+5. **Error Recovery:**
+   - Implement retry logic with exponential backoff
+   - Save checkpoints for long-running exports
+   - Keep partial results for debugging
+
+## Production Checklist
+
+- [ ] Environment variables configured
+- [ ] S3 bucket created with proper permissions
+- [ ] OAuth2 credentials tested
+- [ ] GraphQL queries validated
+- [ ] Extraction mappings tested with sample data
+- [ ] Incremental extraction state directory created
+- [ ] Logging configured (file + console)
+- [ ] Error alerting configured
+- [ ] Scheduled execution tested
+- [ ] S3 export verified in analytics tools
+- [ ] Monitoring dashboards created
+- [ ] Backup strategy for state files
+- [ ] Documentation for operators