@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md

@@ -1,514 +1,514 @@
-# Module 1: Foundations
-
-**Level:** Beginner
-**Estimated Time:** 15 minutes
-
-## Overview
-
-This module covers the fundamental concepts of data extraction from Fluent Commerce, including workflow patterns, schema validation, and key architectural decisions.
-
-## Learning Objectives
-
-By the end of this module, you will:
-
-- ✅ Understand the data extraction workflow and its components
-- ✅ Know how to validate GraphQL queries against the Fluent schema
-- ✅ Recognize when to use extraction vs ingestion
-- ✅ Understand the role of pagination, enrichment, and transformation
-
-## What is Data Extraction?
-
-Data extraction is the process of retrieving data from Fluent Commerce and transforming it for use in external systems. This is the opposite of ingestion (writing data to Fluent).
-
-**Common use cases:**
-
-- **Analytics & Reporting**: Export orders, inventory to data warehouses
-- **Synchronization**: Keep external systems in sync with Fluent data
-- **Backups**: Archive historical data for compliance
-- **Integrations**: Feed data to ERPs, WMS, analytics tools
-- **Data Migration**: Move data between environments or systems
-
-## Extraction Workflow
-
-### High-Level Flow
-
-```mermaid
-graph LR
-    A[Source] --> B[Extract]
-    B --> C[Enrich]
-    C --> D[Transform]
-    D --> E[Export]
-
-    A --> A1[GraphQL API]
-    A --> A2[Parquet Files]
-
-    B --> B1[GraphQL Queries]
-    B --> B2[Auto-Pagination]
-
-    C --> C1[Batch Queries]
-    C --> C2[Related Data]
-
-    D --> D1[Field Mapping]
-    D --> D2[Custom Logic]
-
-    E --> E1[S3/SFTP]
-    E --> E2[JSON/XML/Parquet]
-```
-
-### Workflow Components
-
-#### 1. Source
-
-Where data originates:
-
-- **Fluent GraphQL API**: Direct queries to Fluent Commerce
-- **Parquet Files**: Pre-exported data from S3/SFTP
-- **Hybrid**: Parquet base data enriched with GraphQL
-
-#### 2. Extract
-
-Retrieve data from source:
-
-- **GraphQL Queries**: Fetch entities using validated queries
-- **Auto-Pagination**: Automatically handle large datasets
-- **Filtering**: Use GraphQL filters to narrow results
-
-#### 3. Enrich
-
-Add related data:
-
-- **Batch Queries**: Fetch related entities efficiently
-- **Nested Relations**: Use GraphQL to expand relations
-- **Computed Fields**: Calculate derived values
-
-#### 4. Transform
-
-Convert to target format:
-
-- **Field Mapping**: Rename/restructure fields
-- **Data Conversion**: Change types, formats, units
-- **Business Logic**: Apply custom transformations
-
-#### 5. Export
-
-Output to destination:
-
-- **File Formats**: JSON, JSONL, CSV, XML, Parquet
-- **Destinations**: S3, SFTP, HTTP endpoints
-- **Compression**: Gzip, zip for large files
-
-## Schema Validation - Critical First Step
-
-⚠️ **IMPORTANT**: Before writing any extraction query, you MUST validate it against your Fluent Commerce schema. Using incorrect field names will cause runtime errors.
-
-### Why Schema Validation Matters
-
-**Problem**: GraphQL queries with invalid fields fail at runtime:
-
-```typescript
-// ❌ This query will FAIL if 'quantity' doesn't exist
-query GetInventory {
-  inventoryPositions(first: 100) {
-    edges {
-      node {
-        quantity  // ❌ Field might not exist!
-      }
-    }
-  }
-}
-```
-
-**Solution**: Introspect schema first to discover real field names:
-
-```bash
-# Step 1: Introspect your Fluent schema
-npx fc-connect introspect-schema \
-  --url https://YOUR_ACCOUNT.api.fluentcommerce.com/graphql \
-  --client-id YOUR_CLIENT_ID \
-  --client-secret YOUR_CLIENT_SECRET \
-  --output schema.json
-
-# Step 2: Open schema.json and search for entity types
-# Example: Search for "type InventoryPosition" to see real fields
-```
-
-### Validated Query Example
-
-✅ **All queries in this guide are validated against** [`docs/schema/fluent-commerce-schema.json`](../../../04-REFERENCE/schema/fluent-commerce-schema.json)
-
-```graphql
-# ✅ VALIDATED: These fields exist in Fluent schema
-query GetInventory($first: Int!, $after: String) {
-  inventoryPositions(first: $first, after: $after) {
-    edges {
-      node {
-        id # ✅ InventoryPosition.id (ID!)
-        ref # ✅ InventoryPosition.ref (String!)
-        productRef # ✅ InventoryPosition.productRef (String!)
-        locationRef # ✅ InventoryPosition.locationRef (String!)
-        qty # ✅ InventoryPosition.qty (Int!) - NOT 'quantity'!
-        status # ✅ InventoryPosition.status (String)
-        createdOn # ✅ InventoryPosition.createdOn (DateTime)
-        updatedOn # ✅ InventoryPosition.updatedOn (DateTime)
-      }
-      cursor
-    }
-    pageInfo {
-      hasNextPage # ✅ PageInfo.hasNextPage (Boolean!)
-    }
-  }
-}
-```
-
-### Schema Field Verification Process
-
-1. **Introspect schema** (see command above)
-2. **Find entity type** in schema.json:
-   ```json
-   {
-     "name": "InventoryPosition",
-     "fields": [
-       { "name": "id", "type": { "name": "ID" } },
-       { "name": "ref", "type": { "name": "String" } },
-       { "name": "qty", "type": { "name": "Int" } }
-     ]
-   }
-   ```
-3. **Use exact field names** in your queries
-4. **Check field types** to ensure correct usage
-
-## Extraction vs Ingestion
-
-| Aspect           | Extraction                                 | Ingestion                               |
-| ---------------- | ------------------------------------------ | --------------------------------------- |
-| **Direction**    | Fluent → External                          | External → Fluent                       |
-| **Operations**   | GraphQL Queries                            | GraphQL Mutations                       |
-| **Pagination**   | Auto-pagination                            | Batch API                               |
-| **Use Case**     | Analytics, sync, backup                    | Create/update entities                  |
-| **SDK Services** | `client.graphql()`, `ExtractionOrchestrator`, `ParquetParserService` | `FluentBatchManager`, `UniversalMapper` |
-
-## Key Concepts
-
-### 1. Cursor-Based Pagination
-
-Fluent Commerce uses Relay-style cursor pagination with support for both forward and backward pagination:
-
-**Forward Pagination (most common):**
-```typescript
-// First page
-{
-  first: 100,
-  after: null
-}
-
-// Subsequent pages
-{
-  first: 100,
-  after: "cursor_from_last_edge"
-}
-
-// Response structure
-{
-  edges: [
-    { node: { ...data }, cursor: "abc123" }
-  ],
-  pageInfo: {
-    hasNextPage: true  // More pages available
-  }
-}
-```
-
-**Backward Pagination (for reverse order):**
-```typescript
-// Last page
-{
-  last: 100,
-  before: null
-}
-
-// Previous pages
-{
-  last: 100,
-  before: "cursor_from_first_edge"
-}
-
-// Response structure
-{
-  edges: [
-    { node: { ...data }, cursor: "abc123" }
-  ],
-  pageInfo: {
-    hasPreviousPage: true  // More pages available (backwards)
-  }
-}
-```
-
-**Important**: Fluent does NOT return `pageInfo.endCursor` or `pageInfo.startCursor` (deviation from Relay spec). Use edge cursors instead:
-
-```typescript
-// Forward pagination - use last cursor
-const lastEdge = result.data.inventoryPositions.edges[edges.length - 1];
-const cursor = lastEdge?.cursor;
-
-// Backward pagination - use first cursor
-const firstEdge = result.data.inventoryPositions.edges[0];
-const cursor = firstEdge?.cursor;
-```
-
-See [Auto-Pagination Guide](../../auto-pagination/) and [Module 8: ExtractionOrchestrator](./02-core-guides-extraction-08-extraction-orchestrator.md) for automatic handling of both directions.
-
-### 2. Data Enrichment
-
-**Flat extraction** (basic fields only):
-
-```typescript
-query GetInventory {
-  inventoryPositions(first: 100) {
-    edges {
-      node {
-        ref
-        productRef
-        qty
-      }
-    }
-  }
-}
-```
-
-**Enriched extraction** (with related data):
-
-```typescript
-query GetInventoryEnriched {
-  inventoryPositions(first: 100) {
-    edges {
-      node {
-        ref
-        productRef
-        qty
-        product {
-          ref
-          name
-          gtin
-          catalogue { ref name }
-        }
-        location {
-          ref
-          name
-          type
-        }
-      }
-    }
-  }
-}
-```
-
-**Benefits of enrichment**:
-
-- Fewer total API calls (1 query vs N+1 queries)
-- Consistent data snapshot
-- Simplified downstream processing
-
-### 3. Field Mapping
-
-Transform Fluent data to external format:
-
-```typescript
-// Source (Fluent)
-{
-  "productRef": "SKU-001",
-  "locationRef": "WH-001",
-  "qty": 100,
-  "status": "AVAILABLE"
-}
-
-// Target (External System)
-{
-  "sku": "SKU-001",
-  "warehouse": "WH-001",
-  "quantity": 100,
-  "status": "in_stock"
-}
-```
-
-**Use `UniversalMapper` for automated transformation**:
-
-```typescript
-import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-
-const mapper = new UniversalMapper({
-  fields: {
-    sku: { source: 'productRef' },
-    warehouse: { source: 'locationRef' },
-    quantity: { source: 'qty', resolver: 'sdk.parseInt' },
-    status: { source: 'status', resolver: 'sdk.lowercase' },
-  },
-});
-
-const result = await mapper.map(sourceData);
-```
-
-### 4. Export Formats
-
-| Format      | Use Case               | SDK Support                |
-| ----------- | ---------------------- | -------------------------- |
-| **JSON**    | REST APIs, web apps    | Native                     |
-| **JSONL**   | Streaming, big data    | Native                     |
-| **CSV**     | Spreadsheets, BI tools | Via `csv-stringify`        |
-| **XML**     | Enterprise integrations    | Via `xml2js`               |
-| **Parquet** | Data lakes, analytics  | Via `ParquetParserService` |
-
-## Extraction Strategies
-
-### 1. Full Extraction
-
-Extract entire dataset periodically:
-
-**Pros:**
-
-- Simple to implement
-- Complete data snapshot
-- No state tracking needed
-
-**Cons:**
-
-- Large data transfer
-- Slower for large datasets
-- May hit rate limits
-
-**When to use:** Small datasets, infrequent syncs, backups
-
-### 2. Delta (Incremental) Extraction
-
-Extract only changed records since last run:
-
-**Pros:**
-
-- Minimal data transfer
-- Fast execution
-- Efficient for large datasets
-
-**Cons:**
-
-- Requires state management
-- More complex logic
-- Risk of missing updates if state is lost
-
-**When to use:** Large datasets, frequent syncs, real-time needs
-
-### 3. Streaming Extraction
-
-Process data in batches as it's fetched:
-
-**Pros:**
-
-- Memory efficient
-- Can handle unlimited dataset size
-- Start processing immediately
-
-**Cons:**
-
-- More complex implementation
-- Cannot compute totals upfront
-- Requires streaming destination
-
-**When to use:** Very large datasets, memory constraints, real-time processing
-
-## Performance Considerations
-
-### Query Optimization
-
-**❌ BAD: Fetch all fields**
-
-```graphql
-query GetProducts {
-  products(first: 100) {
-    edges {
-      node {
-        id ref name description gtin price
-        catalogue { ... }
-        attributes { ... }
-        categories { ... }
-        inventoryPositions { ... }
-      }
-    }
-  }
-}
-```
-
-**✅ GOOD: Fetch only needed fields**
-
-```graphql
-query GetProducts {
-  products(first: 100) {
-    edges {
-      node {
-        ref
-        name
-        gtin
-      }
-    }
-  }
-}
-```
-
-### Pagination Tuning
-
-| Entity Complexity          | Page Size | Rationale                     |
-| -------------------------- | --------- | ----------------------------- |
-| Simple (5-10 fields)       | 100-200   | Small payload per record      |
-| Medium (10-20 fields)      | 50-100    | Moderate payload              |
-| Complex (nested relations) | 20-50     | Large payload, avoid timeouts |
-
-### Batch Enrichment
-
-**❌ BAD: N+1 queries**
-
-```typescript
-for (const record of records) {
-  const product = await client.graphql({
-    query: `query { product(ref: "${record.productRef}") { ... } }`,
-  });
-}
-// 1000 records = 1000 API calls!
-```
-
-**✅ GOOD: Batch query**
-
-```typescript
-const refs = records.map(r => r.productRef);
-const products = await client.graphql({
-  query: `
-    query GetProducts($refs: [String!]!) {
-      products(filter: { ref: { in: $refs } }, first: 100) {
-        edges { node { ref name } }
-      }
-    }
-  `,
-  variables: { refs },
-});
-// 1000 records = 10 API calls (batches of 100)
-```
-
-## Practice Exercise
-
-**Goal**: Extract all virtual positions with product details
-
-1. Create a GraphQL query for `virtualPositions`
-2. Include product relation fields
-3. Validate query against schema
-4. Implement with auto-pagination
-
-**Solution**: See [Module 2: Basic Extraction](./02-core-guides-extraction-02-basic-extraction.md#complete-example)
-
-## Key Takeaways
-
-- 🎯 Always validate GraphQL queries against schema before use
-- 🎯 Understand the extraction workflow: Extract → Enrich → Transform → Export
-- 🎯 Choose the right strategy: Full, Delta, or Streaming extraction
-- 🎯 Optimize queries: Select only needed fields, batch enrichment
-- 🎯 Use auto-pagination for large datasets
-- 🎯 Plan for error handling and retry logic
-
-## Next Steps
-
-Continue to [Module 2: Basic Extraction](./02-core-guides-extraction-02-basic-extraction.md) to learn how to write your first extraction query with auto-pagination.
+# Module 1: Foundations
+
+**Level:** Beginner
+**Estimated Time:** 15 minutes
+
+## Overview
+
+This module covers the fundamental concepts of data extraction from Fluent Commerce, including workflow patterns, schema validation, and key architectural decisions.
+
+## Learning Objectives
+
+By the end of this module, you will:
+
+- ✅ Understand the data extraction workflow and its components
+- ✅ Know how to validate GraphQL queries against the Fluent schema
+- ✅ Recognize when to use extraction vs ingestion
+- ✅ Understand the role of pagination, enrichment, and transformation
+
+## What is Data Extraction?
+
+Data extraction is the process of retrieving data from Fluent Commerce and transforming it for use in external systems. This is the opposite of ingestion (writing data to Fluent).
+
+**Common use cases:**
+
+- **Analytics & Reporting**: Export orders, inventory to data warehouses
+- **Synchronization**: Keep external systems in sync with Fluent data
+- **Backups**: Archive historical data for compliance
+- **Integrations**: Feed data to ERPs, WMS, analytics tools
+- **Data Migration**: Move data between environments or systems
+
+## Extraction Workflow
+
+### High-Level Flow
+
+```mermaid
+graph LR
+    A[Source] --> B[Extract]
+    B --> C[Enrich]
+    C --> D[Transform]
+    D --> E[Export]
+
+    A --> A1[GraphQL API]
+    A --> A2[Parquet Files]
+
+    B --> B1[GraphQL Queries]
+    B --> B2[Auto-Pagination]
+
+    C --> C1[Batch Queries]
+    C --> C2[Related Data]
+
+    D --> D1[Field Mapping]
+    D --> D2[Custom Logic]
+
+    E --> E1[S3/SFTP]
+    E --> E2[JSON/XML/Parquet]
+```
+
+### Workflow Components
+
+#### 1. Source
+
+Where data originates:
+
+- **Fluent GraphQL API**: Direct queries to Fluent Commerce
+- **Parquet Files**: Pre-exported data from S3/SFTP
+- **Hybrid**: Parquet base data enriched with GraphQL
+
+#### 2. Extract
+
+Retrieve data from source:
+
+- **GraphQL Queries**: Fetch entities using validated queries
+- **Auto-Pagination**: Automatically handle large datasets
+- **Filtering**: Use GraphQL filters to narrow results
+
+#### 3. Enrich
+
+Add related data:
+
+- **Batch Queries**: Fetch related entities efficiently
+- **Nested Relations**: Use GraphQL to expand relations
+- **Computed Fields**: Calculate derived values
+
+#### 4. Transform
+
+Convert to target format:
+
+- **Field Mapping**: Rename/restructure fields
+- **Data Conversion**: Change types, formats, units
+- **Business Logic**: Apply custom transformations
+
+#### 5. Export
+
+Output to destination:
+
+- **File Formats**: JSON, JSONL, CSV, XML, Parquet
+- **Destinations**: S3, SFTP, HTTP endpoints
+- **Compression**: Gzip, zip for large files
+
+## Schema Validation - Critical First Step
+
+⚠️ **IMPORTANT**: Before writing any extraction query, you MUST validate it against your Fluent Commerce schema. Using incorrect field names will cause runtime errors.
+
+### Why Schema Validation Matters
+
+**Problem**: GraphQL queries with invalid fields fail at runtime:
+
+```typescript
+// ❌ This query will FAIL if 'quantity' doesn't exist
+query GetInventory {
+  inventoryPositions(first: 100) {
+    edges {
+      node {
+        quantity  // ❌ Field might not exist!
+      }
+    }
+  }
+}
+```
+
+**Solution**: Introspect schema first to discover real field names:
+
+```bash
+# Step 1: Introspect your Fluent schema
+npx fc-connect introspect-schema \
+  --url https://YOUR_ACCOUNT.api.fluentcommerce.com/graphql \
+  --client-id YOUR_CLIENT_ID \
+  --client-secret YOUR_CLIENT_SECRET \
+  --output schema.json
+
+# Step 2: Open schema.json and search for entity types
+# Example: Search for "type InventoryPosition" to see real fields
+```
+
+### Validated Query Example
+
+✅ **All queries in this guide are validated against** [`docs/schema/fluent-commerce-schema.json`](../../../04-REFERENCE/schema/fluent-commerce-schema.json)
+
+```graphql
+# ✅ VALIDATED: These fields exist in Fluent schema
+query GetInventory($first: Int!, $after: String) {
+  inventoryPositions(first: $first, after: $after) {
+    edges {
+      node {
+        id # ✅ InventoryPosition.id (ID!)
+        ref # ✅ InventoryPosition.ref (String!)
+        productRef # ✅ InventoryPosition.productRef (String!)
+        locationRef # ✅ InventoryPosition.locationRef (String!)
+        qty # ✅ InventoryPosition.qty (Int!) - NOT 'quantity'!
+        status # ✅ InventoryPosition.status (String)
+        createdOn # ✅ InventoryPosition.createdOn (DateTime)
+        updatedOn # ✅ InventoryPosition.updatedOn (DateTime)
+      }
+      cursor
+    }
+    pageInfo {
+      hasNextPage # ✅ PageInfo.hasNextPage (Boolean!)
+    }
+  }
+}
+```
+
+### Schema Field Verification Process
+
+1. **Introspect schema** (see command above)
+2. **Find entity type** in schema.json:
+   ```json
+   {
+     "name": "InventoryPosition",
+     "fields": [
+       { "name": "id", "type": { "name": "ID" } },
+       { "name": "ref", "type": { "name": "String" } },
+       { "name": "qty", "type": { "name": "Int" } }
+     ]
+   }
+   ```
+3. **Use exact field names** in your queries
+4. **Check field types** to ensure correct usage
+
+## Extraction vs Ingestion
+
+| Aspect           | Extraction                                 | Ingestion                               |
+| ---------------- | ------------------------------------------ | --------------------------------------- |
+| **Direction**    | Fluent → External                          | External → Fluent                       |
+| **Operations**   | GraphQL Queries                            | GraphQL Mutations                       |
+| **Pagination**   | Auto-pagination                            | Batch API                               |
+| **Use Case**     | Analytics, sync, backup                    | Create/update entities                  |
+| **SDK Services** | `client.graphql()`, `ExtractionOrchestrator`, `ParquetParserService` | `FluentBatchManager`, `UniversalMapper` |
+
+## Key Concepts
+
+### 1. Cursor-Based Pagination
+
+Fluent Commerce uses Relay-style cursor pagination with support for both forward and backward pagination:
+
+**Forward Pagination (most common):**
+```typescript
+// First page
+{
+  first: 100,
+  after: null
+}
+
+// Subsequent pages
+{
+  first: 100,
+  after: "cursor_from_last_edge"
+}
+
+// Response structure
+{
+  edges: [
+    { node: { ...data }, cursor: "abc123" }
+  ],
+  pageInfo: {
+    hasNextPage: true  // More pages available
+  }
+}
+```
+
+**Backward Pagination (for reverse order):**
+```typescript
+// Last page
+{
+  last: 100,
+  before: null
+}
+
+// Previous pages
+{
+  last: 100,
+  before: "cursor_from_first_edge"
+}
+
+// Response structure
+{
+  edges: [
+    { node: { ...data }, cursor: "abc123" }
+  ],
+  pageInfo: {
+    hasPreviousPage: true  // More pages available (backwards)
+  }
+}
+```
+
+**Important**: Fluent does NOT return `pageInfo.endCursor` or `pageInfo.startCursor` (deviation from Relay spec). Use edge cursors instead:
+
+```typescript
+// Forward pagination - use last cursor
+const lastEdge = result.data.inventoryPositions.edges[edges.length - 1];
+const cursor = lastEdge?.cursor;
+
+// Backward pagination - use first cursor
+const firstEdge = result.data.inventoryPositions.edges[0];
+const cursor = firstEdge?.cursor;
+```
+
+See [Auto-Pagination Guide](../../auto-pagination/) and [Module 8: ExtractionOrchestrator](./02-core-guides-extraction-08-extraction-orchestrator.md) for automatic handling of both directions.
+
+### 2. Data Enrichment
+
+**Flat extraction** (basic fields only):
+
+```typescript
+query GetInventory {
+  inventoryPositions(first: 100) {
+    edges {
+      node {
+        ref
+        productRef
+        qty
+      }
+    }
+  }
+}
+```
+
+**Enriched extraction** (with related data):
+
+```typescript
+query GetInventoryEnriched {
+  inventoryPositions(first: 100) {
+    edges {
+      node {
+        ref
+        productRef
+        qty
+        product {
+          ref
+          name
+          gtin
+          catalogue { ref name }
+        }
+        location {
+          ref
+          name
+          type
+        }
+      }
+    }
+  }
+}
+```
+
+**Benefits of enrichment**:
+
+- Fewer total API calls (1 query vs N+1 queries)
+- Consistent data snapshot
+- Simplified downstream processing
+
+### 3. Field Mapping
+
+Transform Fluent data to external format:
+
+```typescript
+// Source (Fluent)
+{
+  "productRef": "SKU-001",
+  "locationRef": "WH-001",
+  "qty": 100,
+  "status": "AVAILABLE"
+}
+
+// Target (External System)
+{
+  "sku": "SKU-001",
+  "warehouse": "WH-001",
+  "quantity": 100,
+  "status": "in_stock"
+}
+```
+
+**Use `UniversalMapper` for automated transformation**:
+
+```typescript
+import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+
+const mapper = new UniversalMapper({
+  fields: {
+    sku: { source: 'productRef' },
+    warehouse: { source: 'locationRef' },
+    quantity: { source: 'qty', resolver: 'sdk.parseInt' },
+    status: { source: 'status', resolver: 'sdk.lowercase' },
+  },
+});
+
+const result = await mapper.map(sourceData);
+```
+
+### 4. Export Formats
+
+| Format      | Use Case               | SDK Support                |
+| ----------- | ---------------------- | -------------------------- |
+| **JSON**    | REST APIs, web apps    | Native                     |
+| **JSONL**   | Streaming, big data    | Native                     |
+| **CSV**     | Spreadsheets, BI tools | Via `csv-stringify`        |
+| **XML**     | Enterprise integrations    | Via `xml2js`               |
+| **Parquet** | Data lakes, analytics  | Via `ParquetParserService` |
+
+## Extraction Strategies
+
+### 1. Full Extraction
+
+Extract entire dataset periodically:
+
+**Pros:**
+
+- Simple to implement
+- Complete data snapshot
+- No state tracking needed
+
+**Cons:**
+
+- Large data transfer
+- Slower for large datasets
+- May hit rate limits
+
+**When to use:** Small datasets, infrequent syncs, backups
+
+### 2. Delta (Incremental) Extraction
+
+Extract only changed records since last run:
+
+**Pros:**
+
+- Minimal data transfer
+- Fast execution
+- Efficient for large datasets
+
+**Cons:**
+
+- Requires state management
+- More complex logic
+- Risk of missing updates if state is lost
+
+**When to use:** Large datasets, frequent syncs, real-time needs
+
+### 3. Streaming Extraction
+
+Process data in batches as it's fetched:
+
+**Pros:**
+
+- Memory efficient
+- Can handle unlimited dataset size
+- Start processing immediately
+
+**Cons:**
+
+- More complex implementation
+- Cannot compute totals upfront
+- Requires streaming destination
+
+**When to use:** Very large datasets, memory constraints, real-time processing
+
+## Performance Considerations
+
+### Query Optimization
+
+**❌ BAD: Fetch all fields**
+
+```graphql
+query GetProducts {
+  products(first: 100) {
+    edges {
+      node {
+        id ref name description gtin price
+        catalogue { ... }
+        attributes { ... }
+        categories { ... }
+        inventoryPositions { ... }
+      }
+    }
+  }
+}
+```
+
+**✅ GOOD: Fetch only needed fields**
+
+```graphql
+query GetProducts {
+  products(first: 100) {
+    edges {
+      node {
+        ref
+        name
+        gtin
+      }
+    }
+  }
+}
+```
+
+### Pagination Tuning
+
+| Entity Complexity          | Page Size | Rationale                     |
+| -------------------------- | --------- | ----------------------------- |
+| Simple (5-10 fields)       | 100-200   | Small payload per record      |
+| Medium (10-20 fields)      | 50-100    | Moderate payload              |
+| Complex (nested relations) | 20-50     | Large payload, avoid timeouts |
+
+### Batch Enrichment
+
+**❌ BAD: N+1 queries**
+
+```typescript
+for (const record of records) {
+  const product = await client.graphql({
+    query: `query { product(ref: "${record.productRef}") { ... } }`,
+  });
+}
+// 1000 records = 1000 API calls!
+```
+
+**✅ GOOD: Batch query**
+
+```typescript
+const refs = records.map(r => r.productRef);
+const products = await client.graphql({
+  query: `
+    query GetProducts($refs: [String!]!) {
+      products(filter: { ref: { in: $refs } }, first: 100) {
+        edges { node { ref name } }
+      }
+    }
+  `,
+  variables: { refs },
+});
+// 1000 records = 10 API calls (batches of 100)
+```
+
+## Practice Exercise
+
+**Goal**: Extract all virtual positions with product details
+
+1. Create a GraphQL query for `virtualPositions`
+2. Include product relation fields
+3. Validate query against schema
+4. Implement with auto-pagination
+
+**Solution**: See [Module 2: Basic Extraction](./02-core-guides-extraction-02-basic-extraction.md#complete-example)
+
+## Key Takeaways
+
+- 🎯 Always validate GraphQL queries against schema before use
+- 🎯 Understand the extraction workflow: Extract → Enrich → Transform → Export
+- 🎯 Choose the right strategy: Full, Delta, or Streaming extraction
+- 🎯 Optimize queries: Select only needed fields, batch enrichment
+- 🎯 Use auto-pagination for large datasets
+- 🎯 Plan for error handling and retry logic
+
+## Next Steps
+
+Continue to [Module 2: Basic Extraction](./02-core-guides-extraction-02-basic-extraction.md) to learn how to write your first extraction query with auto-pagination.