@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md

@@ -1,834 +1,834 @@
-# Service Layer Architecture
-
-This document explains the organization and responsibilities of services within the FC Connect SDK.
-
-The SDK organizes services into logical modules, each with clear responsibilities and minimal coupling.
-
-```mermaid
-graph TB
-    subgraph "Client Layer"
-        CLIENT[FluentClient / FluentVersoriClient]
-    end
-
-    subgraph "Service Modules"
-        BATCH[Batch Services]
-        CORE[Core Services]
-        MAPPING[Mapping Services]
-        TRANSFORM[Transformation Services]
-        RESOLVERS[Resolver Services]
-        INTEGRATION[Integration Services]
-        ORCHESTRATION[Orchestration Services]
-        VALIDATION[Validation Services]
-        MONITORING[Monitoring Services]
-    end
-
-    subgraph "Data Layer"
-        SOURCES[Data Sources]
-        PARSERS[Parsers]
-    end
-
-    CLIENT --> BATCH
-    CLIENT --> MAPPING
-    CLIENT --> TRANSFORM
-    CLIENT --> ORCHESTRATION
-
-    BATCH --> CORE
-    MAPPING --> RESOLVERS
-    MAPPING --> CORE
-    TRANSFORM --> MAPPING
-    ORCHESTRATION --> BATCH
-    ORCHESTRATION --> SOURCES
-    ORCHESTRATION --> PARSERS
-    ORCHESTRATION --> MAPPING
-
-    SOURCES --> PARSERS
-    VALIDATION --> CORE
-
-    style CLIENT fill:#e3f2fd
-    style BATCH fill:#fff3e0
-    style CORE fill:#f3e5f5
-    style MAPPING fill:#e8f5e9
-    style ORCHESTRATION fill:#e1f5ff
-```
-
-## Service Modules
-
-The SDK organizes services into **11 service modules** (10 exported from `services/index.ts`, plus `s3/` exported directly from root):
-
-1. **auth/** - Webhook authentication services
-2. **batch/** - Job creation, batch management, partial batch recovery
-3. **core/** - Logging, state, job tracking, KV storage
-4. **extraction/** - High-level extraction orchestration (includes statistics tracking)
-5. **integration/** - S3 presign, webhook validation
-6. **mapping/** - Field mapping, GraphQL mutation generation
-7. **orchestration/** - High-level ingestion workflows (includes statistics tracking)
-8. **resolvers/** - GraphQL path resolution, SDK resolvers
-9. **s3/** - S3 operations (SDK and presigned implementations) - exported from root, not services/index.ts
-10. **transformation/** - Data transformation and extraction
-11. **validation/** - Schema and preflight validation
-
-**Note:** Statistics and monitoring capabilities are built into `extraction/` and `orchestration/` services, not a separate module.
-
-### 1. Batch Services
-
-**Location:** `src/services/batch/`
-
-Handles job creation, batch management, file archival, and partial batch recovery.
-
-```mermaid
-graph TB
-    subgraph "Batch Services"
-        MANAGER[FluentBatchManager]
-        ARCHIVAL[BatchArchivalService]
-        RECOVERY[PartialBatchRecovery]
-    end
-
-    MANAGER --> ARCHIVAL
-
-    ARCHIVAL --> MOVE[Move Files]
-    ARCHIVAL --> COPY[Copy Files]
-    ARCHIVAL --> DELETE[Delete Files]
-
-    RECOVERY --> RETRY[Retry Failed Records]
-    RECOVERY --> CHECKPOINT[Checkpoint/Resume]
-    RECOVERY --> TRACK[Track Failures]
-
-    style MANAGER fill:#e3f2fd
-    style ARCHIVAL fill:#f3e5f5
-    style RECOVERY fill:#fff3e0
-    style MOVE fill:#e8f5e9
-    style COPY fill:#e8f5e9
-    style DELETE fill:#ffebee
-    style RETRY fill:#fff4e6
-    style CHECKPOINT fill:#fff4e6
-    style TRACK fill:#fff4e6
-```
-
-**Key Services:**
-
-#### FluentBatchManager
-**Source:** `src/services/batch/manager.ts`
-
-- Create and manage batch jobs
-- Send batches to Fluent Commerce
-- Handle job expiration and recreation
-- Archive processed files to S3
-
-**Example Usage:**
-
-```typescript
-const manager = new FluentBatchManager(fluentClient, logger);
-
-// Create job
-const job = await manager.createJob('inventory-update', { retailerId: 'my-retailer' });
-
-// Send batch
-await manager.submitBatch(job.id, {
-  entityType: 'INVENTORY',
-  action: 'UPSERT',
-  entities: inventoryData,
-});
-
-// Check status
-const status = await manager.getJobStatus(job.id);
-```
-
-#### PartialBatchRecovery
-**Source:** `src/services/batch/partial-batch-recovery.ts` (lines 1-357)
-
-Tracks per-record success/failure in batch operations and enables:
-- Retrying only failed records instead of entire batch
-- Checkpoint/resume functionality
-- Detailed error reporting per record
-- Progress tracking
-
-**Key Methods:**
-- `processBatchWithRecovery(records, batchProcessor, options)` - Process batch with automatic recovery and retry
-- `resumeFromCheckpoint(checkpointId, batchProcessor, options)` - Resume from checkpoint
-- `getCheckpoint(checkpointId)` - Get checkpoint by ID
-- `listCheckpoints()` - List all checkpoints
-- `deleteCheckpoint(checkpointId)` - Delete checkpoint
-
-**Example Usage:**
-
-```typescript
-const recovery = new PartialBatchRecovery(logger);
-
-// Process batch with recovery tracking
-const result = await recovery.processBatchWithRecovery(
-  records,
-  async (batch) => await client.sendBatch(jobId, batch),
-  {
-    maxRetries: 3,
-    retryOnlyFailed: true,
-    checkpointKey: 'inventory-2025-01-24'
-  }
-);
-
-console.log(`Success: ${result.successCount}, Failed: ${result.failedCount}`);
-if (result.failedRecords.length > 0) {
-  console.log('Failed records:', result.failedRecords);
-}
-```
-
-### 2. Core Services
-
-**Location:** `src/services/core/`
-
-Foundation services used across the SDK.
-
-```mermaid
-graph TB
-    subgraph "Core Services"
-        STATE[StateService]
-        KV[KVStorageAdapter]
-        TRACKER[JobTracker]
-    end
-
-    LOG --> CONSOLE[createConsoleLogger]
-    STATE --> KV
-    STATE --> LOCK[Distributed Locking]
-    STATE --> DEDUP[Deduplication]
-
-    KV --> VERSORI[VersoriKVAdapter]
-    KV --> CUSTOM[Custom Implementations]
-
-    TRACKER --> KV
-    TRACKER --> CREATE[Create Job]
-    TRACKER --> UPDATE[Update Status]
-    TRACKER --> COMPLETE[Mark Complete/Failed]
-
-    style LOG fill:#fff3e0
-    style STATE fill:#f3e5f5
-    style KV fill:#e8f5e9
-    style TRACKER fill:#e1f5ff
-#### Logging Utilities
-
-## Logging Architecture
-
-### Function-Based Pattern
-
-**Versori Platform:**
-```typescript
-// Use native Versori log
-export const workflow = http('POST', '/sync', async (ctx) => {
-  ctx.log('info', 'Processing sync', { recordCount: 100 });
-
-  // For SDK services, wrap native log with toStructuredLogger
-  const logger = toStructuredLogger(ctx.log, { service: 'sync' });
-  const client = await createClient({ ...ctx, log: logger });
-});
-```
-
-**Standalone (Node.js/Deno):**
-```typescript
-// Use function-based utilities
-import {
-  createClient,
-  createConsoleLogger,
-  toStructuredLogger,
-  createWorkflowLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = createConsoleLogger();
-const client = await createClient({ config }, { logger });
-```
-
-**Core Functions:**
-- `createConsoleLogger()` - Simple console logger
-- `toStructuredLogger(logger, context)` - Add context to logger
-- `generateCorrelationId()` - Generate unique request IDs
-- `createWorkflowLogger()` - Pre-configured workflow logger
-- `createServiceLogger()` - Pre-configured service logger
-- `createErrorLogger()` - Error context logger
-
-## Logging Decision Flowchart
-
-```
-Are you writing code for Versori platform?
-│
-├─ YES: Use native ctx.log
-│   │
-│   └─ Need to pass logger to SDK services?
-│       │
-│       └─ YES: Wrap with toStructuredLogger(ctx.log, context)
-│
-└─ NO (Standalone Node.js/Deno):
-    │
-    └─ Use createConsoleLogger() or createWorkflowLogger()
-```
-
-**Quick Reference:**
-- **Versori workflows:** `ctx.log()` directly
-- **Versori + SDK services:** `toStructuredLogger(ctx.log, { service: 'name' })`
-- **Standalone apps:** `createConsoleLogger()`
-
-**See:** [Logging Guide](../../02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md#logging-utilities) for complete patterns.
-
-#### StateService
-**Source:** `src/services/core/state-service.ts` (lines 1-382)
-
-Distributed state management with key-value storage.
-
-**Constructor:** `new StateService(logger?)` - Logger is optional
-
-**Key Features:**
-- Distributed state management with KV storage
-- File deduplication tracking
-- Distributed locking with stale detection
-- Daily job tracking
-- Sync state management
-
-**IMPORTANT:** All methods require `kv` parameter as first argument.
-
-**Key Methods:**
-- `acquireLock(lockName, kv, timeoutMinutes?)` - Acquire distributed lock
-- `releaseLock(lockName, kv)` - Release lock
-- `getSyncState(kv, workflowId?)` - Get sync state
-- `updateSyncState(kv, processedFiles, workflowId?)` - Update sync state
-- `getDailyJob(kv, workflowId)` - Get or check daily job
-- `setDailyJob(kv, workflowId, jobId, expirationHours?)` - Store daily job
-- `isFileProcessed(kv, fileName, workflowId?)` - Check if file processed
-- `getMetrics(kv, workflowId?)` - Get processing metrics
-- `clearState(kv, workflowId)` - Clear all state for workflow
-
-**State Management Example:**
-
-```typescript
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-
-const kvAdapter = new VersoriKVAdapter(openKv());
-const stateService = new StateService(logger);
-
-// Check if file processed (note: kv is first parameter)
-const processed = await stateService.isFileProcessed(
-  kvAdapter,
-  'file-key',
-  'my-workflow'
-);
-
-if (!processed) {
-  // Process file
-  await processFile();
-
-  // Update sync state with processed file metadata
-  await stateService.updateSyncState(
-    kvAdapter,
-    [{
-      fileName: 'file-key',
-      lastModified: new Date().toISOString(),
-      recordCount: 100
-    }],
-    'my-workflow'
-  );
-}
-```
-
-#### JobTracker
-**Source:** `src/services/core/job-tracker.ts` (lines 1-355)
-
-Job status tracking for async workflows. Provides standardized way to track job status in Versori integrations, making it easy to monitor scheduled workflows and webhook-triggered operations.
-
-**Key Features:**
-- Standardized job status tracking (queued, processing, completed, failed)
-- Automatic timestamp management
-- TTL support for automatic cleanup
-- Safe error handling (never throws on KV errors)
-- Detailed logging
-
-**Key Methods:**
-- `createJob(jobId, metadata)` - Creates a new job with 'queued' status
-- `updateJob(jobId, update)` - Updates job status
-- `getJob(jobId)` - Retrieves job status
-- `markCompleted(jobId, details)` - Marks job as completed
-- `markFailed(jobId, error)` - Marks job as failed
-
-**Example Usage:**
-
-```typescript
-const tracker = new JobTracker(kvAdapter, logger);
-
-// Create job
-await tracker.createJob('job_123', {
-  triggeredBy: 'schedule',
-  stage: 'extraction'
-});
-
-// Update progress
-await tracker.updateJob('job_123', {
-  status: 'processing',
-  stage: 'transformation',
-  message: 'Processing 1000 records'
-});
-
-// Mark as completed
-await tracker.markCompleted('job_123', {
-  recordCount: 1000,
-  fileName: 'export.xml'
-});
-```
-
-### 3. Mapping Services
-
-**Location:** `src/services/mapping/`
-
-Universal field mapping for all data transformations.
-
-```mermaid
-graph TB
-    subgraph "Mapping Services"
-        UNIVERSAL[UniversalMapper]
-        MUTATION[GraphQLMutationMapper]
-        PATH[PathResolver]
-    end
-
-    UNIVERSAL --> RESOLVE[Field Resolution]
-    UNIVERSAL --> VALIDATE[Validation]
-    UNIVERSAL --> TRANSFORM[Apply Resolvers]
-
-    MUTATION --> PARSE[Parse Input]
-    MUTATION --> MAP[Apply Mappings]
-    MUTATION --> BUILD[Build Mutation]
-
-    PATH --> JSON[JSONPath Resolution]
-    PATH --> XML[XMLPath Resolution]
-    PATH --> GRAPHQL[GraphQL Path Resolution]
-
-    style UNIVERSAL fill:#e3f2fd
-    style MUTATION fill:#fff3e0
-    style PATH fill:#f3e5f5
-```
-
-**Key Responsibilities:**
-
-- Universal field mapping configuration
-- GraphQL mutation generation from XML/JSON
-- Path resolution for nested data
-- Field validation and transformation
-
-**Universal Mapping Example:**
-
-```typescript
-const mapper = new UniversalMapper({
-  fields: {
-    skuRef: { source: 'sku_id', required: true },
-    qty: { source: 'quantity', resolver: 'sdk.parseInt' },
-    expectedOn: { source: 'date', resolver: 'sdk.formatDate' },
-  },
-});
-
-const result = await mapper.map(sourceData);
-// { success: true, data: { skuRef: '...', qty: 10, expectedOn: '2025-01-15' } }
-```
-
-**GraphQL Mutation Mapping Example:**
-
-```typescript
-const mutationMapper = new GraphQLMutationMapper(xmlParser, logger);
-
-const mutation = await mutationMapper.generateMutation(xmlData, mappingConfig);
-
-// Execute mutation
-const result = await fluentClient.graphql({ query: mutation });
-```
-
-### 4. Transformation & Extraction Services
-
-**Location:** `src/services/transformation/` and `src/services/extraction/`
-
-Data extraction and transformation services.
-
-```mermaid
-graph TB
-    subgraph "Transformation & Extraction Services"
-        ORCH[ExtractionOrchestrator]
-    end
-
-    ORCH --> QUERY[Execute GraphQL Query]
-    ORCH --> AUTOPAGE[Automatic Pagination]
-    ORCH --> PATHEXTRACT[Path-based Extraction]
-    ORCH --> VALIDATE[Per-item Validation]
-    ORCH --> STATS[Statistics Tracking]
-    ORCH --> LIMITS[Limits: maxRecords, maxPages, timeout]
-
-    style ORCH fill:#fff3e0
-```
-
-**Key Services:**
-
-#### ExtractionOrchestrator (Optional Helper)
-**Source:** `src/services/extraction/extraction-orchestrator.ts` (lines 1-460)
-Generic data extraction from Fluent Commerce GraphQL API with automatic pagination, error handling, and statistics tracking. This is an **optional convenience service** for extraction workflows.
-
-> **Note**: For maximum flexibility, consider using `client.graphql()` with auto-pagination directly instead of this orchestrator. Building blocks give you more control over the workflow.
-
-**Key Features:**
-- Automatic pagination handling
-- Extraction limits (maxRecords, maxPages, timeout)
-- Path-based result extraction from nested GraphQL responses
-- Per-item validation
-- Comprehensive error handling
-- Statistics tracking
-
-**Key Methods:**
-- `extract<T>(options)` - Extract data from Fluent Commerce GraphQL API
-  - Returns: `Promise<ExtractionResult<T>>` with data, stats, and errors
-
-**Example Usage:**
-
-```typescript
-const orchestrator = new ExtractionOrchestrator(client, logger);
-
-const result = await orchestrator.extract({
-  query: VIRTUAL_POSITIONS_QUERY,
-  variables: {
-    catalogues: [{ ref: 'DEFAULT:1' }],
-    dateRangeFilter: { from: '2024-01-01' }
-  },
-  resultPath: 'virtualPositions.edges.node',
-  pageSize: 200,
-  maxRecords: 100000,
-  validateItem: (item) => item.ref && item.quantity > 0
-});
-
-console.log(`Extracted ${result.stats.totalRecords} records in ${result.stats.duration}ms`);
-console.log(`Pages fetched: ${result.stats.totalPages}`);
-if (result.stats.truncated) {
-  console.log(`Truncation reason: ${result.stats.truncationReason}`);
-}
-```
-
-**Alternative (Building Blocks - More Flexible):**
-
-For more control, compose `client.graphql()` with auto-pagination and UniversalMapper:
-
-```typescript
-// Direct approach with client auto-pagination
-const result = await client.graphql({
-  query: VIRTUAL_POSITIONS_QUERY,
-  variables: { catalogues: [{ ref: 'DEFAULT:1' }] },
-  pagination: { maxRecords: 100000, pageSize: 200 }
-});
-
-const items = result.data?.virtualPositions?.edges?.map(e => e.node) || [];
-
-// Transform with UniversalMapper if needed
-const mapper = new UniversalMapper(mappingConfig);
-const transformed = await Promise.all(items.map(item => mapper.map(item)));
-
-console.log(`Extracted ${items.length} records`);
-```
-
-> Note: Any references below to Orchestration are historical and should be interpreted as building-block patterns using clients + services.
-
-### 5. Resolver Services
-
-**Location:** `src/services/resolvers/`
-
-Built-in transformation functions (SDK resolvers) and helper utilities.
-
-**SDK Resolver Categories:**
-
-```typescript
-// String transformations
-('sdk.uppercase', 'sdk.lowercase', 'sdk.trim', 'sdk.toString');
-
-// Number parsing
-('sdk.parseInt', 'sdk.parseFloat', 'sdk.number');
-
-// Date formatting
-('sdk.formatDate', 'sdk.formatDateShort', 'sdk.parseDate');
-
-// Type conversions
-('sdk.boolean', 'sdk.parseJson', 'sdk.toJson');
-
-// Utilities
-('sdk.identity', 'sdk.coalesce');
-```
-
-### 6. Integration Services
-
-**Location:** `src/services/integration/`
-
-External system integration utilities.
-
-```mermaid
-graph TB
-    subgraph "Integration Services"
-        S3[S3PresignService]
-        WEBHOOK[WebhookValidationService]
-    end
-
-    S3 --> PRESIGN[Generate Presigned URLs]
-    S3 --> UPLOAD[Upload via Presigned URL]
-    S3 --> DOWNLOAD[Download via Presigned URL]
-
-    WEBHOOK --> VALIDATE[Cryptographic Signature Validation]
-    WEBHOOK --> PARSE[Parse Payloads]
-    WEBHOOK --> VERIFY[Verify Timestamps]
-    WEBHOOK --> CLIENT[FluentClient.validateWebhook()]
-
-    style S3 fill:#e3f2fd
-    style WEBHOOK fill:#fff3e0
-```
-
-**Key Services:**
-
-#### S3PresignService
-Generate S3 presigned URLs for cross-account access without sharing credentials.
-
-#### WebhookValidationService
-**Integration:** Accessible via `FluentClient.validateWebhook()`
-
-Cryptographic signature validation for Fluent Commerce webhooks.
-
-**Key Responsibilities:**
-
-- Generate S3 presigned URLs (SigV4)
-- Validate webhook signatures (HMAC-SHA256)
-- Parse webhook payloads
-- Support cross-account S3 access
-- Verify webhook timestamp freshness
-
-**S3 Presign Example:**
-
-```typescript
-const s3PresignService = new S3PresignService(config);
-
-// Generate upload URL
-const uploadUrl = await s3PresignService.getPresignedUploadUrl(
-  'my-bucket',
-  'path/to/file.csv',
-  3600 // 1 hour expiration
-);
-
-// Upload file
-await fetch(uploadUrl, {
-  method: 'PUT',
-  body: fileData,
-});
-```
-
-**Webhook Validation Example:**
-
-```typescript
-// Via FluentClient (recommended) - returns boolean
-const isValid = await client.validateWebhook(
-  webhookBody,
-  request.headers['x-fluent-signature'],
-  rawRequestBody
-);
-
-if (!isValid) {
-  throw new Error('Invalid webhook signature');
-}
-
-// Process validated webhook
-console.log('Event type:', webhookBody.name);
-```
-
-### 8. Auth Services
-
-**Location:** `src/services/auth/`
-
-Webhook authentication services for API key and token management.
-
-**Key Services:**
-
-#### WebhookAuthService
-
-Provides API key and token validation for webhook endpoints.
-
-**Key Features:**
-- API key validation
-- Token issuance and validation
-- User validation
-- Support for multiple authentication methods
-
-### 9. S3 Services
-
-**Location:** `src/services/s3/`
-
-**Note:** S3Service is exported directly from the SDK root (`src/index.ts`), not from `services/index.ts`.
-
-Unified S3 operations with dual implementation strategies.
-
-```mermaid
-graph TB
-    subgraph "S3 Services"
-        S3SERVICE[S3Service]
-        SDK_IMPL[SDKImplementation<br/>AWS SDK v3]
-        PRESIGNED_IMPL[PresignedImplementation<br/>SigV4 URLs]
-    end
-
-    S3SERVICE --> SDK_IMPL
-    S3SERVICE --> PRESIGNED_IMPL
-
-    SDK_IMPL --> BASIC[Basic Ops<br/>get, put, list, delete]
-    SDK_IMPL --> ADVANCED[Advanced Ops<br/>multipart, S3 Select]
-
-    PRESIGNED_IMPL --> BASIC_PS[Basic Ops<br/>get, put, list, delete]
-
-    style S3SERVICE fill:#e3f2fd
-    style SDK_IMPL fill:#e8f5e9
-    style PRESIGNED_IMPL fill:#fff3e0
-```
-
-**Key Services:**
-
-#### S3Service
-**Source:** `src/services/s3/s3-service.ts`
-
-Main interface for S3 operations with automatic implementation selection.
-
-**Two Implementation Modes:**
-
-1. **SDK Mode (Default)** - Uses AWS SDK v3
-   - Full feature set including multipart upload and S3 Select
-   - Works in Node.js and environments with AWS SDK support
-   - Best for production workloads with large files
-
-2. **Presigned Mode (Explicit)** - Uses presigned URLs
-   - Works in restricted environments (browsers, Versori platform)
-   - Basic operations only (no multipart, no S3 Select)
-   - Requires `ctx.fetch` when used in Versori
-
-**Key Methods:**
-
-**Basic Operations (Both modes):**
-- `listObjects(bucket, options)` - List objects in bucket
-- `getObject(bucket, key)` - Get object content
-- `putObject(bucket, key, body, options)` - Upload object
-- `deleteObject(bucket, key)` - Delete object
-- `copyObject(source, dest, options)` - Copy object
-- `headObject(bucket, key)` - Get metadata without downloading
-- `moveObject(source, dest, options)` - Move object (copy + delete)
-- `objectExists(bucket, key)` - Check if object exists
-
-**Advanced Operations (SDK mode only):**
-- `createMultipartUpload(bucket, key, options)` - Start multipart upload
-- `uploadPart(bucket, key, uploadId, partNumber, body)` - Upload part
-- `completeMultipartUpload(bucket, key, uploadId, parts)` - Complete upload
-- `abortMultipartUpload(bucket, key, uploadId)` - Abort upload
-- `selectObjectContent(bucket, key, expression, type)` - S3 Select query
-- `uploadLargeFile(bucket, key, data, options)` - Helper for large files
-- `queryCSV(bucket, key, query)` - Query CSV with S3 Select
-
-**Example Usage:**
-
-```typescript
-// Default - uses AWS SDK v3
-const s3 = new S3Service({
-  accessKeyId: 'xxx',
-  secretAccessKey: 'yyy',
-  region: 'us-east-1'
-});
-
-// Basic operations
-const objects = await s3.listObjects('my-bucket', { prefix: 'incoming/' });
-const content = await s3.getObject('my-bucket', 'file.csv');
-await s3.putObject('my-bucket', 'output.json', JSON.stringify(data));
-
-// Advanced - multipart upload for large files
-await s3.uploadLargeFile('my-bucket', 'large.parquet', buffer, {
-  partSize: 10 * 1024 * 1024 // 10MB parts
-});
-
-// Presigned mode (for Versori)
-const s3Presigned = new S3Service({
-  accessKeyId: 'xxx',
-  secretAccessKey: 'yyy',
-  region: 'us-east-1',
-  usePresigned: true,
-  fetch: ctx.fetch
-});
-
-// Basic operations work in presigned mode
-await s3Presigned.putObject('my-bucket', 'output.json', data);
-```
-
-### 10. Validation Services
-
-**Location:** `src/services/validation/`
-
-Data and schema validation services.
-
-```mermaid
-graph TB
-    subgraph "Validation Services"
-        SCHEMA[SchemaValidationService]
-        PREFLIGHT[PreflightValidator]
-    end
-
-    SCHEMA --> MAPPING[Validate Mappings]
-    SCHEMA --> CONFIG[Validate Config]
-    SCHEMA --> GRAPHQL[Validate GraphQL Schema]
-
-    PREFLIGHT --> REQUIRED[Required Fields]
-    PREFLIGHT --> TYPE[Type Checking]
-    PREFLIGHT --> DUPLICATE[Duplicate Detection]
-    PREFLIGHT --> CUSTOM[Custom Validation]
-    PREFLIGHT --> BATCH[Batch Size Check]
-
-    style SCHEMA fill:#e3f2fd
-    style PREFLIGHT fill:#fff3e0
-```
-
-**Key Services:**
-
-#### SchemaValidationService
-**Source:** `src/services/validation/schema-validation-service.ts`
-
-Validates mapping configurations and GraphQL schema compliance.
-
-- Validate mapping configurations
-- Validate GraphQL schema compliance
-- Check field existence in schema
-
-#### PreflightValidator
-**Source:** `src/services/validation/preflight-validator.ts` (lines 1-352)
-Validates data BEFORE sending to Fluent API to catch errors early and save API quota. Pre-flight validation helps identify data quality issues before submission.
-
-**Key Features:**
-- Catch errors early and save API quota
-- Validate against GraphQL schema requirements
-- Check for duplicates and data quality issues
-- Provide actionable error messages
-- Support custom validation functions
-
-**Key Methods:**
-- `validateBatch(records, options)` - Validate a batch of records before sending to API
-- `validateQuick(records, options)` - Quick validation (returns first error only)
-
-**Example Usage:**
-
-```typescript
-const validator = new PreflightValidator(logger);
-
-// Validate batch before sending
-const result = await validator.validateBatch(records, {
-  entityType: 'INVENTORY',
-  requiredFields: ['ref', 'qty'],
-  checkDuplicates: true,
-  validateTypes: true,
-  maxBatchSize: 500,
-  customValidator: (record, index) => {
-    if (record.qty < 0) {
-      return `Record ${index}: qty cannot be negative`;
-    }
-    return null;
-  }
-});
-
-if (!result.isValid) {
-  console.error(`Validation failed: ${result.errors.length} errors`);
-  result.errors.forEach(err => {
-    console.error(`Record ${err.index}: ${err.message}`);
-  });
-  // Don't send to API - fix data first
-  return;
-}
-
-// Validation passed - safe to send to API
-await client.sendBatch(jobId, { entities: records });
-```
+# Service Layer Architecture
+
+This document explains the organization and responsibilities of services within the FC Connect SDK.
+
+The SDK organizes services into logical modules, each with clear responsibilities and minimal coupling.
+
+```mermaid
+graph TB
+    subgraph "Client Layer"
+        CLIENT[FluentClient / FluentVersoriClient]
+    end
+
+    subgraph "Service Modules"
+        BATCH[Batch Services]
+        CORE[Core Services]
+        MAPPING[Mapping Services]
+        TRANSFORM[Transformation Services]
+        RESOLVERS[Resolver Services]
+        INTEGRATION[Integration Services]
+        ORCHESTRATION[Orchestration Services]
+        VALIDATION[Validation Services]
+        MONITORING[Monitoring Services]
+    end
+
+    subgraph "Data Layer"
+        SOURCES[Data Sources]
+        PARSERS[Parsers]
+    end
+
+    CLIENT --> BATCH
+    CLIENT --> MAPPING
+    CLIENT --> TRANSFORM
+    CLIENT --> ORCHESTRATION
+
+    BATCH --> CORE
+    MAPPING --> RESOLVERS
+    MAPPING --> CORE
+    TRANSFORM --> MAPPING
+    ORCHESTRATION --> BATCH
+    ORCHESTRATION --> SOURCES
+    ORCHESTRATION --> PARSERS
+    ORCHESTRATION --> MAPPING
+
+    SOURCES --> PARSERS
+    VALIDATION --> CORE
+
+    style CLIENT fill:#e3f2fd
+    style BATCH fill:#fff3e0
+    style CORE fill:#f3e5f5
+    style MAPPING fill:#e8f5e9
+    style ORCHESTRATION fill:#e1f5ff
+```
+
+## Service Modules
+
+The SDK organizes services into **11 service modules** (10 exported from `services/index.ts`, plus `s3/` exported directly from root):
+
+1. **auth/** - Webhook authentication services
+2. **batch/** - Job creation, batch management, partial batch recovery
+3. **core/** - Logging, state, job tracking, KV storage
+4. **extraction/** - High-level extraction orchestration (includes statistics tracking)
+5. **integration/** - S3 presign, webhook validation
+6. **mapping/** - Field mapping, GraphQL mutation generation
+7. **orchestration/** - High-level ingestion workflows (includes statistics tracking)
+8. **resolvers/** - GraphQL path resolution, SDK resolvers
+9. **s3/** - S3 operations (SDK and presigned implementations) - exported from root, not services/index.ts
+10. **transformation/** - Data transformation and extraction
+11. **validation/** - Schema and preflight validation
+
+**Note:** Statistics and monitoring capabilities are built into `extraction/` and `orchestration/` services, not a separate module.
+
+### 1. Batch Services
+
+**Location:** `src/services/batch/`
+
+Handles job creation, batch management, file archival, and partial batch recovery.
+
+```mermaid
+graph TB
+    subgraph "Batch Services"
+        MANAGER[FluentBatchManager]
+        ARCHIVAL[BatchArchivalService]
+        RECOVERY[PartialBatchRecovery]
+    end
+
+    MANAGER --> ARCHIVAL
+
+    ARCHIVAL --> MOVE[Move Files]
+    ARCHIVAL --> COPY[Copy Files]
+    ARCHIVAL --> DELETE[Delete Files]
+
+    RECOVERY --> RETRY[Retry Failed Records]
+    RECOVERY --> CHECKPOINT[Checkpoint/Resume]
+    RECOVERY --> TRACK[Track Failures]
+
+    style MANAGER fill:#e3f2fd
+    style ARCHIVAL fill:#f3e5f5
+    style RECOVERY fill:#fff3e0
+    style MOVE fill:#e8f5e9
+    style COPY fill:#e8f5e9
+    style DELETE fill:#ffebee
+    style RETRY fill:#fff4e6
+    style CHECKPOINT fill:#fff4e6
+    style TRACK fill:#fff4e6
+```
+
+**Key Services:**
+
+#### FluentBatchManager
+**Source:** `src/services/batch/manager.ts`
+
+- Create and manage batch jobs
+- Send batches to Fluent Commerce
+- Handle job expiration and recreation
+- Archive processed files to S3
+
+**Example Usage:**
+
+```typescript
+const manager = new FluentBatchManager(fluentClient, logger);
+
+// Create job
+const job = await manager.createJob('inventory-update', { retailerId: 'my-retailer' });
+
+// Send batch
+await manager.submitBatch(job.id, {
+  entityType: 'INVENTORY',
+  action: 'UPSERT',
+  entities: inventoryData,
+});
+
+// Check status
+const status = await manager.getJobStatus(job.id);
+```
+
+#### PartialBatchRecovery
+**Source:** `src/services/batch/partial-batch-recovery.ts` (lines 1-357)
+
+Tracks per-record success/failure in batch operations and enables:
+- Retrying only failed records instead of entire batch
+- Checkpoint/resume functionality
+- Detailed error reporting per record
+- Progress tracking
+
+**Key Methods:**
+- `processBatchWithRecovery(records, batchProcessor, options)` - Process batch with automatic recovery and retry
+- `resumeFromCheckpoint(checkpointId, batchProcessor, options)` - Resume from checkpoint
+- `getCheckpoint(checkpointId)` - Get checkpoint by ID
+- `listCheckpoints()` - List all checkpoints
+- `deleteCheckpoint(checkpointId)` - Delete checkpoint
+
+**Example Usage:**
+
+```typescript
+const recovery = new PartialBatchRecovery(logger);
+
+// Process batch with recovery tracking
+const result = await recovery.processBatchWithRecovery(
+  records,
+  async (batch) => await client.sendBatch(jobId, batch),
+  {
+    maxRetries: 3,
+    retryOnlyFailed: true,
+    checkpointKey: 'inventory-2025-01-24'
+  }
+);
+
+console.log(`Success: ${result.successCount}, Failed: ${result.failedCount}`);
+if (result.failedRecords.length > 0) {
+  console.log('Failed records:', result.failedRecords);
+}
+```
+
+### 2. Core Services
+
+**Location:** `src/services/core/`
+
+Foundation services used across the SDK.
+
+```mermaid
+graph TB
+    subgraph "Core Services"
+        STATE[StateService]
+        KV[KVStorageAdapter]
+        TRACKER[JobTracker]
+    end
+
+    LOG --> CONSOLE[createConsoleLogger]
+    STATE --> KV
+    STATE --> LOCK[Distributed Locking]
+    STATE --> DEDUP[Deduplication]
+
+    KV --> VERSORI[VersoriKVAdapter]
+    KV --> CUSTOM[Custom Implementations]
+
+    TRACKER --> KV
+    TRACKER --> CREATE[Create Job]
+    TRACKER --> UPDATE[Update Status]
+    TRACKER --> COMPLETE[Mark Complete/Failed]
+
+    style LOG fill:#fff3e0
+    style STATE fill:#f3e5f5
+    style KV fill:#e8f5e9
+    style TRACKER fill:#e1f5ff
+#### Logging Utilities
+
+## Logging Architecture
+
+### Function-Based Pattern
+
+**Versori Platform:**
+```typescript
+// Use native Versori log
+export const workflow = http('POST', '/sync', async (ctx) => {
+  ctx.log('info', 'Processing sync', { recordCount: 100 });
+
+  // For SDK services, wrap native log with toStructuredLogger
+  const logger = toStructuredLogger(ctx.log, { service: 'sync' });
+  const client = await createClient({ ...ctx, log: logger });
+});
+```
+
+**Standalone (Node.js/Deno):**
+```typescript
+// Use function-based utilities
+import {
+  createClient,
+  createConsoleLogger,
+  toStructuredLogger,
+  createWorkflowLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = createConsoleLogger();
+const client = await createClient({ config }, { logger });
+```
+
+**Core Functions:**
+- `createConsoleLogger()` - Simple console logger
+- `toStructuredLogger(logger, context)` - Add context to logger
+- `generateCorrelationId()` - Generate unique request IDs
+- `createWorkflowLogger()` - Pre-configured workflow logger
+- `createServiceLogger()` - Pre-configured service logger
+- `createErrorLogger()` - Error context logger
+
+## Logging Decision Flowchart
+
+```
+Are you writing code for Versori platform?
+│
+├─ YES: Use native ctx.log
+│   │
+│   └─ Need to pass logger to SDK services?
+│       │
+│       └─ YES: Wrap with toStructuredLogger(ctx.log, context)
+│
+└─ NO (Standalone Node.js/Deno):
+    │
+    └─ Use createConsoleLogger() or createWorkflowLogger()
+```
+
+**Quick Reference:**
+- **Versori workflows:** `ctx.log()` directly
+- **Versori + SDK services:** `toStructuredLogger(ctx.log, { service: 'name' })`
+- **Standalone apps:** `createConsoleLogger()`
+
+**See:** [Logging Guide](../../02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md#logging-utilities) for complete patterns.
+
+#### StateService
+**Source:** `src/services/core/state-service.ts` (lines 1-382)
+
+Distributed state management with key-value storage.
+
+**Constructor:** `new StateService(logger?)` - Logger is optional
+
+**Key Features:**
+- Distributed state management with KV storage
+- File deduplication tracking
+- Distributed locking with stale detection
+- Daily job tracking
+- Sync state management
+
+**IMPORTANT:** All methods require `kv` parameter as first argument.
+
+**Key Methods:**
+- `acquireLock(lockName, kv, timeoutMinutes?)` - Acquire distributed lock
+- `releaseLock(lockName, kv)` - Release lock
+- `getSyncState(kv, workflowId?)` - Get sync state
+- `updateSyncState(kv, processedFiles, workflowId?)` - Update sync state
+- `getDailyJob(kv, workflowId)` - Get or check daily job
+- `setDailyJob(kv, workflowId, jobId, expirationHours?)` - Store daily job
+- `isFileProcessed(kv, fileName, workflowId?)` - Check if file processed
+- `getMetrics(kv, workflowId?)` - Get processing metrics
+- `clearState(kv, workflowId)` - Clear all state for workflow
+
+**State Management Example:**
+
+```typescript
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+
+const kvAdapter = new VersoriKVAdapter(openKv());
+const stateService = new StateService(logger);
+
+// Check if file processed (note: kv is first parameter)
+const processed = await stateService.isFileProcessed(
+  kvAdapter,
+  'file-key',
+  'my-workflow'
+);
+
+if (!processed) {
+  // Process file
+  await processFile();
+
+  // Update sync state with processed file metadata
+  await stateService.updateSyncState(
+    kvAdapter,
+    [{
+      fileName: 'file-key',
+      lastModified: new Date().toISOString(),
+      recordCount: 100
+    }],
+    'my-workflow'
+  );
+}
+```
+
+#### JobTracker
+**Source:** `src/services/core/job-tracker.ts` (lines 1-355)
+
+Job status tracking for async workflows. Provides standardized way to track job status in Versori integrations, making it easy to monitor scheduled workflows and webhook-triggered operations.
+
+**Key Features:**
+- Standardized job status tracking (queued, processing, completed, failed)
+- Automatic timestamp management
+- TTL support for automatic cleanup
+- Safe error handling (never throws on KV errors)
+- Detailed logging
+
+**Key Methods:**
+- `createJob(jobId, metadata)` - Creates a new job with 'queued' status
+- `updateJob(jobId, update)` - Updates job status
+- `getJob(jobId)` - Retrieves job status
+- `markCompleted(jobId, details)` - Marks job as completed
+- `markFailed(jobId, error)` - Marks job as failed
+
+**Example Usage:**
+
+```typescript
+const tracker = new JobTracker(kvAdapter, logger);
+
+// Create job
+await tracker.createJob('job_123', {
+  triggeredBy: 'schedule',
+  stage: 'extraction'
+});
+
+// Update progress
+await tracker.updateJob('job_123', {
+  status: 'processing',
+  stage: 'transformation',
+  message: 'Processing 1000 records'
+});
+
+// Mark as completed
+await tracker.markCompleted('job_123', {
+  recordCount: 1000,
+  fileName: 'export.xml'
+});
+```
+
+### 3. Mapping Services
+
+**Location:** `src/services/mapping/`
+
+Universal field mapping for all data transformations.
+
+```mermaid
+graph TB
+    subgraph "Mapping Services"
+        UNIVERSAL[UniversalMapper]
+        MUTATION[GraphQLMutationMapper]
+        PATH[PathResolver]
+    end
+
+    UNIVERSAL --> RESOLVE[Field Resolution]
+    UNIVERSAL --> VALIDATE[Validation]
+    UNIVERSAL --> TRANSFORM[Apply Resolvers]
+
+    MUTATION --> PARSE[Parse Input]
+    MUTATION --> MAP[Apply Mappings]
+    MUTATION --> BUILD[Build Mutation]
+
+    PATH --> JSON[JSONPath Resolution]
+    PATH --> XML[XMLPath Resolution]
+    PATH --> GRAPHQL[GraphQL Path Resolution]
+
+    style UNIVERSAL fill:#e3f2fd
+    style MUTATION fill:#fff3e0
+    style PATH fill:#f3e5f5
+```
+
+**Key Responsibilities:**
+
+- Universal field mapping configuration
+- GraphQL mutation generation from XML/JSON
+- Path resolution for nested data
+- Field validation and transformation
+
+**Universal Mapping Example:**
+
+```typescript
+const mapper = new UniversalMapper({
+  fields: {
+    skuRef: { source: 'sku_id', required: true },
+    qty: { source: 'quantity', resolver: 'sdk.parseInt' },
+    expectedOn: { source: 'date', resolver: 'sdk.formatDate' },
+  },
+});
+
+const result = await mapper.map(sourceData);
+// { success: true, data: { skuRef: '...', qty: 10, expectedOn: '2025-01-15' } }
+```
+
+**GraphQL Mutation Mapping Example:**
+
+```typescript
+const mutationMapper = new GraphQLMutationMapper(xmlParser, logger);
+
+const mutation = await mutationMapper.generateMutation(xmlData, mappingConfig);
+
+// Execute mutation
+const result = await fluentClient.graphql({ query: mutation });
+```
+
+### 4. Transformation & Extraction Services
+
+**Location:** `src/services/transformation/` and `src/services/extraction/`
+
+Data extraction and transformation services.
+
+```mermaid
+graph TB
+    subgraph "Transformation & Extraction Services"
+        ORCH[ExtractionOrchestrator]
+    end
+
+    ORCH --> QUERY[Execute GraphQL Query]
+    ORCH --> AUTOPAGE[Automatic Pagination]
+    ORCH --> PATHEXTRACT[Path-based Extraction]
+    ORCH --> VALIDATE[Per-item Validation]
+    ORCH --> STATS[Statistics Tracking]
+    ORCH --> LIMITS[Limits: maxRecords, maxPages, timeout]
+
+    style ORCH fill:#fff3e0
+```
+
+**Key Services:**
+
+#### ExtractionOrchestrator (Optional Helper)
+**Source:** `src/services/extraction/extraction-orchestrator.ts` (lines 1-460)
+Generic data extraction from Fluent Commerce GraphQL API with automatic pagination, error handling, and statistics tracking. This is an **optional convenience service** for extraction workflows.
+
+> **Note**: For maximum flexibility, consider using `client.graphql()` with auto-pagination directly instead of this orchestrator. Building blocks give you more control over the workflow.
+
+**Key Features:**
+- Automatic pagination handling
+- Extraction limits (maxRecords, maxPages, timeout)
+- Path-based result extraction from nested GraphQL responses
+- Per-item validation
+- Comprehensive error handling
+- Statistics tracking
+
+**Key Methods:**
+- `extract<T>(options)` - Extract data from Fluent Commerce GraphQL API
+  - Returns: `Promise<ExtractionResult<T>>` with data, stats, and errors
+
+**Example Usage:**
+
+```typescript
+const orchestrator = new ExtractionOrchestrator(client, logger);
+
+const result = await orchestrator.extract({
+  query: VIRTUAL_POSITIONS_QUERY,
+  variables: {
+    catalogues: [{ ref: 'DEFAULT:1' }],
+    dateRangeFilter: { from: '2024-01-01' }
+  },
+  resultPath: 'virtualPositions.edges.node',
+  pageSize: 200,
+  maxRecords: 100000,
+  validateItem: (item) => item.ref && item.quantity > 0
+});
+
+console.log(`Extracted ${result.stats.totalRecords} records in ${result.stats.duration}ms`);
+console.log(`Pages fetched: ${result.stats.totalPages}`);
+if (result.stats.truncated) {
+  console.log(`Truncation reason: ${result.stats.truncationReason}`);
+}
+```
+
+**Alternative (Building Blocks - More Flexible):**
+
+For more control, compose `client.graphql()` with auto-pagination and UniversalMapper:
+
+```typescript
+// Direct approach with client auto-pagination
+const result = await client.graphql({
+  query: VIRTUAL_POSITIONS_QUERY,
+  variables: { catalogues: [{ ref: 'DEFAULT:1' }] },
+  pagination: { maxRecords: 100000, pageSize: 200 }
+});
+
+const items = result.data?.virtualPositions?.edges?.map(e => e.node) || [];
+
+// Transform with UniversalMapper if needed
+const mapper = new UniversalMapper(mappingConfig);
+const transformed = await Promise.all(items.map(item => mapper.map(item)));
+
+console.log(`Extracted ${items.length} records`);
+```
+
+> Note: Any references below to Orchestration are historical and should be interpreted as building-block patterns using clients + services.
+
+### 5. Resolver Services
+
+**Location:** `src/services/resolvers/`
+
+Built-in transformation functions (SDK resolvers) and helper utilities.
+
+**SDK Resolver Categories:**
+
+```typescript
+// String transformations
+('sdk.uppercase', 'sdk.lowercase', 'sdk.trim', 'sdk.toString');
+
+// Number parsing
+('sdk.parseInt', 'sdk.parseFloat', 'sdk.number');
+
+// Date formatting
+('sdk.formatDate', 'sdk.formatDateShort', 'sdk.parseDate');
+
+// Type conversions
+('sdk.boolean', 'sdk.parseJson', 'sdk.toJson');
+
+// Utilities
+('sdk.identity', 'sdk.coalesce');
+```
+
+### 6. Integration Services
+
+**Location:** `src/services/integration/`
+
+External system integration utilities.
+
+```mermaid
+graph TB
+    subgraph "Integration Services"
+        S3[S3PresignService]
+        WEBHOOK[WebhookValidationService]
+    end
+
+    S3 --> PRESIGN[Generate Presigned URLs]
+    S3 --> UPLOAD[Upload via Presigned URL]
+    S3 --> DOWNLOAD[Download via Presigned URL]
+
+    WEBHOOK --> VALIDATE[Cryptographic Signature Validation]
+    WEBHOOK --> PARSE[Parse Payloads]
+    WEBHOOK --> VERIFY[Verify Timestamps]
+    WEBHOOK --> CLIENT[FluentClient.validateWebhook()]
+
+    style S3 fill:#e3f2fd
+    style WEBHOOK fill:#fff3e0
+```
+
+**Key Services:**
+
+#### S3PresignService
+Generate S3 presigned URLs for cross-account access without sharing credentials.
+
+#### WebhookValidationService
+**Integration:** Accessible via `FluentClient.validateWebhook()`
+
+Cryptographic signature validation for Fluent Commerce webhooks.
+
+**Key Responsibilities:**
+
+- Generate S3 presigned URLs (SigV4)
+- Validate webhook signatures (HMAC-SHA256)
+- Parse webhook payloads
+- Support cross-account S3 access
+- Verify webhook timestamp freshness
+
+**S3 Presign Example:**
+
+```typescript
+const s3PresignService = new S3PresignService(config);
+
+// Generate upload URL
+const uploadUrl = await s3PresignService.getPresignedUploadUrl(
+  'my-bucket',
+  'path/to/file.csv',
+  3600 // 1 hour expiration
+);
+
+// Upload file
+await fetch(uploadUrl, {
+  method: 'PUT',
+  body: fileData,
+});
+```
+
+**Webhook Validation Example:**
+
+```typescript
+// Via FluentClient (recommended) - returns boolean
+const isValid = await client.validateWebhook(
+  webhookBody,
+  request.headers['x-fluent-signature'],
+  rawRequestBody
+);
+
+if (!isValid) {
+  throw new Error('Invalid webhook signature');
+}
+
+// Process validated webhook
+console.log('Event type:', webhookBody.name);
+```
+
+### 8. Auth Services
+
+**Location:** `src/services/auth/`
+
+Webhook authentication services for API key and token management.
+
+**Key Services:**
+
+#### WebhookAuthService
+
+Provides API key and token validation for webhook endpoints.
+
+**Key Features:**
+- API key validation
+- Token issuance and validation
+- User validation
+- Support for multiple authentication methods
+
+### 9. S3 Services
+
+**Location:** `src/services/s3/`
+
+**Note:** S3Service is exported directly from the SDK root (`src/index.ts`), not from `services/index.ts`.
+
+Unified S3 operations with dual implementation strategies.
+
+```mermaid
+graph TB
+    subgraph "S3 Services"
+        S3SERVICE[S3Service]
+        SDK_IMPL[SDKImplementation<br/>AWS SDK v3]
+        PRESIGNED_IMPL[PresignedImplementation<br/>SigV4 URLs]
+    end
+
+    S3SERVICE --> SDK_IMPL
+    S3SERVICE --> PRESIGNED_IMPL
+
+    SDK_IMPL --> BASIC[Basic Ops<br/>get, put, list, delete]
+    SDK_IMPL --> ADVANCED[Advanced Ops<br/>multipart, S3 Select]
+
+    PRESIGNED_IMPL --> BASIC_PS[Basic Ops<br/>get, put, list, delete]
+
+    style S3SERVICE fill:#e3f2fd
+    style SDK_IMPL fill:#e8f5e9
+    style PRESIGNED_IMPL fill:#fff3e0
+```
+
+**Key Services:**
+
+#### S3Service
+**Source:** `src/services/s3/s3-service.ts`
+
+Main interface for S3 operations with automatic implementation selection.
+
+**Two Implementation Modes:**
+
+1. **SDK Mode (Default)** - Uses AWS SDK v3
+   - Full feature set including multipart upload and S3 Select
+   - Works in Node.js and environments with AWS SDK support
+   - Best for production workloads with large files
+
+2. **Presigned Mode (Explicit)** - Uses presigned URLs
+   - Works in restricted environments (browsers, Versori platform)
+   - Basic operations only (no multipart, no S3 Select)
+   - Requires `ctx.fetch` when used in Versori
+
+**Key Methods:**
+
+**Basic Operations (Both modes):**
+- `listObjects(bucket, options)` - List objects in bucket
+- `getObject(bucket, key)` - Get object content
+- `putObject(bucket, key, body, options)` - Upload object
+- `deleteObject(bucket, key)` - Delete object
+- `copyObject(source, dest, options)` - Copy object
+- `headObject(bucket, key)` - Get metadata without downloading
+- `moveObject(source, dest, options)` - Move object (copy + delete)
+- `objectExists(bucket, key)` - Check if object exists
+
+**Advanced Operations (SDK mode only):**
+- `createMultipartUpload(bucket, key, options)` - Start multipart upload
+- `uploadPart(bucket, key, uploadId, partNumber, body)` - Upload part
+- `completeMultipartUpload(bucket, key, uploadId, parts)` - Complete upload
+- `abortMultipartUpload(bucket, key, uploadId)` - Abort upload
+- `selectObjectContent(bucket, key, expression, type)` - S3 Select query
+- `uploadLargeFile(bucket, key, data, options)` - Helper for large files
+- `queryCSV(bucket, key, query)` - Query CSV with S3 Select
+
+**Example Usage:**
+
+```typescript
+// Default - uses AWS SDK v3
+const s3 = new S3Service({
+  accessKeyId: 'xxx',
+  secretAccessKey: 'yyy',
+  region: 'us-east-1'
+});
+
+// Basic operations
+const objects = await s3.listObjects('my-bucket', { prefix: 'incoming/' });
+const content = await s3.getObject('my-bucket', 'file.csv');
+await s3.putObject('my-bucket', 'output.json', JSON.stringify(data));
+
+// Advanced - multipart upload for large files
+await s3.uploadLargeFile('my-bucket', 'large.parquet', buffer, {
+  partSize: 10 * 1024 * 1024 // 10MB parts
+});
+
+// Presigned mode (for Versori)
+const s3Presigned = new S3Service({
+  accessKeyId: 'xxx',
+  secretAccessKey: 'yyy',
+  region: 'us-east-1',
+  usePresigned: true,
+  fetch: ctx.fetch
+});
+
+// Basic operations work in presigned mode
+await s3Presigned.putObject('my-bucket', 'output.json', data);
+```
+
+### 10. Validation Services
+
+**Location:** `src/services/validation/`
+
+Data and schema validation services.
+
+```mermaid
+graph TB
+    subgraph "Validation Services"
+        SCHEMA[SchemaValidationService]
+        PREFLIGHT[PreflightValidator]
+    end
+
+    SCHEMA --> MAPPING[Validate Mappings]
+    SCHEMA --> CONFIG[Validate Config]
+    SCHEMA --> GRAPHQL[Validate GraphQL Schema]
+
+    PREFLIGHT --> REQUIRED[Required Fields]
+    PREFLIGHT --> TYPE[Type Checking]
+    PREFLIGHT --> DUPLICATE[Duplicate Detection]
+    PREFLIGHT --> CUSTOM[Custom Validation]
+    PREFLIGHT --> BATCH[Batch Size Check]
+
+    style SCHEMA fill:#e3f2fd
+    style PREFLIGHT fill:#fff3e0
+```
+
+**Key Services:**
+
+#### SchemaValidationService
+**Source:** `src/services/validation/schema-validation-service.ts`
+
+Validates mapping configurations and GraphQL schema compliance.
+
+- Validate mapping configurations
+- Validate GraphQL schema compliance
+- Check field existence in schema
+
+#### PreflightValidator
+**Source:** `src/services/validation/preflight-validator.ts` (lines 1-352)
+Validates data BEFORE sending to Fluent API to catch errors early and save API quota. Pre-flight validation helps identify data quality issues before submission.
+
+**Key Features:**
+- Catch errors early and save API quota
+- Validate against GraphQL schema requirements
+- Check for duplicates and data quality issues
+- Provide actionable error messages
+- Support custom validation functions
+
+**Key Methods:**
+- `validateBatch(records, options)` - Validate a batch of records before sending to API
+- `validateQuick(records, options)` - Quick validation (returns first error only)
+
+**Example Usage:**
+
+```typescript
+const validator = new PreflightValidator(logger);
+
+// Validate batch before sending
+const result = await validator.validateBatch(records, {
+  entityType: 'INVENTORY',
+  requiredFields: ['ref', 'qty'],
+  checkDuplicates: true,
+  validateTypes: true,
+  maxBatchSize: 500,
+  customValidator: (record, index) => {
+    if (record.qty < 0) {
+      return `Record ${index}: qty cannot be negative`;
+    }
+    return null;
+  }
+});
+
+if (!result.isValid) {
+  console.error(`Validation failed: ${result.errors.length} errors`);
+  result.errors.forEach(err => {
+    console.error(`Record ${err.index}: ${err.message}`);
+  });
+  // Don't send to API - fix data first
+  return;
+}
+
+// Validation passed - safe to send to API
+await client.sendBatch(jobId, { entities: records });
+```