@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md

@@ -1,407 +1,407 @@
-# Advanced Services
-
-**Level:** Advanced
-**Category:** Production-Ready Services
-
-## Overview
-
-Advanced services provide production-grade capabilities for resilience, monitoring, and operational excellence in Fluent Commerce integrations. These services handle complex scenarios like partial failures, job tracking, and batch archival.
-
-## When to Use Advanced Services
-
-Use these services when you need:
-- **Resilience** - Recover from partial failures without losing all data
-- **Observability** - Track job progress and status across workflows
-- **Compliance** - Archive batches for audit trails
-- **Production-Grade** - Enterprise-ready error handling and recovery
-
-## Quick Links
-
-- [JobTracker](#jobtracker) - Track job lifecycle and status
-- [PartialBatchRecovery](#partialbatchrecovery) - Recover from partial batch failures
-- [BatchArchivalService](#batcharchivalservice) - Archive batches to S3
-- [Quick Reference](./advanced-services-quick-reference.md) - One-page cheatsheet
-
----
-
-## JobTracker
-
-Track job status throughout the workflow lifecycle with automatic timestamp management and TTL support.
-
-**Use Cases:**
-- Monitor scheduled workflow progress
-- Track webhook-triggered operations
-- Implement job retry logic
-- Dashboard status displays
-
-**Key Features:**
-- Standardized status values (queued, processing, completed, failed)
-- Automatic timestamp tracking (createdAt, startedAt, completedAt, failedAt)
-- TTL support for automatic cleanup (default: 7 days)
-- Safe error handling (never throws on KV errors)
-
-**Quick Example:**
-```typescript
-import { JobTracker, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-
-const tracker = new JobTracker(kvAdapter, logger);
-
-// Create job
-await tracker.createJob('scheduled_1729681200000', {
-  triggeredBy: 'schedule',
-  stage: 'extraction',
-  details: { catalogueRef: 'DEFAULT:1' }
-});
-
-// Update progress
-await tracker.updateJob('scheduled_1729681200000', {
-  status: 'processing',
-  stage: 'transformation',
-  message: 'Processing 1000 records'
-});
-
-// Mark complete
-await tracker.markCompleted('scheduled_1729681200000', {
-  recordCount: 1000,
-  fileName: 'export.xml'
-});
-```
-
-**Learn More:** [JobTracker Guide](./advanced-services-job-tracker.md)
-
----
-
-## PartialBatchRecovery
-
-Recover from partial batch failures by retrying only failed records instead of the entire batch.
-
-**Use Cases:**
-- Handle transient API failures gracefully
-- Implement checkpoint/resume functionality
-- Retry individual failed records
-- Track detailed error information per record
-
-**Key Features:**
-- Per-record success/failure tracking
-- Configurable retry logic (retry only failed or entire batch)
-- Checkpoint support for resume functionality
-- Detailed error reporting with timestamps
-- Custom retry predicates
-
-**Quick Example:**
-```typescript
-import { PartialBatchRecovery } from '@fluentcommerce/fc-connect-sdk';
-
-const recovery = new PartialBatchRecovery(logger);
-
-const result = await recovery.processBatchWithRecovery(
-  inventoryRecords,
-  async (batch) => await client.sendBatch(jobId, {
-    action: 'UPSERT',
-    entityType: 'INVENTORY',
-    entities: batch
-  }),
-  {
-    maxRetries: 3,
-    retryOnlyFailed: true,
-    checkpointKey: 'inventory-2025-01-30'
-  }
-);
-
-console.log(`Success: ${result.successCount}, Failed: ${result.failedCount}`);
-if (result.failedRecords.length > 0) {
-  // Handle failed records
-  await handleFailures(result.failedRecords);
-}
-```
-
-**Learn More:** [PartialBatchRecovery Guide](./advanced-services-partial-batch-recovery.md)
-
----
-
-## BatchArchivalService
-
-Archive batch payloads to S3 with hierarchical time-based paths for compliance and audit trails.
-
-**Use Cases:**
-- Compliance and audit requirements
-- Batch history tracking
-- Debugging batch issues
-- Data retention policies
-
-**Key Features:**
-- Hierarchical S3 paths: `{basePath}/{YYYY}/{MM}/{DD}/{HH}/{MM}/batch-{jobId}-{batchId}.json`
-- Includes metadata (clientId, archivedAt, batch status)
-- Presigned URL uploads for security
-- Bulk archival support
-- Automatic timestamp tracking
-
-**Quick Example:**
-```typescript
-import { BatchArchivalService, S3PresignService } from '@fluentcommerce/fc-connect-sdk';
-
-const archival = new BatchArchivalService(
-  {
-    bucketName: 'fluent-batch-archives',
-    region: 'us-east-1',
-    credentials: { accessKeyId, secretAccessKey },
-    basePath: 'production/batches',
-    expiresSeconds: 300
-  },
-  new S3PresignService(),
-  logger
-);
-
-const result = await archival.archiveBatch({
-  batch: batchPayload,
-  batchResponse: { id: 'batch_123', jobId: 'job_456', status: 'COMPLETED' },
-  jobId: 'job_456'
-}, 'client-123');
-
-console.log(`Archived to: ${result.archivedPath}`);
-```
-
-**Learn More:** [BatchArchivalService Guide](./advanced-services-batch-archival.md)
-
----
-
-
-Manage KV store operations including list, export, bulk operations, and debugging.
-
-**Use Cases:**
-- Debug KV state in production
-- Export/import KV data for backup
-- Bulk delete operations
-- KV store migrations
-
-**Key Features:**
-- List all KV entries with prefix filtering
-- Export KV data to JSON
-- Bulk operations (set, delete)
-- Safe iteration with error handling
-- Progress callbacks for long operations
-
-**Quick Example:**
-```typescript
-
-
-// List all entries
-const entries = await kvManager.listKeys({ prefix: 'job:' });
-console.log(`Found ${entries.length} job entries`);
-
-// Export to JSON
-const exportData = await kvManager.exportKeys({ prefix: 'state:' });
-await fs.writeFile('kv-backup.json', JSON.stringify(exportData, null, 2));
-
-// Bulk delete entries
-await kvManager.bulkDelete([
-  ['job', 'old_job_1'],
-  ['job', 'old_job_2']
-]);
-```
-
-
----
-
-## Comparison Matrix
-
-| Service | Purpose | When to Use | Complexity | Impact |
-|---------|---------|-------------|------------|---------|
-| **JobTracker** | Job lifecycle tracking | Always in production workflows | Low | High |
-| **PartialBatchRecovery** | Partial failure recovery | When batch failures are costly | Medium | High |
-| **BatchArchivalService** | Audit trails | Compliance requirements | Low | Medium |
-
----
-
-## Patterns
-
-### 1. Track Jobs
-```typescript
-await tracker.createJob(jobId, { triggeredBy: 'schedule' });
-try {
-  await processWorkflow();
-  await tracker.markCompleted(jobId, { recordCount: 1000 });
-} catch (error) {
-  await tracker.markFailed(jobId, error);
-}
-```
-
-### 2. Use Partial Recovery for Critical Data
-```typescript
-const result = await recovery.processBatchWithRecovery(
-  records,
-  batchProcessor,
-  { maxRetries: 3, retryOnlyFailed: true }
-);
-
-if (result.failedRecords.length > 0) {
-  await sendToDeadLetterQueue(result.failedRecords);
-}
-```
-
-### 3. Archive Batches for Audit Trails
-```typescript
-const batchResponse = await client.sendBatch(jobId, batch);
-await archival.archiveBatch({
-  batch,
-  batchResponse,
-  jobId
-}, clientId);
-```
-
-### 4. Export KV State for Analysis
-```typescript
-const entries = await kvManager.listKeys({ prefix: 'job:' });
-const failedJobs = entries.filter(e => e.value.status === 'failed');
-console.log('Failed jobs:', failedJobs);
-```
-
----
-
-## Integration Patterns
-
-### Complete Production Workflow
-
-```typescript
-import {
-  createClient,
-  JobTracker,
-  PartialBatchRecovery,
-  BatchArchivalService,
-  S3PresignService,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function productionWorkflow(ctx: any) {
-  const { log, openKv } = ctx;
-  const kv = new VersoriKVAdapter(openKv(':project:'));
-
-  // Initialize services
-  const client = await createClient(ctx);
-  const tracker = new JobTracker(kv, log);
-  const recovery = new PartialBatchRecovery(log);
-  const archival = new BatchArchivalService(archivalConfig, new S3PresignService(), log);
-
-  const jobId = `scheduled_${Date.now()}`;
-
-  try {
-    // 1. Create job
-    await tracker.createJob(jobId, {
-      triggeredBy: 'schedule',
-      stage: 'initialization'
-    });
-
-    // 2. Extract data
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'extraction',
-      message: 'Extracting inventory data'
-    });
-
-    const records = await extractInventory(client);
-
-    // 3. Process with recovery
-    await tracker.updateJob(jobId, {
-      stage: 'transformation',
-      message: `Processing ${records.length} records`
-    });
-
-    const result = await recovery.processBatchWithRecovery(
-      records,
-      async (batch) => {
-        const batchPayload = {
-          action: 'UPSERT',
-          entityType: 'INVENTORY',
-          source: 'SCHEDULED_SYNC',
-          event: 'INVENTORY_UPDATE',
-          entities: batch
-        };
-
-        const response = await client.sendBatch(jobId, batchPayload);
-
-        // Archive successful batch
-        await archival.archiveBatch({
-          batch: batchPayload,
-          batchResponse: response,
-          jobId
-        });
-
-        return response;
-      },
-      {
-        maxRetries: 3,
-        retryOnlyFailed: true,
-        checkpointKey: `inventory-${Date.now()}`
-      }
-    );
-
-    // 4. Mark complete
-    await tracker.markCompleted(jobId, {
-      recordCount: result.successCount,
-      failedCount: result.failedCount,
-      durationMs: result.durationMs
-    });
-
-    log.info('Workflow completed successfully', {
-      jobId,
-      successCount: result.successCount,
-      failedCount: result.failedCount
-    });
-
-  } catch (error) {
-    // Mark job as failed
-    await tracker.markFailed(jobId, error);
-    throw error;
-  }
-}
-```
-
----
-
-## Error Handling
-
-All advanced services follow these error handling principles:
-
-1. **JobTracker** - Never throws on KV errors, logs warnings
-2. **PartialBatchRecovery** - Returns detailed failure information
-3. **BatchArchivalService** - Returns success/failure result objects
-
-```typescript
-// All services are safe to use without try/catch
-await tracker.createJob(jobId, metadata); // Never throws
-
-// But recovery needs result checking
-const result = await recovery.processBatchWithRecovery(records, processor);
-if (result.failedCount > 0) {
-  // Handle failures
-}
-
-// Archival returns result object
-const archiveResult = await archival.archiveBatch(batch);
-if (!archiveResult.success) {
-  log.error('Archival failed', { error: archiveResult.error });
-}
-```
-
----
-
-## Performance Considerations
-
-- **JobTracker**: Minimal overhead, ~10ms per KV operation
-- **PartialBatchRecovery**: Adds ~5-10% overhead for tracking
-- **BatchArchivalService**: Async S3 uploads, doesn't block main workflow
-
----
-
-## See Also
-
-- [Ingestion Guide](advanced-services-readme.md) - Core ingestion workflows
-- [State Management](../ingestion/modules/02-core-guides-ingestion-07-state-management.md) - Basic state tracking
-- [Monitoring](advanced-services-readme.md) - Processing statistics
-- [Error Handling](../api-reference/modules/api-reference-09-error-handling.md) - Error strategies
-
----
-
-## Quick Reference
-
-See [Advanced Services Quick Reference](./advanced-services-quick-reference.md) for a one-page cheatsheet.
+# Advanced Services
+
+**Level:** Advanced
+**Category:** Production-Ready Services
+
+## Overview
+
+Advanced services provide production-grade capabilities for resilience, monitoring, and operational excellence in Fluent Commerce integrations. These services handle complex scenarios like partial failures, job tracking, and batch archival.
+
+## When to Use Advanced Services
+
+Use these services when you need:
+- **Resilience** - Recover from partial failures without losing all data
+- **Observability** - Track job progress and status across workflows
+- **Compliance** - Archive batches for audit trails
+- **Production-Grade** - Enterprise-ready error handling and recovery
+
+## Quick Links
+
+- [JobTracker](#jobtracker) - Track job lifecycle and status
+- [PartialBatchRecovery](#partialbatchrecovery) - Recover from partial batch failures
+- [BatchArchivalService](#batcharchivalservice) - Archive batches to S3
+- [Quick Reference](./advanced-services-quick-reference.md) - One-page cheatsheet
+
+---
+
+## JobTracker
+
+Track job status throughout the workflow lifecycle with automatic timestamp management and TTL support.
+
+**Use Cases:**
+- Monitor scheduled workflow progress
+- Track webhook-triggered operations
+- Implement job retry logic
+- Dashboard status displays
+
+**Key Features:**
+- Standardized status values (queued, processing, completed, failed)
+- Automatic timestamp tracking (createdAt, startedAt, completedAt, failedAt)
+- TTL support for automatic cleanup (default: 7 days)
+- Safe error handling (never throws on KV errors)
+
+**Quick Example:**
+```typescript
+import { JobTracker, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+
+const tracker = new JobTracker(kvAdapter, logger);
+
+// Create job
+await tracker.createJob('scheduled_1729681200000', {
+  triggeredBy: 'schedule',
+  stage: 'extraction',
+  details: { catalogueRef: 'DEFAULT:1' }
+});
+
+// Update progress
+await tracker.updateJob('scheduled_1729681200000', {
+  status: 'processing',
+  stage: 'transformation',
+  message: 'Processing 1000 records'
+});
+
+// Mark complete
+await tracker.markCompleted('scheduled_1729681200000', {
+  recordCount: 1000,
+  fileName: 'export.xml'
+});
+```
+
+**Learn More:** [JobTracker Guide](./advanced-services-job-tracker.md)
+
+---
+
+## PartialBatchRecovery
+
+Recover from partial batch failures by retrying only failed records instead of the entire batch.
+
+**Use Cases:**
+- Handle transient API failures gracefully
+- Implement checkpoint/resume functionality
+- Retry individual failed records
+- Track detailed error information per record
+
+**Key Features:**
+- Per-record success/failure tracking
+- Configurable retry logic (retry only failed or entire batch)
+- Checkpoint support for resume functionality
+- Detailed error reporting with timestamps
+- Custom retry predicates
+
+**Quick Example:**
+```typescript
+import { PartialBatchRecovery } from '@fluentcommerce/fc-connect-sdk';
+
+const recovery = new PartialBatchRecovery(logger);
+
+const result = await recovery.processBatchWithRecovery(
+  inventoryRecords,
+  async (batch) => await client.sendBatch(jobId, {
+    action: 'UPSERT',
+    entityType: 'INVENTORY',
+    entities: batch
+  }),
+  {
+    maxRetries: 3,
+    retryOnlyFailed: true,
+    checkpointKey: 'inventory-2025-01-30'
+  }
+);
+
+console.log(`Success: ${result.successCount}, Failed: ${result.failedCount}`);
+if (result.failedRecords.length > 0) {
+  // Handle failed records
+  await handleFailures(result.failedRecords);
+}
+```
+
+**Learn More:** [PartialBatchRecovery Guide](./advanced-services-partial-batch-recovery.md)
+
+---
+
+## BatchArchivalService
+
+Archive batch payloads to S3 with hierarchical time-based paths for compliance and audit trails.
+
+**Use Cases:**
+- Compliance and audit requirements
+- Batch history tracking
+- Debugging batch issues
+- Data retention policies
+
+**Key Features:**
+- Hierarchical S3 paths: `{basePath}/{YYYY}/{MM}/{DD}/{HH}/{MM}/batch-{jobId}-{batchId}.json`
+- Includes metadata (clientId, archivedAt, batch status)
+- Presigned URL uploads for security
+- Bulk archival support
+- Automatic timestamp tracking
+
+**Quick Example:**
+```typescript
+import { BatchArchivalService, S3PresignService } from '@fluentcommerce/fc-connect-sdk';
+
+const archival = new BatchArchivalService(
+  {
+    bucketName: 'fluent-batch-archives',
+    region: 'us-east-1',
+    credentials: { accessKeyId, secretAccessKey },
+    basePath: 'production/batches',
+    expiresSeconds: 300
+  },
+  new S3PresignService(),
+  logger
+);
+
+const result = await archival.archiveBatch({
+  batch: batchPayload,
+  batchResponse: { id: 'batch_123', jobId: 'job_456', status: 'COMPLETED' },
+  jobId: 'job_456'
+}, 'client-123');
+
+console.log(`Archived to: ${result.archivedPath}`);
+```
+
+**Learn More:** [BatchArchivalService Guide](./advanced-services-batch-archival.md)
+
+---
+
+
+Manage KV store operations including list, export, bulk operations, and debugging.
+
+**Use Cases:**
+- Debug KV state in production
+- Export/import KV data for backup
+- Bulk delete operations
+- KV store migrations
+
+**Key Features:**
+- List all KV entries with prefix filtering
+- Export KV data to JSON
+- Bulk operations (set, delete)
+- Safe iteration with error handling
+- Progress callbacks for long operations
+
+**Quick Example:**
+```typescript
+
+
+// List all entries
+const entries = await kvManager.listKeys({ prefix: 'job:' });
+console.log(`Found ${entries.length} job entries`);
+
+// Export to JSON
+const exportData = await kvManager.exportKeys({ prefix: 'state:' });
+await fs.writeFile('kv-backup.json', JSON.stringify(exportData, null, 2));
+
+// Bulk delete entries
+await kvManager.bulkDelete([
+  ['job', 'old_job_1'],
+  ['job', 'old_job_2']
+]);
+```
+
+
+---
+
+## Comparison Matrix
+
+| Service | Purpose | When to Use | Complexity | Impact |
+|---------|---------|-------------|------------|---------|
+| **JobTracker** | Job lifecycle tracking | Always in production workflows | Low | High |
+| **PartialBatchRecovery** | Partial failure recovery | When batch failures are costly | Medium | High |
+| **BatchArchivalService** | Audit trails | Compliance requirements | Low | Medium |
+
+---
+
+## Patterns
+
+### 1. Track Jobs
+```typescript
+await tracker.createJob(jobId, { triggeredBy: 'schedule' });
+try {
+  await processWorkflow();
+  await tracker.markCompleted(jobId, { recordCount: 1000 });
+} catch (error) {
+  await tracker.markFailed(jobId, error);
+}
+```
+
+### 2. Use Partial Recovery for Critical Data
+```typescript
+const result = await recovery.processBatchWithRecovery(
+  records,
+  batchProcessor,
+  { maxRetries: 3, retryOnlyFailed: true }
+);
+
+if (result.failedRecords.length > 0) {
+  await sendToDeadLetterQueue(result.failedRecords);
+}
+```
+
+### 3. Archive Batches for Audit Trails
+```typescript
+const batchResponse = await client.sendBatch(jobId, batch);
+await archival.archiveBatch({
+  batch,
+  batchResponse,
+  jobId
+}, clientId);
+```
+
+### 4. Export KV State for Analysis
+```typescript
+const entries = await kvManager.listKeys({ prefix: 'job:' });
+const failedJobs = entries.filter(e => e.value.status === 'failed');
+console.log('Failed jobs:', failedJobs);
+```
+
+---
+
+## Integration Patterns
+
+### Complete Production Workflow
+
+```typescript
+import {
+  createClient,
+  JobTracker,
+  PartialBatchRecovery,
+  BatchArchivalService,
+  S3PresignService,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function productionWorkflow(ctx: any) {
+  const { log, openKv } = ctx;
+  const kv = new VersoriKVAdapter(openKv(':project:'));
+
+  // Initialize services
+  const client = await createClient(ctx);
+  const tracker = new JobTracker(kv, log);
+  const recovery = new PartialBatchRecovery(log);
+  const archival = new BatchArchivalService(archivalConfig, new S3PresignService(), log);
+
+  const jobId = `scheduled_${Date.now()}`;
+
+  try {
+    // 1. Create job
+    await tracker.createJob(jobId, {
+      triggeredBy: 'schedule',
+      stage: 'initialization'
+    });
+
+    // 2. Extract data
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'extraction',
+      message: 'Extracting inventory data'
+    });
+
+    const records = await extractInventory(client);
+
+    // 3. Process with recovery
+    await tracker.updateJob(jobId, {
+      stage: 'transformation',
+      message: `Processing ${records.length} records`
+    });
+
+    const result = await recovery.processBatchWithRecovery(
+      records,
+      async (batch) => {
+        const batchPayload = {
+          action: 'UPSERT',
+          entityType: 'INVENTORY',
+          source: 'SCHEDULED_SYNC',
+          event: 'INVENTORY_UPDATE',
+          entities: batch
+        };
+
+        const response = await client.sendBatch(jobId, batchPayload);
+
+        // Archive successful batch
+        await archival.archiveBatch({
+          batch: batchPayload,
+          batchResponse: response,
+          jobId
+        });
+
+        return response;
+      },
+      {
+        maxRetries: 3,
+        retryOnlyFailed: true,
+        checkpointKey: `inventory-${Date.now()}`
+      }
+    );
+
+    // 4. Mark complete
+    await tracker.markCompleted(jobId, {
+      recordCount: result.successCount,
+      failedCount: result.failedCount,
+      durationMs: result.durationMs
+    });
+
+    log.info('Workflow completed successfully', {
+      jobId,
+      successCount: result.successCount,
+      failedCount: result.failedCount
+    });
+
+  } catch (error) {
+    // Mark job as failed
+    await tracker.markFailed(jobId, error);
+    throw error;
+  }
+}
+```
+
+---
+
+## Error Handling
+
+All advanced services follow these error handling principles:
+
+1. **JobTracker** - Never throws on KV errors, logs warnings
+2. **PartialBatchRecovery** - Returns detailed failure information
+3. **BatchArchivalService** - Returns success/failure result objects
+
+```typescript
+// All services are safe to use without try/catch
+await tracker.createJob(jobId, metadata); // Never throws
+
+// But recovery needs result checking
+const result = await recovery.processBatchWithRecovery(records, processor);
+if (result.failedCount > 0) {
+  // Handle failures
+}
+
+// Archival returns result object
+const archiveResult = await archival.archiveBatch(batch);
+if (!archiveResult.success) {
+  log.error('Archival failed', { error: archiveResult.error });
+}
+```
+
+---
+
+## Performance Considerations
+
+- **JobTracker**: Minimal overhead, ~10ms per KV operation
+- **PartialBatchRecovery**: Adds ~5-10% overhead for tracking
+- **BatchArchivalService**: Async S3 uploads, doesn't block main workflow
+
+---
+
+## See Also
+
+- [Ingestion Guide](advanced-services-readme.md) - Core ingestion workflows
+- [State Management](../ingestion/modules/02-core-guides-ingestion-07-state-management.md) - Basic state tracking
+- [Monitoring](advanced-services-readme.md) - Processing statistics
+- [Error Handling](../api-reference/modules/api-reference-09-error-handling.md) - Error strategies
+
+---
+
+## Quick Reference
+
+See [Advanced Services Quick Reference](./advanced-services-quick-reference.md) for a one-page cheatsheet.