npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.54 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (475) hide show

package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md CHANGED Viewed

@@ -1,724 +1,724 @@
-# BatchArchivalService
-**Level:** Advanced
-**Category:** Compliance & Audit
-**Purpose:** Archive batch payloads to S3 for audit trails and compliance
-## Overview
-`BatchArchivalService` provides automatic archival of batch payloads to S3 with hierarchical time-based paths. It creates audit trails for all batch operations, enabling compliance tracking, debugging, and historical analysis.
-**Key Features:**
-- Hierarchical time-based S3 paths (`YYYY/MM/DD/HH/MM`)
-- Presigned URL uploads (no direct S3 SDK dependencies)
-- Bulk archival with parallel uploads
-- Automatic metadata enrichment
-- Failed upload handling
-## Use Cases
-- ✅ Regulatory compliance (SOX, GDPR, HIPAA)
-- ✅ Audit trail requirements
-- ✅ Debug production batch failures
-- ✅ Historical data analysis
-- ✅ Replay failed batches
-- ✅ Track data lineage
----
-## Installation
-```typescript
-import {
-  BatchArchivalService,
-  S3PresignService
-} from '@fluentcommerce/fc-connect-sdk';
-```
----
-## Basic Usage
-### Archive Single Batch
-```typescript
-import { BatchArchivalService, S3PresignService } from '@fluentcommerce/fc-connect-sdk';
-// Configure archival service
-const config = {
-  bucketName: 'my-batch-archives',
-  region: 'us-east-1',
-  credentials: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
-  },
-  basePath: 'fluent_batches', // Optional: defaults to 'fluent_batches'
-  expiresSeconds: 300 // Optional: presigned URL expiry (default: 300s)
-};
-// Initialize services
-const s3Service = new S3PresignService();
-const archivalService = new BatchArchivalService(config, s3Service, logger);
-// Archive a batch
-const result = await archivalService.archiveBatch(
-  {
-    batch: {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      entities: [{ skuRef: 'SKU001', qty: 100 }]
-    },
-    batchResponse: {
-      id: 'batch-123',
-      jobId: 'job-456',
-      status: 'COMPLETED'
-    },
-    jobId: 'job-456'
-  },
-  'client-abc' // Optional client ID for tracking
-);
-if (result.success) {
-  console.log('Archived to:', result.archivedPath);
-  // Output: fluent_batches/2025/01/30/14/30/batch-job-456-batch-123.json
-} else {
-  console.error('Archival failed:', result.error);
-}
-```
----
-## API Reference
-### Constructor
-```typescript
-new BatchArchivalService(
-  config: BatchArchivalConfig,
-  s3PresignService: S3PresignService,
-  logger?: Logger
-)
-```
-**Parameters:**
-- `config` - Archival configuration
-  - `bucketName` - S3 bucket name
-  - `region` - AWS region
-  - `credentials` - AWS credentials
-  - `basePath` - Optional base path (default: 'fluent_batches')
-  - `expiresSeconds` - Optional presigned URL TTL (default: 300)
-- `s3PresignService` - S3 presign service instance
-- `logger` - Optional structured logger
-**Example:**
-```typescript
-const archivalService = new BatchArchivalService(
-  {
-    bucketName: 'production-archives',
-    region: 'us-west-2',
-    credentials: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
-    },
-    basePath: 'compliance/batches',
-    expiresSeconds: 600
-  },
-  s3PresignService,
-  logger
-);
-```
----
-### archiveBatch()
-Archive a single batch to S3.
-```typescript
-await archivalService.archiveBatch(
-  batchItem: BatchToArchive,
-  clientId?: string
-): Promise<ArchivalResult>
-```
-**Parameters:**
-- `batchItem` - Batch data to archive
-  - `batch` - Original batch payload
-  - `batchResponse` - Response from Batch API
-  - `jobId` - Job identifier
-- `clientId` - Optional client identifier for multi-tenant tracking
-**Returns:** `ArchivalResult`
-```typescript
-{
-  success: boolean;
-  archivedPath?: string;  // S3 key if successful
-  error?: string;          // Error message if failed
-}
-```
-**Example:**
-```typescript
-const result = await archivalService.archiveBatch(
-  {
-    batch: {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      catalogueRef: 'DEFAULT:1',
-      entities: inventoryRecords
-    },
-    batchResponse: {
-      id: batchResponse.id,
-      jobId: batchResponse.jobId,
-      status: batchResponse.status
-    },
-    jobId: 'inventory-sync-2025-01-30'
-  },
-  'tenant-123'
-);
-```
----
-### archiveBatches()
-Archive multiple batches in parallel.
-```typescript
-await archivalService.archiveBatches(
-  batches: BatchToArchive[],
-  clientId?: string
-): Promise<ArchivalResult[]>
-```
-**Parameters:**
-- `batches` - Array of batches to archive
-- `clientId` - Optional client identifier
-**Returns:** Array of `ArchivalResult` (one per batch)
-**Example:**
-```typescript
-const batches = [
-  { batch: batch1, batchResponse: response1, jobId: 'job-1' },
-  { batch: batch2, batchResponse: response2, jobId: 'job-2' },
-  { batch: batch3, batchResponse: response3, jobId: 'job-3' }
-];
-const results = await archivalService.archiveBatches(batches, 'tenant-123');
-const successCount = results.filter(r => r.success).length;
-console.log(`Archived ${successCount} of ${results.length} batches`);
-```
----
-### generateArchivalPath()
-Generate S3 path without uploading (useful for pre-computation).
-```typescript
-const path = archivalService.generateArchivalPath(
-  jobId: string,
-  batchId?: string,
-  timestamp?: Date
-): string
-```
-**Parameters:**
-- `jobId` - Job identifier
-- `batchId` - Optional batch identifier
-- `timestamp` - Optional timestamp (defaults to now)
-**Returns:** S3 key string
-**Example:**
-```typescript
-const path = archivalService.generateArchivalPath(
-  'inventory-sync-2025-01-30',
-  'batch-123',
-  new Date('2025-01-30T14:30:00Z')
-);
-console.log(path);
-// Output: fluent_batches/2025/01/30/14/30/batch-inventory-sync-2025-01-30-batch-123.json
-```
----
-### listArchivalsByDate()
-List all archived batches for a specific date.
-```typescript
-const keys = await archivalService.listArchivalsByDate(
-  date: Date
-): Promise<string[]>
-```
-**Parameters:**
-- `date` - Date to query
-**Returns:** Array of S3 keys
-**Example:**
-```typescript
-const today = new Date('2025-01-30');
-const archivedBatches = await archivalService.listArchivalsByDate(today);
-console.log(`Found ${archivedBatches.length} archived batches for today:`);
-archivedBatches.forEach(key => console.log(`  - ${key}`));
-```
----
-## S3 Path Structure
-BatchArchivalService creates hierarchical paths for efficient querying:
-```
-{basePath}/{YYYY}/{MM}/{DD}/{HH}/{MM}/batch-{jobId}-{batchId}.json
-```
-**Example Paths:**
-```
-fluent_batches/2025/01/30/14/30/batch-inventory-sync-batch-123.json
-fluent_batches/2025/01/30/14/31/batch-inventory-sync-batch-124.json
-compliance/batches/2025/01/30/15/00/batch-order-sync-batch-456.json
-```
-**Benefits:**
-- Easy date-based queries
-- Efficient S3 lifecycle policies
-- Time-series analysis
-- Quick drill-down debugging
----
-## Archived Payload Format
-Each archived file contains:
-```json
-{
-  "action": "UPSERT",
-  "entityType": "INVENTORY",
-  "catalogueRef": "DEFAULT:1",
-  "entities": [
-    { "skuRef": "SKU001", "qty": 100 },
-    { "skuRef": "SKU002", "qty": 200 }
-  ],
-  "clientId": "tenant-123",
-  "archivedAt": "2025-01-30T14:30:00.000Z",
-  "batchMetadata": {
-    "jobId": "inventory-sync-2025-01-30",
-    "batchId": "batch-123",
-    "status": "COMPLETED"
-  }
-}
-```
----
-## Common Patterns
-### Pattern 1: Archive After Successful Ingestion
-```typescript
-import {
-  FluentBatchManager,
-  BatchArchivalService,
-  S3PresignService
-} from '@fluentcommerce/fc-connect-sdk';
-async function ingestWithArchival(records: any[]) {
-  const batchManager = new FluentBatchManager(client, logger);
-  const archivalService = new BatchArchivalService(archivalConfig, s3Service, logger);
-  const jobId = `inventory-sync-${Date.now()}`;
-  // Send batch
-  const batchResponse = await client.sendBatch(jobId, {
-    action: 'UPSERT',
-    entityType: 'INVENTORY',
-    entities: records
-  });
-  // Archive immediately after successful send
-  await archivalService.archiveBatch({
-    batch: {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      entities: records
-    },
-    batchResponse,
-    jobId
-  });
-  logger.info('Batch sent and archived', { jobId, batchId: batchResponse.id });
-}
-```
----
-### Pattern 2: Archive All Batches (Bulk)
-```typescript
-async function bulkIngestWithArchival(
-  fileRecords: Record<string, any[]>
-) {
-  const archivalService = new BatchArchivalService(config, s3Service, logger);
-  const batchesToArchive: BatchToArchive[] = [];
-  // Process all files
-  for (const [fileName, records] of Object.entries(fileRecords)) {
-    const jobId = `ingestion-${fileName}-${Date.now()}`;
-    // Send batch
-    const batchResponse = await client.sendBatch(jobId, {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      entities: records
-    });
-    // Queue for archival
-    batchesToArchive.push({
-      batch: {
-        action: 'UPSERT',
-        entityType: 'INVENTORY',
-        entities: records
-      },
-      batchResponse,
-      jobId
-    });
-  }
-  // Archive all batches in parallel
-  const results = await archivalService.archiveBatches(
-    batchesToArchive,
-    'tenant-abc'
-  );
-  const successCount = results.filter(r => r.success).length;
-  logger.info('Bulk archival completed', {
-    total: results.length,
-    successful: successCount,
-    failed: results.length - successCount
-  });
-}
-```
----
-### Pattern 3: Archive Failed Batches Only
-```typescript
-async function archiveFailedBatches(
-  batchResults: BatchResult[]
-) {
-  const archivalService = new BatchArchivalService(config, s3Service, logger);
-  const failedBatches = batchResults
-    .filter(result => result.status === 'FAILED')
-    .map(result => ({
-      batch: result.originalPayload,
-      batchResponse: {
-        id: result.batchId,
-        jobId: result.jobId,
-        status: result.status
-      },
-      jobId: result.jobId
-    }));
-  if (failedBatches.length > 0) {
-    await archivalService.archiveBatches(failedBatches);
-    logger.warn(`Archived ${failedBatches.length} failed batches for investigation`);
-  }
-}
-```
----
-### Pattern 4: S3 Lifecycle Integration
-Configure S3 lifecycle policies to manage archived data:
-```typescript
-// AWS CLI example for setting up lifecycle policy
-const lifecyclePolicy = {
-  Rules: [
-    {
-      Id: 'ArchiveTo Glacier After 90 Days',
-      Status: 'Enabled',
-      Prefix: 'fluent_batches/',
-      Transitions: [
-        {
-          Days: 90,
-          StorageClass: 'GLACIER'
-        }
-      ]
-    },
-    {
-      Id: 'Delete After 7 Years',
-      Status: 'Enabled',
-      Prefix: 'fluent_batches/',
-      Expiration: {
-        Days: 2555 // 7 years
-      }
-    }
-  ]
-};
-// Apply via AWS CLI:
-// aws s3api put-bucket-lifecycle-configuration \
-//   --bucket my-batch-archives \
-//   --lifecycle-configuration file://lifecycle-policy.json
-```
----
-### Pattern 5: Query Archived Batches
-```typescript
-async function queryArchivedBatches(
-  startDate: Date,
-  endDate: Date
-): Promise<string[]> {
-  const archivalService = new BatchArchivalService(config, s3Service, logger);
-  const allKeys: string[] = [];
-  // Iterate through date range
-  const currentDate = new Date(startDate);
-  while (currentDate <= endDate) {
-    const keys = await archivalService.listArchivalsByDate(currentDate);
-    allKeys.push(...keys);
-    // Move to next day
-    currentDate.setDate(currentDate.getDate() + 1);
-  }
-  return allKeys;
-}
-// Usage
-const lastWeek = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
-const today = new Date();
-const archivedBatches = await queryArchivedBatches(lastWeek, today);
-console.log(`Found ${archivedBatches.length} archived batches in the last 7 days`);
-```
----
-## Versori Integration
-### Pattern: Archive in Versori Workflow
-```typescript
-import { Buffer } from 'node:buffer';  // Required for Deno/Versori runtime
-import { http } from '@versori/run';
-import {
-  createClient,
-  BatchArchivalService,
-  S3PresignService
-} from '@fluentcommerce/fc-connect-sdk';
-export const inventoryIngestion = http('inventory-ingestion', {
-  connection: 'fluent_commerce'
-}, async ctx => {
-  const { log } = ctx;
-  // Initialize services
-  const client = await createClient(ctx); // Auto-detects Versori context
-  const s3Service = new S3PresignService();
-  const archivalService = new BatchArchivalService(
-    {
-      bucketName: process.env.ARCHIVE_BUCKET!,
-      region: process.env.AWS_REGION!,
-      credentials: {
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
-      }
-    },
-    s3Service,
-    log
-  );
-  // Process ingestion
-  const jobId = `versori-ingestion-${Date.now()}`;
-  const batchResponse = await client.sendBatch(jobId, {
-    action: 'UPSERT',
-    entityType: 'INVENTORY',
-    entities: inventoryRecords
-  });
-  // Archive
-  const archiveResult = await archivalService.archiveBatch({
-    batch: {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      entities: inventoryRecords
-    },
-    batchResponse,
-    jobId
-  });
-  return {
-    jobId,
-    batchId: batchResponse.id,
-    archived: archiveResult.success,
-    archivePath: archiveResult.archivedPath
-  };
-});
-```
----
-## Error Handling
-`BatchArchivalService` never throws errors - it returns results with error details.
-```typescript
-const result = await archivalService.archiveBatch(batchItem);
-if (!result.success) {
-  logger.error('Archival failed', {
-    error: result.error,
-    jobId: batchItem.jobId
-  });
-  // Categorize error
-  if (result.error?.includes('403') || result.error?.includes('Access Denied')) {
-    // AWS credentials issue
-    logger.error('Check AWS credentials and S3 bucket permissions');
-  } else if (result.error?.includes('404') || result.error?.includes('NoSuchBucket')) {
-    // Bucket doesn't exist
-    logger.error('S3 bucket not found - check bucket name');
-  } else if (result.error?.includes('timeout')) {
-    // Network timeout
-    logger.error('S3 upload timeout - check network connectivity');
-  }
-  // Continue processing even if archival fails
-  // Archival is non-critical for data ingestion
-}
-```
----
-## Patterns
-### 1. Archive After Batch Send
-```typescript
-const batchResponse = await client.sendBatch(jobId, payload);
-await archivalService.archiveBatch({ batch: payload, batchResponse, jobId });
-```
-### 2. Use Client IDs for Multi-Tenant
-```typescript
-await archivalService.archiveBatch(batchItem, 'client-abc');
-```
-### 3. Handle Archival Failures Gracefully
-```typescript
-const batchResponse = await client.sendBatch(jobId, payload);
-const archiveResult = await archivalService.archiveBatch(batchItem);
-if (!archiveResult.success) {
-  logger.warn('Archival failed but ingestion succeeded', {
-    jobId,
-    error: archiveResult.error
-  });
-}
-```
-### 4. Configure S3 Lifecycle Policies
-Set up lifecycle policies for cost optimization:
-- Move to Glacier after 90 days (compliance requirements)
-- Delete after 7 years (retention policies)
----
-## Troubleshooting
-### Archival Fails with "Access Denied"
-**Problem:** S3 upload returns 403 Forbidden
-**Causes:**
-1. Invalid AWS credentials
-2. Missing S3 bucket permissions
-3. Bucket policy restrictions
-**Solution:**
-```typescript
-// 1. Verify credentials
-console.log('AWS Key:', process.env.AWS_ACCESS_KEY_ID?.substring(0, 8) + '...');
-// 2. Check IAM policy - needs these permissions:
-// {
-//   "Version": "2012-10-17",
-//   "Statement": [{
-//     "Effect": "Allow",
-//     "Action": [
-//       "s3:PutObject",
-//       "s3:ListBucket"
-//     ],
-//     "Resource": [
-//       "arn:aws:s3:::my-batch-archives",
-//       "arn:aws:s3:::my-batch-archives/*"
-//     ]
-//   }]
-// }
-// 3. Test S3 access directly
-const testResult = await s3Service.presignPutObject({
-  bucket: config.bucketName,
-  region: config.region,
-  credentials: config.credentials,
-  key: 'test.txt',
-  contentType: 'text/plain',
-  expiresSeconds: 300
-});
-console.log('Presigned URL generated:', testResult.url);
-```
-### Archival Path Not Found When Querying
-**Problem:** `listArchivalsByDate()` returns empty array
-**Causes:**
-1. Wrong date/timezone
-2. Incorrect base path
-3. Batches not yet archived
-**Solution:**
-```typescript
-// Use UTC dates consistently
-const today = new Date();
-today.setUTCHours(0, 0, 0, 0); // Start of day UTC
-const keys = await archivalService.listArchivalsByDate(today);
-// Debug the prefix being searched
-const prefix = `${config.basePath}/2025/01/30/`;
-logger.debug('Searching S3 prefix:', prefix);
-```
----
-## See Also
-- [FluentBatchManager](../ingestion/modules/02-core-guides-ingestion-06-batch-api.md) - Send batches to Fluent
-- [JobTracker](./advanced-services-job-tracker.md) - Track job lifecycle
-- [PartialBatchRecovery](./advanced-services-partial-batch-recovery.md) - Handle partial failures
-- [S3PresignService](../data-sources/modules/data-sources-02-s3-operations.md) - S3 presigned URLs
-- [Advanced Services Overview](./advanced-services-readme.md) - All advanced services
----
+# BatchArchivalService
+**Level:** Advanced
+**Category:** Compliance & Audit
+**Purpose:** Archive batch payloads to S3 for audit trails and compliance
+## Overview
+`BatchArchivalService` provides automatic archival of batch payloads to S3 with hierarchical time-based paths. It creates audit trails for all batch operations, enabling compliance tracking, debugging, and historical analysis.
+**Key Features:**
+- Hierarchical time-based S3 paths (`YYYY/MM/DD/HH/MM`)
+- Presigned URL uploads (no direct S3 SDK dependencies)
+- Bulk archival with parallel uploads
+- Automatic metadata enrichment
+- Failed upload handling
+## Use Cases
+- ✅ Regulatory compliance (SOX, GDPR, HIPAA)
+- ✅ Audit trail requirements
+- ✅ Debug production batch failures
+- ✅ Historical data analysis
+- ✅ Replay failed batches
+- ✅ Track data lineage
+---
+## Installation
+```typescript
+import {
+  BatchArchivalService,
+  S3PresignService
+} from '@fluentcommerce/fc-connect-sdk';
+```
+---
+## Basic Usage
+### Archive Single Batch
+```typescript
+import { BatchArchivalService, S3PresignService } from '@fluentcommerce/fc-connect-sdk';
+// Configure archival service
+const config = {
+  bucketName: 'my-batch-archives',
+  region: 'us-east-1',
+  credentials: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
+  },
+  basePath: 'fluent_batches', // Optional: defaults to 'fluent_batches'
+  expiresSeconds: 300 // Optional: presigned URL expiry (default: 300s)
+};
+// Initialize services
+const s3Service = new S3PresignService();
+const archivalService = new BatchArchivalService(config, s3Service, logger);
+// Archive a batch
+const result = await archivalService.archiveBatch(
+  {
+    batch: {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      entities: [{ skuRef: 'SKU001', qty: 100 }]
+    },
+    batchResponse: {
+      id: 'batch-123',
+      jobId: 'job-456',
+      status: 'COMPLETED'
+    },
+    jobId: 'job-456'
+  },
+  'client-abc' // Optional client ID for tracking
+);
+if (result.success) {
+  console.log('Archived to:', result.archivedPath);
+  // Output: fluent_batches/2025/01/30/14/30/batch-job-456-batch-123.json
+} else {
+  console.error('Archival failed:', result.error);
+}
+```
+---
+## API Reference
+### Constructor
+```typescript
+new BatchArchivalService(
+  config: BatchArchivalConfig,
+  s3PresignService: S3PresignService,
+  logger?: Logger
+)
+```
+**Parameters:**
+- `config` - Archival configuration
+  - `bucketName` - S3 bucket name
+  - `region` - AWS region
+  - `credentials` - AWS credentials
+  - `basePath` - Optional base path (default: 'fluent_batches')
+  - `expiresSeconds` - Optional presigned URL TTL (default: 300)
+- `s3PresignService` - S3 presign service instance
+- `logger` - Optional structured logger
+**Example:**
+```typescript
+const archivalService = new BatchArchivalService(
+  {
+    bucketName: 'production-archives',
+    region: 'us-west-2',
+    credentials: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
+    },
+    basePath: 'compliance/batches',
+    expiresSeconds: 600
+  },
+  s3PresignService,
+  logger
+);
+```
+---
+### archiveBatch()
+Archive a single batch to S3.
+```typescript
+await archivalService.archiveBatch(
+  batchItem: BatchToArchive,
+  clientId?: string
+): Promise<ArchivalResult>
+```
+**Parameters:**
+- `batchItem` - Batch data to archive
+  - `batch` - Original batch payload
+  - `batchResponse` - Response from Batch API
+  - `jobId` - Job identifier
+- `clientId` - Optional client identifier for multi-tenant tracking
+**Returns:** `ArchivalResult`
+```typescript
+{
+  success: boolean;
+  archivedPath?: string;  // S3 key if successful
+  error?: string;          // Error message if failed
+}
+```
+**Example:**
+```typescript
+const result = await archivalService.archiveBatch(
+  {
+    batch: {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      catalogueRef: 'DEFAULT:1',
+      entities: inventoryRecords
+    },
+    batchResponse: {
+      id: batchResponse.id,
+      jobId: batchResponse.jobId,
+      status: batchResponse.status
+    },
+    jobId: 'inventory-sync-2025-01-30'
+  },
+  'tenant-123'
+);
+```
+---
+### archiveBatches()
+Archive multiple batches in parallel.
+```typescript
+await archivalService.archiveBatches(
+  batches: BatchToArchive[],
+  clientId?: string
+): Promise<ArchivalResult[]>
+```
+**Parameters:**
+- `batches` - Array of batches to archive
+- `clientId` - Optional client identifier
+**Returns:** Array of `ArchivalResult` (one per batch)
+**Example:**
+```typescript
+const batches = [
+  { batch: batch1, batchResponse: response1, jobId: 'job-1' },
+  { batch: batch2, batchResponse: response2, jobId: 'job-2' },
+  { batch: batch3, batchResponse: response3, jobId: 'job-3' }
+];
+const results = await archivalService.archiveBatches(batches, 'tenant-123');
+const successCount = results.filter(r => r.success).length;
+console.log(`Archived ${successCount} of ${results.length} batches`);
+```
+---
+### generateArchivalPath()
+Generate S3 path without uploading (useful for pre-computation).
+```typescript
+const path = archivalService.generateArchivalPath(
+  jobId: string,
+  batchId?: string,
+  timestamp?: Date
+): string
+```
+**Parameters:**
+- `jobId` - Job identifier
+- `batchId` - Optional batch identifier
+- `timestamp` - Optional timestamp (defaults to now)
+**Returns:** S3 key string
+**Example:**
+```typescript
+const path = archivalService.generateArchivalPath(
+  'inventory-sync-2025-01-30',
+  'batch-123',
+  new Date('2025-01-30T14:30:00Z')
+);
+console.log(path);
+// Output: fluent_batches/2025/01/30/14/30/batch-inventory-sync-2025-01-30-batch-123.json
+```
+---
+### listArchivalsByDate()
+List all archived batches for a specific date.
+```typescript
+const keys = await archivalService.listArchivalsByDate(
+  date: Date
+): Promise<string[]>
+```
+**Parameters:**
+- `date` - Date to query
+**Returns:** Array of S3 keys
+**Example:**
+```typescript
+const today = new Date('2025-01-30');
+const archivedBatches = await archivalService.listArchivalsByDate(today);
+console.log(`Found ${archivedBatches.length} archived batches for today:`);
+archivedBatches.forEach(key => console.log(`  - ${key}`));
+```
+---
+## S3 Path Structure
+BatchArchivalService creates hierarchical paths for efficient querying:
+```
+{basePath}/{YYYY}/{MM}/{DD}/{HH}/{MM}/batch-{jobId}-{batchId}.json
+```
+**Example Paths:**
+```
+fluent_batches/2025/01/30/14/30/batch-inventory-sync-batch-123.json
+fluent_batches/2025/01/30/14/31/batch-inventory-sync-batch-124.json
+compliance/batches/2025/01/30/15/00/batch-order-sync-batch-456.json
+```
+**Benefits:**
+- Easy date-based queries
+- Efficient S3 lifecycle policies
+- Time-series analysis
+- Quick drill-down debugging
+---
+## Archived Payload Format
+Each archived file contains:
+```json
+{
+  "action": "UPSERT",
+  "entityType": "INVENTORY",
+  "catalogueRef": "DEFAULT:1",
+  "entities": [
+    { "skuRef": "SKU001", "qty": 100 },
+    { "skuRef": "SKU002", "qty": 200 }
+  ],
+  "clientId": "tenant-123",
+  "archivedAt": "2025-01-30T14:30:00.000Z",
+  "batchMetadata": {
+    "jobId": "inventory-sync-2025-01-30",
+    "batchId": "batch-123",
+    "status": "COMPLETED"
+  }
+}
+```
+---
+## Common Patterns
+### Pattern 1: Archive After Successful Ingestion
+```typescript
+import {
+  FluentBatchManager,
+  BatchArchivalService,
+  S3PresignService
+} from '@fluentcommerce/fc-connect-sdk';
+async function ingestWithArchival(records: any[]) {
+  const batchManager = new FluentBatchManager(client, logger);
+  const archivalService = new BatchArchivalService(archivalConfig, s3Service, logger);
+  const jobId = `inventory-sync-${Date.now()}`;
+  // Send batch
+  const batchResponse = await client.sendBatch(jobId, {
+    action: 'UPSERT',
+    entityType: 'INVENTORY',
+    entities: records
+  });
+  // Archive immediately after successful send
+  await archivalService.archiveBatch({
+    batch: {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      entities: records
+    },
+    batchResponse,
+    jobId
+  });
+  logger.info('Batch sent and archived', { jobId, batchId: batchResponse.id });
+}
+```
+---
+### Pattern 2: Archive All Batches (Bulk)
+```typescript
+async function bulkIngestWithArchival(
+  fileRecords: Record<string, any[]>
+) {
+  const archivalService = new BatchArchivalService(config, s3Service, logger);
+  const batchesToArchive: BatchToArchive[] = [];
+  // Process all files
+  for (const [fileName, records] of Object.entries(fileRecords)) {
+    const jobId = `ingestion-${fileName}-${Date.now()}`;
+    // Send batch
+    const batchResponse = await client.sendBatch(jobId, {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      entities: records
+    });
+    // Queue for archival
+    batchesToArchive.push({
+      batch: {
+        action: 'UPSERT',
+        entityType: 'INVENTORY',
+        entities: records
+      },
+      batchResponse,
+      jobId
+    });
+  }
+  // Archive all batches in parallel
+  const results = await archivalService.archiveBatches(
+    batchesToArchive,
+    'tenant-abc'
+  );
+  const successCount = results.filter(r => r.success).length;
+  logger.info('Bulk archival completed', {
+    total: results.length,
+    successful: successCount,
+    failed: results.length - successCount
+  });
+}
+```
+---
+### Pattern 3: Archive Failed Batches Only
+```typescript
+async function archiveFailedBatches(
+  batchResults: BatchResult[]
+) {
+  const archivalService = new BatchArchivalService(config, s3Service, logger);
+  const failedBatches = batchResults
+    .filter(result => result.status === 'FAILED')
+    .map(result => ({
+      batch: result.originalPayload,
+      batchResponse: {
+        id: result.batchId,
+        jobId: result.jobId,
+        status: result.status
+      },
+      jobId: result.jobId
+    }));
+  if (failedBatches.length > 0) {
+    await archivalService.archiveBatches(failedBatches);
+    logger.warn(`Archived ${failedBatches.length} failed batches for investigation`);
+  }
+}
+```
+---
+### Pattern 4: S3 Lifecycle Integration
+Configure S3 lifecycle policies to manage archived data:
+```typescript
+// AWS CLI example for setting up lifecycle policy
+const lifecyclePolicy = {
+  Rules: [
+    {
+      Id: 'ArchiveTo Glacier After 90 Days',
+      Status: 'Enabled',
+      Prefix: 'fluent_batches/',
+      Transitions: [
+        {
+          Days: 90,
+          StorageClass: 'GLACIER'
+        }
+      ]
+    },
+    {
+      Id: 'Delete After 7 Years',
+      Status: 'Enabled',
+      Prefix: 'fluent_batches/',
+      Expiration: {
+        Days: 2555 // 7 years
+      }
+    }
+  ]
+};
+// Apply via AWS CLI:
+// aws s3api put-bucket-lifecycle-configuration \
+//   --bucket my-batch-archives \
+//   --lifecycle-configuration file://lifecycle-policy.json
+```
+---
+### Pattern 5: Query Archived Batches
+```typescript
+async function queryArchivedBatches(
+  startDate: Date,
+  endDate: Date
+): Promise<string[]> {
+  const archivalService = new BatchArchivalService(config, s3Service, logger);
+  const allKeys: string[] = [];
+  // Iterate through date range
+  const currentDate = new Date(startDate);
+  while (currentDate <= endDate) {
+    const keys = await archivalService.listArchivalsByDate(currentDate);
+    allKeys.push(...keys);
+    // Move to next day
+    currentDate.setDate(currentDate.getDate() + 1);
+  }
+  return allKeys;
+}
+// Usage
+const lastWeek = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
+const today = new Date();
+const archivedBatches = await queryArchivedBatches(lastWeek, today);
+console.log(`Found ${archivedBatches.length} archived batches in the last 7 days`);
+```
+---
+## Versori Integration
+### Pattern: Archive in Versori Workflow
+```typescript
+import { Buffer } from 'node:buffer';  // Required for Deno/Versori runtime
+import { http } from '@versori/run';
+import {
+  createClient,
+  BatchArchivalService,
+  S3PresignService
+} from '@fluentcommerce/fc-connect-sdk';
+export const inventoryIngestion = http('inventory-ingestion', {
+  connection: 'fluent_commerce'
+}, async ctx => {
+  const { log } = ctx;
+  // Initialize services
+  const client = await createClient(ctx); // Auto-detects Versori context
+  const s3Service = new S3PresignService();
+  const archivalService = new BatchArchivalService(
+    {
+      bucketName: process.env.ARCHIVE_BUCKET!,
+      region: process.env.AWS_REGION!,
+      credentials: {
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
+      }
+    },
+    s3Service,
+    log
+  );
+  // Process ingestion
+  const jobId = `versori-ingestion-${Date.now()}`;
+  const batchResponse = await client.sendBatch(jobId, {
+    action: 'UPSERT',
+    entityType: 'INVENTORY',
+    entities: inventoryRecords
+  });
+  // Archive
+  const archiveResult = await archivalService.archiveBatch({
+    batch: {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      entities: inventoryRecords
+    },
+    batchResponse,
+    jobId
+  });
+  return {
+    jobId,
+    batchId: batchResponse.id,
+    archived: archiveResult.success,
+    archivePath: archiveResult.archivedPath
+  };
+});
+```
+---
+## Error Handling
+`BatchArchivalService` never throws errors - it returns results with error details.
+```typescript
+const result = await archivalService.archiveBatch(batchItem);
+if (!result.success) {
+  logger.error('Archival failed', {
+    error: result.error,
+    jobId: batchItem.jobId
+  });
+  // Categorize error
+  if (result.error?.includes('403') || result.error?.includes('Access Denied')) {
+    // AWS credentials issue
+    logger.error('Check AWS credentials and S3 bucket permissions');
+  } else if (result.error?.includes('404') || result.error?.includes('NoSuchBucket')) {
+    // Bucket doesn't exist
+    logger.error('S3 bucket not found - check bucket name');
+  } else if (result.error?.includes('timeout')) {
+    // Network timeout
+    logger.error('S3 upload timeout - check network connectivity');
+  }
+  // Continue processing even if archival fails
+  // Archival is non-critical for data ingestion
+}
+```
+---
+## Patterns
+### 1. Archive After Batch Send
+```typescript
+const batchResponse = await client.sendBatch(jobId, payload);
+await archivalService.archiveBatch({ batch: payload, batchResponse, jobId });
+```
+### 2. Use Client IDs for Multi-Tenant
+```typescript
+await archivalService.archiveBatch(batchItem, 'client-abc');
+```
+### 3. Handle Archival Failures Gracefully
+```typescript
+const batchResponse = await client.sendBatch(jobId, payload);
+const archiveResult = await archivalService.archiveBatch(batchItem);
+if (!archiveResult.success) {
+  logger.warn('Archival failed but ingestion succeeded', {
+    jobId,
+    error: archiveResult.error
+  });
+}
+```
+### 4. Configure S3 Lifecycle Policies
+Set up lifecycle policies for cost optimization:
+- Move to Glacier after 90 days (compliance requirements)
+- Delete after 7 years (retention policies)
+---
+## Troubleshooting
+### Archival Fails with "Access Denied"
+**Problem:** S3 upload returns 403 Forbidden
+**Causes:**
+1. Invalid AWS credentials
+2. Missing S3 bucket permissions
+3. Bucket policy restrictions
+**Solution:**
+```typescript
+// 1. Verify credentials
+console.log('AWS Key:', process.env.AWS_ACCESS_KEY_ID?.substring(0, 8) + '...');
+// 2. Check IAM policy - needs these permissions:
+// {
+//   "Version": "2012-10-17",
+//   "Statement": [{
+//     "Effect": "Allow",
+//     "Action": [
+//       "s3:PutObject",
+//       "s3:ListBucket"
+//     ],
+//     "Resource": [
+//       "arn:aws:s3:::my-batch-archives",
+//       "arn:aws:s3:::my-batch-archives/*"
+//     ]
+//   }]
+// }
+// 3. Test S3 access directly
+const testResult = await s3Service.presignPutObject({
+  bucket: config.bucketName,
+  region: config.region,
+  credentials: config.credentials,
+  key: 'test.txt',
+  contentType: 'text/plain',
+  expiresSeconds: 300
+});
+console.log('Presigned URL generated:', testResult.url);
+```
+### Archival Path Not Found When Querying
+**Problem:** `listArchivalsByDate()` returns empty array
+**Causes:**
+1. Wrong date/timezone
+2. Incorrect base path
+3. Batches not yet archived
+**Solution:**
+```typescript
+// Use UTC dates consistently
+const today = new Date();
+today.setUTCHours(0, 0, 0, 0); // Start of day UTC
+const keys = await archivalService.listArchivalsByDate(today);
+// Debug the prefix being searched
+const prefix = `${config.basePath}/2025/01/30/`;
+logger.debug('Searching S3 prefix:', prefix);
+```
+---
+## See Also
+- [FluentBatchManager](../ingestion/modules/02-core-guides-ingestion-06-batch-api.md) - Send batches to Fluent
+- [JobTracker](./advanced-services-job-tracker.md) - Track job lifecycle
+- [PartialBatchRecovery](./advanced-services-partial-batch-recovery.md) - Handle partial failures
+- [S3PresignService](../data-sources/modules/data-sources-02-s3-operations.md) - S3 presigned URLs
+- [Advanced Services Overview](./advanced-services-readme.md) - All advanced services
+---