@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md

@@ -1,68 +1,68 @@
-# Pattern Templates
-
-> **Reusable integration patterns for common scenarios**
-> Production-ready templates for CSV validation, error handling, field mapping, and more
-
-## Overview
-
-This directory contains production-ready pattern templates that demonstrate best practices for common integration scenarios. Each pattern is self-contained and can be adapted to your specific needs.
-
-## Available Patterns
-
-| Pattern | Description | Complexity | Use Case |
-|---------|-------------|------------|----------|
-| **[CSV Schema Validation](./csv-schema-validation-and-rejection-report.md)** | Validate CSV data against schema, generate rejection reports | Medium | Data quality enforcement |
-| **[Custom Resolvers](./custom-resolvers.md)** | Build custom transformation functions | Advanced | Complex business logic |
-| **[Error Handling & Retry](./error-handling-retry.md)** | Implement robust error handling with retry logic | Medium | Production reliability |
-| **[Field Mapping Universal](./field-mapping-universal.md)** | Transform data with UniversalMapper | Beginner | Data transformation |
-| **[Large File Splitting](./large-file-splitting.md)** | Split large files into manageable chunks | Advanced | Memory efficiency |
-| **[Pagination & Streaming](./pagination-streaming.md)** | Stream large datasets efficiently | Intermediate | Large data processing |
-| **[State & Duplicate Prevention](./state-duplicate-prevention.md)** | Prevent duplicate processing with state management | Medium | Idempotent workflows |
-| **[Master Data ETL](./master-data-etl.md)** | Complete ETL pipeline for master data | Advanced | Enterprise integration |
-
-## Quick Navigation
-
-### By Use Case
-
-| I Want To... | Pattern | Complexity |
-|--|--|--|
-| **Validate CSV data** | [CSV Schema Validation](./csv-schema-validation-and-rejection-report.md) | Medium |
-| **Transform data fields** | [Field Mapping Universal](./field-mapping-universal.md) | Beginner |
-| **Handle errors gracefully** | [Error Handling & Retry](./error-handling-retry.md) | Medium |
-| **Prevent duplicates** | [State & Duplicate Prevention](./state-duplicate-prevention.md) | Medium |
-| **Process large files** | [Large File Splitting](./large-file-splitting.md) | Advanced |
-| **Stream large datasets** | [Pagination & Streaming](./pagination-streaming.md) | Intermediate |
-| **Custom transformations** | [Custom Resolvers](./custom-resolvers.md) | Advanced |
-| **Build ETL pipeline** | [Master Data ETL](./master-data-etl.md) | Advanced |
-
-## Pattern Structure
-
-Each pattern includes:
-- **Problem Statement** - What problem does this solve?
-- **Solution Overview** - High-level approach
-- **Complete Code** - Production-ready implementation
-- **Configuration** - Required settings
-- **Testing** - How to test the pattern
-- **Troubleshooting** - Common issues and fixes
-
-## Using These Patterns
-
-1. **Choose a pattern** that matches your use case
-2. **Copy the code** to your project
-3. **Customize** for your specific needs
-4. **Test** with sample data
-5. **Deploy** to production
-
-## Related Documentation
-
-- [Versori Patterns](../versori/patterns/) - Versori-specific patterns
-- [Standalone Templates](../standalone/) - Node.js/Deno templates
-- [Versori Workflows](../versori/workflows/) - Scheduled workflows
-
-## Support
-
-Need help implementing these patterns?
-- Review the complete code in each pattern
-- Check troubleshooting sections
-- See related documentation links
-- Contact support@fluentcommerce.com
+# Pattern Templates
+
+> **Reusable integration patterns for common scenarios**
+> Production-ready templates for CSV validation, error handling, field mapping, and more
+
+## Overview
+
+This directory contains production-ready pattern templates that demonstrate best practices for common integration scenarios. Each pattern is self-contained and can be adapted to your specific needs.
+
+## Available Patterns
+
+| Pattern | Description | Complexity | Use Case |
+|---------|-------------|------------|----------|
+| **[CSV Schema Validation](./csv-schema-validation-and-rejection-report.md)** | Validate CSV data against schema, generate rejection reports | Medium | Data quality enforcement |
+| **[Custom Resolvers](./custom-resolvers.md)** | Build custom transformation functions | Advanced | Complex business logic |
+| **[Error Handling & Retry](./error-handling-retry.md)** | Implement robust error handling with retry logic | Medium | Production reliability |
+| **[Field Mapping Universal](./field-mapping-universal.md)** | Transform data with UniversalMapper | Beginner | Data transformation |
+| **[Large File Splitting](./large-file-splitting.md)** | Split large files into manageable chunks | Advanced | Memory efficiency |
+| **[Pagination & Streaming](./pagination-streaming.md)** | Stream large datasets efficiently | Intermediate | Large data processing |
+| **[State & Duplicate Prevention](./state-duplicate-prevention.md)** | Prevent duplicate processing with state management | Medium | Idempotent workflows |
+| **[Master Data ETL](./master-data-etl.md)** | Complete ETL pipeline for master data | Advanced | Enterprise integration |
+
+## Quick Navigation
+
+### By Use Case
+
+| I Want To... | Pattern | Complexity |
+|--|--|--|
+| **Validate CSV data** | [CSV Schema Validation](./csv-schema-validation-and-rejection-report.md) | Medium |
+| **Transform data fields** | [Field Mapping Universal](./field-mapping-universal.md) | Beginner |
+| **Handle errors gracefully** | [Error Handling & Retry](./error-handling-retry.md) | Medium |
+| **Prevent duplicates** | [State & Duplicate Prevention](./state-duplicate-prevention.md) | Medium |
+| **Process large files** | [Large File Splitting](./large-file-splitting.md) | Advanced |
+| **Stream large datasets** | [Pagination & Streaming](./pagination-streaming.md) | Intermediate |
+| **Custom transformations** | [Custom Resolvers](./custom-resolvers.md) | Advanced |
+| **Build ETL pipeline** | [Master Data ETL](./master-data-etl.md) | Advanced |
+
+## Pattern Structure
+
+Each pattern includes:
+- **Problem Statement** - What problem does this solve?
+- **Solution Overview** - High-level approach
+- **Complete Code** - Production-ready implementation
+- **Configuration** - Required settings
+- **Testing** - How to test the pattern
+- **Troubleshooting** - Common issues and fixes
+
+## Using These Patterns
+
+1. **Choose a pattern** that matches your use case
+2. **Copy the code** to your project
+3. **Customize** for your specific needs
+4. **Test** with sample data
+5. **Deploy** to production
+
+## Related Documentation
+
+- [Versori Patterns](../versori/patterns/) - Versori-specific patterns
+- [Standalone Templates](../standalone/) - Node.js/Deno templates
+- [Versori Workflows](../versori/workflows/) - Scheduled workflows
+
+## Support
+
+Need help implementing these patterns?
+- Review the complete code in each pattern
+- Check troubleshooting sections
+- See related documentation links
+- Contact support@fluentcommerce.com

package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md

@@ -1,233 +1,233 @@
-# Pattern: CSV Schema Validation + Rejection Report to S3
-
-**FC Connect SDK Use Case Guide**
-
-> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
-
-**Context**: Validate CSV files (columns and row-level checks) before ingestion. Generate a rejection report CSV for invalid rows and upload to S3 for operators.
-
-**Type**: Reusable Pattern
-
-**Complexity**: Low-Medium
-
-**Lines**: ~220
-
-## When to Use
-
-- Upstream sends variable/messy CSV formats
-- Need deterministic validation and operator feedback
-- Want to avoid partial ingestion or silent skips
-
-## Validation Model
-
-```typescript
-interface ColumnRule {
-  name: string; // CSV header name
-  required?: boolean; // Must exist and be non-empty
-  type?: 'string' | 'number' | 'integer' | 'date' | 'boolean';
-  regex?: string; // Optional format regex
-  min?: number; // For number/integer
-  max?: number; // For number/integer
-  allowed?: string[]; // Enumerated values
-}
-
-interface Schema {
-  columns: ColumnRule[];
-  allowExtraColumns?: boolean;
-}
-```
-
-## Example Schema (inventory.csv)
-
-```json
-{
-  "columns": [
-    { "name": "sku", "required": true },
-    { "name": "location", "required": true },
-    { "name": "quantity", "required": true, "type": "integer", "min": 0 },
-    { "name": "status", "type": "string", "allowed": ["ACTIVE", "INACTIVE"] },
-    { "name": "expected_date", "type": "date" }
-  ],
-  "allowExtraColumns": false
-}
-```
-
-## Implementation Snippet
-
-> **⚠️ RUNTIME COMPATIBILITY NOTE:**
-> This pattern includes `import { Buffer } from 'node:buffer';` for Deno/Versori compatibility.
-> While Buffer is globally available in Node.js, including this import ensures the code works
-> across all runtimes (Node.js, Deno, Versori) without modification.
-
-```typescript
-// FC Connect SDK (latest)
-// Install: npm install @fluentcommerce/fc-connect-sdk@latest
-// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
-// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
-
-import { Buffer } from 'node:buffer'; // Required for Deno/Versori compatibility
-import {
-  CSVParserService,
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function validateCsvAndReport(
-  s3: S3DataSource,
-  inputKey: string,
-  outputPrefix: string,
-  schema: Schema,
-  delimiter = ','
-) {
-  const logging = createConsoleLogger();
-  const logger = logging.createServiceLogger('csv-validation');
-
-  // Download CSV
-  const csvContent = (await s3.downloadFile(inputKey, { encoding: 'utf8' })) as string;
-
-  // Parse
-  const parser = new CSVParserService();
-  const rows = await parser.parse(csvContent, {
-    columns: true,
-    skip_empty_lines: true,
-    trim: true,
-    delimiter,
-  });
-
-  if (rows.length === 0) {
-    return { validCount: 0, invalidCount: 0, reportKey: null };
-  }
-
-  // Header validation
-  const headers = Object.keys(rows[0]);
-  const requiredMissing = schema.columns
-    .filter(c => c.required)
-    .map(c => c.name)
-    .filter(name => !headers.includes(name));
-
-  if (requiredMissing.length > 0) {
-    throw new Error(`Missing required columns: ${requiredMissing.join(', ')}`);
-  }
-
-  if (schema.allowExtraColumns === false) {
-    const allowed = new Set(schema.columns.map(c => c.name));
-    const extra = headers.filter(h => !allowed.has(h));
-    if (extra.length > 0) {
-      throw new Error(`Unexpected columns present: ${extra.join(', ')}`);
-    }
-  }
-
-  // Row validation
-  const invalid: any[] = [];
-  const valid: any[] = [];
-
-  for (let i = 0; i < rows.length; i++) {
-    const r = rows[i];
-    const rowErrors: string[] = [];
-
-    for (const col of schema.columns) {
-      const v = r[col.name];
-
-      if (col.required && (v === undefined || v === null || String(v).trim() === '')) {
-        rowErrors.push(`${col.name}: required`);
-        continue;
-      }
-
-      if (v === undefined || v === null || String(v).trim() === '') continue;
-
-      if (col.type === 'integer') {
-        const n = Number(v);
-        if (!Number.isInteger(n)) rowErrors.push(`${col.name}: not integer`);
-        if (col.min !== undefined && n < col.min) rowErrors.push(`${col.name}: < min`);
-        if (col.max !== undefined && n > col.max) rowErrors.push(`${col.name}: > max`);
-      }
-
-      if (col.type === 'number') {
-        const n = Number(v);
-        if (Number.isNaN(n)) rowErrors.push(`${col.name}: not number`);
-        if (col.min !== undefined && n < col.min) rowErrors.push(`${col.name}: < min`);
-        if (col.max !== undefined && n > col.max) rowErrors.push(`${col.name}: > max`);
-      }
-
-      if (col.type === 'date') {
-        const d = new Date(String(v));
-        if (Number.isNaN(d.getTime())) rowErrors.push(`${col.name}: invalid date`);
-      }
-
-      if (col.type === 'boolean') {
-        const s = String(v).toLowerCase();
-        if (!['true', 'false', '1', '0', 'yes', 'no'].includes(s))
-          rowErrors.push(`${col.name}: invalid boolean`);
-      }
-
-      if (col.regex) {
-        const re = new RegExp(col.regex);
-        if (!re.test(String(v))) rowErrors.push(`${col.name}: regex mismatch`);
-      }
-
-      if (col.allowed && !col.allowed.includes(String(v))) {
-        rowErrors.push(`${col.name}: not in allowed set`);
-      }
-    }
-
-    if (rowErrors.length > 0) invalid.push({ ...r, __errors: rowErrors.join('; ') });
-    else valid.push(r);
-  }
-
-  // Rejection report
-  let reportKey: string | null = null;
-  if (invalid.length > 0) {
-    const header = [...headers, '__errors'];
-    const lines = [header.join(',')];
-
-    for (const row of invalid) {
-      const fields = header.map(h => String(row[h] ?? '').replace(/"/g, '""'));
-      lines.push('"' + fields.join('","') + '"');
-    }
-
-    const reportCsv = lines.join('\n');
-    const ts = new Date().toISOString().replace(/[:.]/g, '-');
-    reportKey = `${outputPrefix}rejections/rejections_${ts}.csv`;
-
-    await s3.uploadFile(reportKey, Buffer.from(reportCsv, 'utf8'), {
-      contentType: 'text/csv; charset=utf-8',
-      metadata: { 'row-count': String(invalid.length) },
-    });
-  }
-
-  logger.info('CSV validation summary', {
-    valid: valid.length,
-    invalid: invalid.length,
-    reportKey,
-  });
-
-  return {
-    validCount: valid.length,
-    invalidCount: invalid.length,
-    reportKey,
-    validRows: valid,
-  };
-}
-```
-
-## Integration Points
-
-- S3 ingestion: run validation before mapping; push only `validRows` to `UniversalMapper`
-- SFTP ingestion: upload rejection report to target S3 bucket for operators
-- Versori response: include `reportKey` in JSON result for quick access
-
-## Best Practices
-
-- Fail fast on missing required columns; do not ingest
-- Keep a minimal, clear error per row (`__errors` field)
-- Version schemas if upstream changes are frequent
-- Store rejection reports under a date-partitioned prefix
-- Alert when invalidCount exceeds threshold
-
-## Related
-
-- CSV Parser options: `../../02-CORE-GUIDES/parsers/modules/02-csv-parser.md`
-- Ingestion guides: S3 and SFTP use-cases
-- Error handling pattern: `docs/use-cases/patterns/error-handling-retry.md`
+# Pattern: CSV Schema Validation + Rejection Report to S3
+
+**FC Connect SDK Use Case Guide**
+
+> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
+
+**Context**: Validate CSV files (columns and row-level checks) before ingestion. Generate a rejection report CSV for invalid rows and upload to S3 for operators.
+
+**Type**: Reusable Pattern
+
+**Complexity**: Low-Medium
+
+**Lines**: ~220
+
+## When to Use
+
+- Upstream sends variable/messy CSV formats
+- Need deterministic validation and operator feedback
+- Want to avoid partial ingestion or silent skips
+
+## Validation Model
+
+```typescript
+interface ColumnRule {
+  name: string; // CSV header name
+  required?: boolean; // Must exist and be non-empty
+  type?: 'string' | 'number' | 'integer' | 'date' | 'boolean';
+  regex?: string; // Optional format regex
+  min?: number; // For number/integer
+  max?: number; // For number/integer
+  allowed?: string[]; // Enumerated values
+}
+
+interface Schema {
+  columns: ColumnRule[];
+  allowExtraColumns?: boolean;
+}
+```
+
+## Example Schema (inventory.csv)
+
+```json
+{
+  "columns": [
+    { "name": "sku", "required": true },
+    { "name": "location", "required": true },
+    { "name": "quantity", "required": true, "type": "integer", "min": 0 },
+    { "name": "status", "type": "string", "allowed": ["ACTIVE", "INACTIVE"] },
+    { "name": "expected_date", "type": "date" }
+  ],
+  "allowExtraColumns": false
+}
+```
+
+## Implementation Snippet
+
+> **⚠️ RUNTIME COMPATIBILITY NOTE:**
+> This pattern includes `import { Buffer } from 'node:buffer';` for Deno/Versori compatibility.
+> While Buffer is globally available in Node.js, including this import ensures the code works
+> across all runtimes (Node.js, Deno, Versori) without modification.
+
+```typescript
+// FC Connect SDK (latest)
+// Install: npm install @fluentcommerce/fc-connect-sdk@latest
+// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
+// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
+
+import { Buffer } from 'node:buffer'; // Required for Deno/Versori compatibility
+import {
+  CSVParserService,
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function validateCsvAndReport(
+  s3: S3DataSource,
+  inputKey: string,
+  outputPrefix: string,
+  schema: Schema,
+  delimiter = ','
+) {
+  const logging = createConsoleLogger();
+  const logger = logging.createServiceLogger('csv-validation');
+
+  // Download CSV
+  const csvContent = (await s3.downloadFile(inputKey, { encoding: 'utf8' })) as string;
+
+  // Parse
+  const parser = new CSVParserService();
+  const rows = await parser.parse(csvContent, {
+    columns: true,
+    skip_empty_lines: true,
+    trim: true,
+    delimiter,
+  });
+
+  if (rows.length === 0) {
+    return { validCount: 0, invalidCount: 0, reportKey: null };
+  }
+
+  // Header validation
+  const headers = Object.keys(rows[0]);
+  const requiredMissing = schema.columns
+    .filter(c => c.required)
+    .map(c => c.name)
+    .filter(name => !headers.includes(name));
+
+  if (requiredMissing.length > 0) {
+    throw new Error(`Missing required columns: ${requiredMissing.join(', ')}`);
+  }
+
+  if (schema.allowExtraColumns === false) {
+    const allowed = new Set(schema.columns.map(c => c.name));
+    const extra = headers.filter(h => !allowed.has(h));
+    if (extra.length > 0) {
+      throw new Error(`Unexpected columns present: ${extra.join(', ')}`);
+    }
+  }
+
+  // Row validation
+  const invalid: any[] = [];
+  const valid: any[] = [];
+
+  for (let i = 0; i < rows.length; i++) {
+    const r = rows[i];
+    const rowErrors: string[] = [];
+
+    for (const col of schema.columns) {
+      const v = r[col.name];
+
+      if (col.required && (v === undefined || v === null || String(v).trim() === '')) {
+        rowErrors.push(`${col.name}: required`);
+        continue;
+      }
+
+      if (v === undefined || v === null || String(v).trim() === '') continue;
+
+      if (col.type === 'integer') {
+        const n = Number(v);
+        if (!Number.isInteger(n)) rowErrors.push(`${col.name}: not integer`);
+        if (col.min !== undefined && n < col.min) rowErrors.push(`${col.name}: < min`);
+        if (col.max !== undefined && n > col.max) rowErrors.push(`${col.name}: > max`);
+      }
+
+      if (col.type === 'number') {
+        const n = Number(v);
+        if (Number.isNaN(n)) rowErrors.push(`${col.name}: not number`);
+        if (col.min !== undefined && n < col.min) rowErrors.push(`${col.name}: < min`);
+        if (col.max !== undefined && n > col.max) rowErrors.push(`${col.name}: > max`);
+      }
+
+      if (col.type === 'date') {
+        const d = new Date(String(v));
+        if (Number.isNaN(d.getTime())) rowErrors.push(`${col.name}: invalid date`);
+      }
+
+      if (col.type === 'boolean') {
+        const s = String(v).toLowerCase();
+        if (!['true', 'false', '1', '0', 'yes', 'no'].includes(s))
+          rowErrors.push(`${col.name}: invalid boolean`);
+      }
+
+      if (col.regex) {
+        const re = new RegExp(col.regex);
+        if (!re.test(String(v))) rowErrors.push(`${col.name}: regex mismatch`);
+      }
+
+      if (col.allowed && !col.allowed.includes(String(v))) {
+        rowErrors.push(`${col.name}: not in allowed set`);
+      }
+    }
+
+    if (rowErrors.length > 0) invalid.push({ ...r, __errors: rowErrors.join('; ') });
+    else valid.push(r);
+  }
+
+  // Rejection report
+  let reportKey: string | null = null;
+  if (invalid.length > 0) {
+    const header = [...headers, '__errors'];
+    const lines = [header.join(',')];
+
+    for (const row of invalid) {
+      const fields = header.map(h => String(row[h] ?? '').replace(/"/g, '""'));
+      lines.push('"' + fields.join('","') + '"');
+    }
+
+    const reportCsv = lines.join('\n');
+    const ts = new Date().toISOString().replace(/[:.]/g, '-');
+    reportKey = `${outputPrefix}rejections/rejections_${ts}.csv`;
+
+    await s3.uploadFile(reportKey, Buffer.from(reportCsv, 'utf8'), {
+      contentType: 'text/csv; charset=utf-8',
+      metadata: { 'row-count': String(invalid.length) },
+    });
+  }
+
+  logger.info('CSV validation summary', {
+    valid: valid.length,
+    invalid: invalid.length,
+    reportKey,
+  });
+
+  return {
+    validCount: valid.length,
+    invalidCount: invalid.length,
+    reportKey,
+    validRows: valid,
+  };
+}
+```
+
+## Integration Points
+
+- S3 ingestion: run validation before mapping; push only `validRows` to `UniversalMapper`
+- SFTP ingestion: upload rejection report to target S3 bucket for operators
+- Versori response: include `reportKey` in JSON result for quick access
+
+## Best Practices
+
+- Fail fast on missing required columns; do not ingest
+- Keep a minimal, clear error per row (`__errors` field)
+- Version schemas if upstream changes are frequent
+- Store rejection reports under a date-partitioned prefix
+- Alert when invalidCount exceeds threshold
+
+## Related
+
+- CSV Parser options: `../../02-CORE-GUIDES/parsers/modules/02-csv-parser.md`
+- Ingestion guides: S3 and SFTP use-cases
+- Error handling pattern: `docs/use-cases/patterns/error-handling-retry.md`