@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md

@@ -1,330 +1,330 @@
-# Standalone: S3 CSV Inventory → Fluent Batch API
-
-**FC Connect SDK Use Case Guide**
-
-> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
-
-**Context**: Node.js script that reads CSV inventory files from S3, transforms with UniversalMapper, and updates Fluent Commerce using Batch API.
-
-**Complexity**: Medium
-
-**Runtime**: Node.js ≥18 / Deno
-
-**Estimated Lines**: ~450
-
-## What You'll Build
-
-- S3 file listing, download, and archival
-- CSV parsing (streaming or buffered)
-- Field mapping with UniversalMapper and SDK resolvers
-- Batch API job create/send with chunking
-- KV-based dedupe and checkpoints (optional)
-
-## Environment (.env)
-
-```bash
-# Fluent Commerce OAuth2
-FLUENT_BASE_URL=https://api.fluentcommerce.com
-FLUENT_CLIENT_ID=your-client-id
-FLUENT_CLIENT_SECRET=your-client-secret
-FLUENT_RETAILER_ID=your-retailer-id
-
-# Source S3 (read)
-AWS_ACCESS_KEY_ID=xxx
-AWS_SECRET_ACCESS_KEY=yyy
-AWS_REGION=us-east-1
-SOURCE_BUCKET=inventory-feed
-SOURCE_PREFIX=incoming/
-ARCHIVE_PREFIX=archive/
-ERROR_PREFIX=errors/
-
-# Processing
-BATCH_SIZE=2000
-CSV_DELIMITER=,
-RECORDS_PER_FILE_CAP=250000
-```
-
-## SDK Methods Used
-
-```typescript
-// FC Connect SDK+
-// Install: npm install @fluentcommerce/fc-connect-sdk@latest
-// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
-// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
-
-import { createClient, S3DataSource, CSVParserService, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-
-await createClient({ config: { baseUrl, clientId, clientSecret, retailerId } });
-const s3 = new S3DataSource({ /* creds */ }, logger);
-const parser = new CSVParserService();
-const mapper = new UniversalMapper(mapping);
-
-await client.createJob(...);
-await client.sendBatch(jobId, { entities });
-```
-
-### Method selection (SDK)
-
-- Use `client.graphql` for both queries and mutations.
-- Drop to `client.graphql` with pagination options for pagination control or when you need extensions.
-
-```typescript
-const page = await client.graphql({ query, variables });
-const saved = await client.graphql({ query: mutation, variables });
-const full = await client.graphql({ query, variables, pagination: { maxPages: 100 } });
-```
-
-## Project Setup
-
-```bash
-# Create project
-mkdir s3-csv-inventory-batch && cd $_
-
-# Initialize package
-npm init -y
-
-# Install dependencies
-npm install @fluentcommerce/fc-connect-sdk dotenv
-
-# Create structure
-mkdir -p src mapping logs
-```
-
-### Package Configuration (package.json)
-
-```json
-{
-  "name": "s3-csv-inventory-batch",
-  "version": "1.0.0",
-  "description": "Standalone S3 CSV → Fluent Batch API inventory sync",
-  "type": "module",
-  "scripts": {
-    "dev": "tsx src/s3-csv-inventory.ts",
-    "build": "tsc",
-    "start": "node dist/s3-csv-inventory.js"
-  },
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "dotenv": "^16.0.0"
-  },
-  "devDependencies": {
-    "tsx": "^4.0.0",
-    "typescript": "^5.0.0",
-    "@types/node": "^20.0.0"
-  }
-}
-```
-
-### TypeScript Configuration (tsconfig.json)
-
-```json
-{
-  "compilerOptions": {
-    "module": "ES2022",
-    "target": "ES2024",
-    "moduleResolution": "node"
-  }
-}
-```
-
-## Mapping Configuration (mapping/inventory.json)
-
-```json
-{
-  "fields": {
-    "ref": { "source": "sku", "required": true },
-    "locationRef": { "source": "location", "required": true },
-    "qty": { "source": "quantity", "resolver": "sdk.parseInt", "required": true },
-    "status": { "source": "status", "resolver": "sdk.uppercase", "defaultValue": "ACTIVE" }
-  }
-}
-```
-
-## Complete Script (src/s3-csv-inventory.ts)
-
-```typescript
-import 'dotenv/config';
-import {
-  createClient,
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-import * as fs from 'node:fs';
-import * as path from 'node:path';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-const config = {
-  fluent: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-  s3: {
-    bucket: process.env.SOURCE_BUCKET!,
-    region: process.env.AWS_REGION!,
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    prefix: process.env.SOURCE_PREFIX || 'incoming/',
-    archivePrefix: process.env.ARCHIVE_PREFIX || 'archive/',
-    errorPrefix: process.env.ERROR_PREFIX || 'errors/',
-  },
-  processing: {
-    batchSize: parseInt(process.env.BATCH_SIZE || '2000', 10),
-    recordsPerFileCap: parseInt(process.env.RECORDS_PER_FILE_CAP || '250000', 10),
-    delimiter: process.env.CSV_DELIMITER || ',',
-  },
-};
-
-async function main() {
-  const client = await createClient({ config: config.fluent });
-
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      connectionId: 'source-s3',
-      name: 'Source S3',
-      s3Config: {
-        region: config.s3.region,
-        credentials: {
-          accessKeyId: config.s3.accessKeyId,
-          secretAccessKey: config.s3.secretAccessKey,
-        },
-        bucket: config.s3.bucket,
-      },
-    },
-    logger
-  );
-
-  const parser = new CSVParserService();
-  const mapping = JSON.parse(
-    fs.readFileSync(path.join(process.cwd(), 'mapping/inventory.json'), 'utf-8')
-  );
-  const mapper = new UniversalMapper(mapping);
-
-  const files = await s3.listFiles({ prefix: config.s3.prefix });
-  logger.info(`Found ${files.length} files under ${config.s3.prefix}`);
-
-  if (files.length === 0) return;
-
-  // Create one job per run (or per file if preferred)
-  const job = await (client as any).createJob({
-    name: `inventory-update-${Date.now()}`,
-    retailerId: config.fluent.retailerId,
-  });
-
-  for (const file of files) {
-    try {
-      logger.info(`Processing: ${file.path}`);
-
-      const content = (await s3.downloadFile(file.path, { encoding: 'utf8' })) as string;
-      const records = await parser.parse(content, { delimiter: config.processing.delimiter });
-
-      // Cap size per file to avoid oversized batches (optional)
-      const limited = records.slice(0, config.processing.recordsPerFileCap);
-
-      // Map and chunk
-      const entities: any[] = [];
-      for (const rec of limited) {
-        const result = await mapper.map(rec);
-        if (!result.success) {
-          logger.warn('Mapping error', { errors: result.errors, rec });
-          continue;
-        }
-        entities.push({
-          ref: result.data.ref,
-          type: 'ADJUSTMENT',
-          qty: result.data.qty,
-          locationRef: result.data.locationRef,
-          status: result.data.status,
-        });
-      }
-
-      // Send in batches
-      const size = config.processing.batchSize;
-      for (let i = 0; i < entities.length; i += size) {
-        const chunk = entities.slice(i, i + size);
-        await (client as any).sendBatch(job.id, { entities: chunk });
-        logger.info(
-          `Sent batch ${Math.floor(i / size) + 1} of ${Math.ceil(entities.length / size)}`
-        );
-      }
-
-      // Archive
-      const archivePath = `${config.s3.archivePrefix}${path.basename(file.path)}`;
-      await s3.moveFile(file.path, archivePath, true);
-      logger.info(`Archived: ${archivePath}`);
-    } catch (err: any) {
-      logger.error('Failed processing file', { file: file.path, error: err?.message });
-      const errorPath = `${config.s3.errorPrefix}${path.basename(file.path)}`;
-      try {
-        await s3.moveFile(file.path, errorPath, true);
-      } catch {}
-    }
-  }
-}
-
-main().catch(e => {
-  logger.error('Fatal error', { error: e?.message });
-  process.exit(1);
-});
-```
-
-## Testing
-
-```bash
-# 1) Prepare .env with proper credentials
-
-# 2) Create sample mapping/inventory.json (above)
-
-# 3) Upload a test CSV
-aws s3 cp test.csv s3://$SOURCE_BUCKET/$SOURCE_PREFIX
-
-# 4) Run
-npm run dev
-
-# 5) Verify in Fluent: job status and inventory
-```
-
-## Deployment Options
-
-- Local: run via Node or tsx
-- Docker: add Dockerfile; copy dist + node_modules; run with .env
-- Windows Task Scheduler / cron: wrap `npm run start` on a cadence
-- Lambda: bundle with esbuild, configure env and IAM; ensure memory for CSV
-
-## Common Issues & Solutions
-
-- S3 permission denied
-  - Verify IAM policy includes List/Get/Put/Delete for bucket and prefix
-- Large CSV memory use
-  - Switch to streaming parse and send batches as you go
-- Batch API timeouts
-  - Reduce BATCH_SIZE and add small delays between batches
-- Mapping failures for valid rows
-  - Trim columns and ensure headers align with mapping sources
-
-## Production Checklist
-
-- Secrets in secure store; no plaintext in repo
-- IAM least-privilege for S3 operations
-- Robust logging and error alerts
-- Backoff and retry for transient network issues
-- Batch sizing tuned with production samples
-- Archival paths date-partitioned and monitored
-- Idempotency plan (e.g., job naming, record keys)
-- Runbooks for failure handling
-
-## Related Guides
-
-- Error handling & retry: `docs/use-cases/patterns/error-handling-retry.md`
-- Universal mapping: `../../02-CORE-GUIDES/mapping/readme.md`
-- Batch API patterns: `../../02-CORE-GUIDES/ingestion/readme.md`
-- S3 operations: `../../02-CORE-GUIDES/data-sources/readme.md`
+# Standalone: S3 CSV Inventory → Fluent Batch API
+
+**FC Connect SDK Use Case Guide**
+
+> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
+
+**Context**: Node.js script that reads CSV inventory files from S3, transforms with UniversalMapper, and updates Fluent Commerce using Batch API.
+
+**Complexity**: Medium
+
+**Runtime**: Node.js ≥18 / Deno
+
+**Estimated Lines**: ~450
+
+## What You'll Build
+
+- S3 file listing, download, and archival
+- CSV parsing (streaming or buffered)
+- Field mapping with UniversalMapper and SDK resolvers
+- Batch API job create/send with chunking
+- KV-based dedupe and checkpoints (optional)
+
+## Environment (.env)
+
+```bash
+# Fluent Commerce OAuth2
+FLUENT_BASE_URL=https://api.fluentcommerce.com
+FLUENT_CLIENT_ID=your-client-id
+FLUENT_CLIENT_SECRET=your-client-secret
+FLUENT_RETAILER_ID=your-retailer-id
+
+# Source S3 (read)
+AWS_ACCESS_KEY_ID=xxx
+AWS_SECRET_ACCESS_KEY=yyy
+AWS_REGION=us-east-1
+SOURCE_BUCKET=inventory-feed
+SOURCE_PREFIX=incoming/
+ARCHIVE_PREFIX=archive/
+ERROR_PREFIX=errors/
+
+# Processing
+BATCH_SIZE=2000
+CSV_DELIMITER=,
+RECORDS_PER_FILE_CAP=250000
+```
+
+## SDK Methods Used
+
+```typescript
+// FC Connect SDK+
+// Install: npm install @fluentcommerce/fc-connect-sdk@latest
+// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
+// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
+
+import { createClient, S3DataSource, CSVParserService, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+
+await createClient({ config: { baseUrl, clientId, clientSecret, retailerId } });
+const s3 = new S3DataSource({ /* creds */ }, logger);
+const parser = new CSVParserService();
+const mapper = new UniversalMapper(mapping);
+
+await client.createJob(...);
+await client.sendBatch(jobId, { entities });
+```
+
+### Method selection (SDK)
+
+- Use `client.graphql` for both queries and mutations.
+- Drop to `client.graphql` with pagination options for pagination control or when you need extensions.
+
+```typescript
+const page = await client.graphql({ query, variables });
+const saved = await client.graphql({ query: mutation, variables });
+const full = await client.graphql({ query, variables, pagination: { maxPages: 100 } });
+```
+
+## Project Setup
+
+```bash
+# Create project
+mkdir s3-csv-inventory-batch && cd $_
+
+# Initialize package
+npm init -y
+
+# Install dependencies
+npm install @fluentcommerce/fc-connect-sdk dotenv
+
+# Create structure
+mkdir -p src mapping logs
+```
+
+### Package Configuration (package.json)
+
+```json
+{
+  "name": "s3-csv-inventory-batch",
+  "version": "1.0.0",
+  "description": "Standalone S3 CSV → Fluent Batch API inventory sync",
+  "type": "module",
+  "scripts": {
+    "dev": "tsx src/s3-csv-inventory.ts",
+    "build": "tsc",
+    "start": "node dist/s3-csv-inventory.js"
+  },
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "dotenv": "^16.0.0"
+  },
+  "devDependencies": {
+    "tsx": "^4.0.0",
+    "typescript": "^5.0.0",
+    "@types/node": "^20.0.0"
+  }
+}
+```
+
+### TypeScript Configuration (tsconfig.json)
+
+```json
+{
+  "compilerOptions": {
+    "module": "ES2022",
+    "target": "ES2024",
+    "moduleResolution": "node"
+  }
+}
+```
+
+## Mapping Configuration (mapping/inventory.json)
+
+```json
+{
+  "fields": {
+    "ref": { "source": "sku", "required": true },
+    "locationRef": { "source": "location", "required": true },
+    "qty": { "source": "quantity", "resolver": "sdk.parseInt", "required": true },
+    "status": { "source": "status", "resolver": "sdk.uppercase", "defaultValue": "ACTIVE" }
+  }
+}
+```
+
+## Complete Script (src/s3-csv-inventory.ts)
+
+```typescript
+import 'dotenv/config';
+import {
+  createClient,
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+import * as fs from 'node:fs';
+import * as path from 'node:path';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+const config = {
+  fluent: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+  s3: {
+    bucket: process.env.SOURCE_BUCKET!,
+    region: process.env.AWS_REGION!,
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    prefix: process.env.SOURCE_PREFIX || 'incoming/',
+    archivePrefix: process.env.ARCHIVE_PREFIX || 'archive/',
+    errorPrefix: process.env.ERROR_PREFIX || 'errors/',
+  },
+  processing: {
+    batchSize: parseInt(process.env.BATCH_SIZE || '2000', 10),
+    recordsPerFileCap: parseInt(process.env.RECORDS_PER_FILE_CAP || '250000', 10),
+    delimiter: process.env.CSV_DELIMITER || ',',
+  },
+};
+
+async function main() {
+  const client = await createClient({ config: config.fluent });
+
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      connectionId: 'source-s3',
+      name: 'Source S3',
+      s3Config: {
+        region: config.s3.region,
+        credentials: {
+          accessKeyId: config.s3.accessKeyId,
+          secretAccessKey: config.s3.secretAccessKey,
+        },
+        bucket: config.s3.bucket,
+      },
+    },
+    logger
+  );
+
+  const parser = new CSVParserService();
+  const mapping = JSON.parse(
+    fs.readFileSync(path.join(process.cwd(), 'mapping/inventory.json'), 'utf-8')
+  );
+  const mapper = new UniversalMapper(mapping);
+
+  const files = await s3.listFiles({ prefix: config.s3.prefix });
+  logger.info(`Found ${files.length} files under ${config.s3.prefix}`);
+
+  if (files.length === 0) return;
+
+  // Create one job per run (or per file if preferred)
+  const job = await (client as any).createJob({
+    name: `inventory-update-${Date.now()}`,
+    retailerId: config.fluent.retailerId,
+  });
+
+  for (const file of files) {
+    try {
+      logger.info(`Processing: ${file.path}`);
+
+      const content = (await s3.downloadFile(file.path, { encoding: 'utf8' })) as string;
+      const records = await parser.parse(content, { delimiter: config.processing.delimiter });
+
+      // Cap size per file to avoid oversized batches (optional)
+      const limited = records.slice(0, config.processing.recordsPerFileCap);
+
+      // Map and chunk
+      const entities: any[] = [];
+      for (const rec of limited) {
+        const result = await mapper.map(rec);
+        if (!result.success) {
+          logger.warn('Mapping error', { errors: result.errors, rec });
+          continue;
+        }
+        entities.push({
+          ref: result.data.ref,
+          type: 'ADJUSTMENT',
+          qty: result.data.qty,
+          locationRef: result.data.locationRef,
+          status: result.data.status,
+        });
+      }
+
+      // Send in batches
+      const size = config.processing.batchSize;
+      for (let i = 0; i < entities.length; i += size) {
+        const chunk = entities.slice(i, i + size);
+        await (client as any).sendBatch(job.id, { entities: chunk });
+        logger.info(
+          `Sent batch ${Math.floor(i / size) + 1} of ${Math.ceil(entities.length / size)}`
+        );
+      }
+
+      // Archive
+      const archivePath = `${config.s3.archivePrefix}${path.basename(file.path)}`;
+      await s3.moveFile(file.path, archivePath, true);
+      logger.info(`Archived: ${archivePath}`);
+    } catch (err: any) {
+      logger.error('Failed processing file', { file: file.path, error: err?.message });
+      const errorPath = `${config.s3.errorPrefix}${path.basename(file.path)}`;
+      try {
+        await s3.moveFile(file.path, errorPath, true);
+      } catch {}
+    }
+  }
+}
+
+main().catch(e => {
+  logger.error('Fatal error', { error: e?.message });
+  process.exit(1);
+});
+```
+
+## Testing
+
+```bash
+# 1) Prepare .env with proper credentials
+
+# 2) Create sample mapping/inventory.json (above)
+
+# 3) Upload a test CSV
+aws s3 cp test.csv s3://$SOURCE_BUCKET/$SOURCE_PREFIX
+
+# 4) Run
+npm run dev
+
+# 5) Verify in Fluent: job status and inventory
+```
+
+## Deployment Options
+
+- Local: run via Node or tsx
+- Docker: add Dockerfile; copy dist + node_modules; run with .env
+- Windows Task Scheduler / cron: wrap `npm run start` on a cadence
+- Lambda: bundle with esbuild, configure env and IAM; ensure memory for CSV
+
+## Common Issues & Solutions
+
+- S3 permission denied
+  - Verify IAM policy includes List/Get/Put/Delete for bucket and prefix
+- Large CSV memory use
+  - Switch to streaming parse and send batches as you go
+- Batch API timeouts
+  - Reduce BATCH_SIZE and add small delays between batches
+- Mapping failures for valid rows
+  - Trim columns and ensure headers align with mapping sources
+
+## Production Checklist
+
+- Secrets in secure store; no plaintext in repo
+- IAM least-privilege for S3 operations
+- Robust logging and error alerts
+- Backoff and retry for transient network issues
+- Batch sizing tuned with production samples
+- Archival paths date-partitioned and monitored
+- Idempotency plan (e.g., job naming, record keys)
+- Runbooks for failure handling
+
+## Related Guides
+
+- Error handling & retry: `docs/use-cases/patterns/error-handling-retry.md`
+- Universal mapping: `../../02-CORE-GUIDES/mapping/readme.md`
+- Batch API patterns: `../../02-CORE-GUIDES/ingestion/readme.md`
+- S3 operations: `../../02-CORE-GUIDES/data-sources/readme.md`