linkedin-secret-sauce 0.12.2 → 0.12.4

package/docs/ENRICHMENT.md

@@ -0,0 +1,580 @@
+# Email Enrichment Guide
+
+Complete guide to using the email enrichment module in LinkedIn Secret Sauce.
+
+## Overview
+
+The enrichment module finds and verifies business emails using multiple providers in an optimized 3-phase strategy. It automatically balances cost, speed, and accuracy.
+
+## Quick Start
+
+```typescript
+import { createEnrichmentClient } from 'linkedin-secret-sauce';
+
+const enrichment = createEnrichmentClient({
+  providers: {
+    // FREE providers (Phase 1)
+    ldd: { apiUrl: process.env.LDD_API_URL, apiToken: process.env.LDD_API_TOKEN },
+    smartprospect: { email: process.env.SMARTLEAD_EMAIL, password: process.env.SMARTLEAD_PASSWORD },
+    trykitt: { apiKey: process.env.TRYKITT_API_KEY },
+    // construct and cosiall are enabled by default (no config needed)
+    
+    // Verification (Phase 2)
+    bounceban: { apiKey: process.env.BOUNCEBAN_API_KEY },
+    
+    // Paid finders (Phase 3 - only if needed)
+    hunter: { apiKey: process.env.HUNTER_API_KEY },
+    snovio: { clientId: process.env.SNOVIO_CLIENT_ID, clientSecret: process.env.SNOVIO_CLIENT_SECRET },
+  },
+});
+
+const result = await enrichment.enrich({
+  firstName: 'John',
+  lastName: 'Doe',
+  company: 'Acme Corp',
+  domain: 'acme.com',
+  linkedinUrl: 'https://linkedin.com/in/johndoe',
+});
+
+console.log(result.business_email);           // john.doe@acme.com
+console.log(result.business_email_source);    // 'trykitt'
+console.log(result.business_email_verified);  // true
+console.log(result.business_email_confidence); // 95
+```
+
+---
+
+## 3-Phase Enrichment Strategy
+
+The default provider order optimizes for cost and accuracy:
+
+```
+PHASE 1 - Free Lookups (Parallel)
+┌─────────────────┬─────────────────┬─────────────────┬─────────────────┐
+│      LDD        │  SmartProspect  │    Cosiall      │    TryKitt      │
+│   (Database)    │   (Database)    │ (LinkedIn API)  │  (AI Finder)    │
+│     FREE        │     FREE*       │     FREE        │     FREE**      │
+└─────────────────┴─────────────────┴─────────────────┴─────────────────┘
+                              ↓
+              If no verified email found (confidence < 80%)
+                              ↓
+PHASE 2 - Pattern + Verification
+┌─────────────────────────────┬─────────────────────────────┐
+│         Construct           │          BounceBan          │
+│   (Pattern Guessing + MX)   │   (Catch-all Verification)  │
+│           FREE              │      FREE single / $0.003   │
+└─────────────────────────────┴─────────────────────────────┘
+                              ↓
+              If still no verified email found
+                              ↓
+PHASE 3 - Paid Finders (Only if needed)
+┌─────────────────────────────┬─────────────────────────────┐
+│           Hunter            │           Snov.io           │
+│      $0.005/email           │       $0.02/email           │
+└─────────────────────────────┴─────────────────────────────┘
+```
+
+*FREE with SmartLead subscription  
+**FREE for individuals with unlimited searches
+
+---
+
+## Supported Providers (11 Total)
+
+### FREE Providers
+
+| Provider | Type | Description | Required Config |
+|----------|------|-------------|-----------------|
+| **Construct** | Pattern | Email pattern guessing + MX verification | None (always enabled) |
+| **Cosiall** | Lookup | Emails from LinkedIn profiles | None (uses global config) |
+| **LDD** | Database | LinkedIn Data Dump (~500M verified emails) | `apiUrl`, `apiToken` |
+| **SmartProspect** | Database | SmartLead's prospect database | `email`, `password` |
+| **TryKitt.ai** | AI Finder | AI-powered email finder with catch-all verification | `apiKey` |
+
+### PAID Providers
+
+| Provider | Cost | Description | Required Config |
+|----------|------|-------------|-----------------|
+| **BounceBan** | FREE single / $0.003 bulk | Catch-all specialist (85-95% accuracy) | `apiKey` |
+| **Hunter** | $0.015/email | Domain search + email finder | `apiKey` |
+| **Snov.io** | $0.02/email | Email finding + verification | `clientId`, `clientSecret` |
+
+---
+
+## Provider Configuration
+
+### LDD (LinkedIn Data Dump)
+
+```typescript
+ldd: {
+  apiUrl: process.env.LDD_API_URL,    // Required: API endpoint
+  apiToken: process.env.LDD_API_TOKEN, // Required: Bearer token
+}
+```
+
+### SmartProspect
+
+```typescript
+smartprospect: {
+  // Option 1: Email/password (recommended)
+  email: process.env.SMARTLEAD_EMAIL,
+  password: process.env.SMARTLEAD_PASSWORD,
+  
+  // Option 2: Direct token
+  apiToken: process.env.SMARTLEAD_TOKEN,
+}
+```
+
+### TryKitt.ai
+
+AI-powered email finding with enterprise identity server verification. FREE for individuals.
+
+```typescript
+trykitt: {
+  apiKey: process.env.TRYKITT_API_KEY, // Required
+  apiUrl: 'https://api.trykitt.ai',     // Optional override
+  timeoutMs: 30000,                     // Optional (default: 30s)
+}
+```
+
+Get your API key at: https://trykitt.ai
+
+### BounceBan
+
+Specializes in catch-all email verification with 85-95% accuracy WITHOUT sending emails.
+
+```typescript
+bounceban: {
+  apiKey: process.env.BOUNCEBAN_API_KEY, // Required
+  useDeepVerify: true,                    // +15-25% accuracy for pattern-guessed emails
+  useWaterfall: true,                     // Recommended: includes 30-min free retry window
+  apiUrl: 'https://api.bounceban.com',    // Optional override
+  waterfallApiUrl: 'https://api-waterfall.bounceban.com', // Optional
+}
+```
+
+Get your API key at: https://bounceban.com
+
+### Hunter
+
+```typescript
+hunter: {
+  apiKey: process.env.HUNTER_API_KEY, // Required
+}
+```
+
+Get your API key at: https://hunter.io
+
+### Snov.io
+
+```typescript
+snovio: {
+  clientId: process.env.SNOVIO_CLIENT_ID,       // Required
+  clientSecret: process.env.SNOVIO_CLIENT_SECRET, // Required
+}
+```
+
+Get your credentials at: https://snov.io
+
+### Cosiall
+
+Enabled by default using global Cosiall configuration. To disable:
+
+```typescript
+cosiall: {
+  enabled: false,
+}
+```
+
+### Construct
+
+Pattern guessing + MX verification. Always enabled, no API key needed.
+
+```typescript
+construct: {
+  maxAttempts: 8,        // Max patterns to try (default: 8)
+  timeoutMs: 5000,       // MX verification timeout (default: 5s)
+  smtpVerifyDelayMs: 2000, // Delay between SMTP checks (default: 2s)
+}
+```
+
+---
+
+## Environment Variables
+
+Complete reference:
+
+```bash
+# LinkedIn API (required for Cosiall)
+COSIALL_API_URL=https://...
+COSIALL_API_KEY=...
+
+# LDD
+LDD_API_URL=https://...
+LDD_API_TOKEN=...
+
+# SmartProspect/SmartLead
+SMARTLEAD_EMAIL=...
+SMARTLEAD_PASSWORD=...
+# OR
+SMARTLEAD_TOKEN=...
+
+# TryKitt.ai (FREE for individuals)
+TRYKITT_API_KEY=...
+
+# BounceBan (FREE single verification)
+BOUNCEBAN_API_KEY=...
+
+# Hunter ($0.015/email)
+HUNTER_API_KEY=...
+
+# Snov.io ($0.02/email)
+SNOVIO_CLIENT_ID=...
+SNOVIO_CLIENT_SECRET=...
+```
+
+---
+
+## API Reference
+
+### `createEnrichmentClient(config)`
+
+Creates an enrichment client instance.
+
+```typescript
+const enrichment = createEnrichmentClient({
+  providers: { /* provider configs */ },
+  options: {
+    providerOrder: ['ldd', 'trykitt', 'construct', 'hunter'], // Custom order
+    maxCostPerEmail: 0.05,    // Max USD to spend per email
+    confidenceThreshold: 70,  // Min confidence to accept (0-100)
+    retryMs: 200,             // Retry delay on transient errors
+  },
+  cache: myRedisCache,        // Optional cache adapter
+  onCost: (provider, cost) => { /* track costs */ },
+  logger: myLogger,           // Optional logger
+});
+```
+
+### `enrichment.enrich(candidate)`
+
+Find a single business email. Returns first verified match.
+
+```typescript
+const result = await enrichment.enrich({
+  firstName: 'John',
+  lastName: 'Doe',
+  company: 'Acme Corp',
+  domain: 'acme.com',          // Optional but improves accuracy
+  linkedinUrl: 'linkedin.com/in/johndoe', // For LDD/Cosiall lookups
+});
+
+// Result
+{
+  business_email: 'john.doe@acme.com',
+  business_email_source: 'trykitt',
+  business_email_verified: true,
+  business_email_confidence: 95,
+  last_checked_at: '2024-01-15T10:30:00.000Z',
+}
+```
+
+### `enrichment.enrichAll(candidate)`
+
+Get ALL emails from ALL providers (doesn't stop at first match).
+
+```typescript
+const result = await enrichment.enrichAll({
+  firstName: 'John',
+  lastName: 'Doe',
+  domain: 'acme.com',
+});
+
+// Result
+{
+  emails: [
+    { email: 'john.doe@acme.com', source: 'trykitt', confidence: 95, verified: true, type: 'business' },
+    { email: 'j.doe@acme.com', source: 'construct', confidence: 70, verified: false, type: 'business' },
+    { email: 'johndoe@gmail.com', source: 'cosiall', confidence: 100, verified: true, type: 'personal' },
+  ],
+  totalCost: 0.005,
+  providersQueried: ['ldd', 'smartprospect', 'cosiall', 'trykitt', 'construct', 'hunter'],
+  completedAt: '2024-01-15T10:30:00.000Z',
+}
+```
+
+### `enrichment.enrichBatch(candidates, options)`
+
+Process multiple candidates efficiently.
+
+```typescript
+const results = await enrichment.enrichBatch(candidates, {
+  batchSize: 50,   // Process 50 at a time
+  delayMs: 200,    // Delay between batches
+});
+```
+
+### `enrichment.enrichAllBatch(candidates, options)`
+
+Batch version of `enrichAll()`.
+
+```typescript
+const results = await enrichment.enrichAllBatch(candidates, {
+  batchSize: 50,
+  delayMs: 200,
+});
+```
+
+---
+
+## Standalone Provider Functions
+
+For advanced use cases, you can call providers directly:
+
+### TryKitt.ai
+
+```typescript
+import { findEmailWithTryKitt, verifyEmailWithTryKitt } from 'linkedin-secret-sauce';
+
+// Find email
+const result = await findEmailWithTryKitt(
+  'John Doe',
+  'acme.com',
+  { apiKey: process.env.TRYKITT_API_KEY },
+  'https://linkedin.com/in/johndoe' // Optional LinkedIn URL
+);
+
+// Verify email
+const verification = await verifyEmailWithTryKitt(
+  'john@acme.com',
+  { apiKey: process.env.TRYKITT_API_KEY }
+);
+```
+
+### BounceBan
+
+```typescript
+import { 
+  verifyEmailWithBounceBan,
+  verifyEmailsBatchWithBounceBan,
+  checkCatchAllWithBounceBan 
+} from 'linkedin-secret-sauce';
+
+// Verify single email (FREE)
+const result = await verifyEmailWithBounceBan('john@acme.com', {
+  apiKey: process.env.BOUNCEBAN_API_KEY,
+  useDeepVerify: true,
+  useWaterfall: true,
+});
+
+// Check catch-all domain
+const isCatchAll = await checkCatchAllWithBounceBan('acme.com', config);
+
+// Batch verification
+const results = await verifyEmailsBatchWithBounceBan(emails, config);
+```
+
+### Snov.io
+
+```typescript
+import { findEmailsWithSnovio, verifyEmailWithSnovio } from 'linkedin-secret-sauce';
+
+const emails = await findEmailsWithSnovio('John', 'Doe', 'acme.com', {
+  clientId: process.env.SNOVIO_CLIENT_ID,
+  clientSecret: process.env.SNOVIO_CLIENT_SECRET,
+});
+```
+
+---
+
+## Email Validation Utilities
+
+```typescript
+import {
+  isValidEmailSyntax,
+  isPersonalEmail,
+  isBusinessEmail,
+  isDisposableEmail,
+  isRoleAccount,
+  verifyEmailMx,
+  checkDomainCatchAll,
+  PERSONAL_DOMAINS,
+  DISPOSABLE_DOMAINS,
+} from 'linkedin-secret-sauce';
+
+// Syntax validation
+isValidEmailSyntax('john@example.com'); // true
+
+// Domain classification
+isPersonalEmail('john@gmail.com');      // true
+isBusinessEmail('john@acme.com');       // true
+isDisposableEmail('x@mailinator.com');  // true
+isRoleAccount('info@company.com');      // true
+
+// MX verification
+const hasMx = await verifyEmailMx('john@acme.com');
+
+// Catch-all detection
+const isCatchAll = await checkDomainCatchAll('acme.com');
+```
+
+---
+
+## Cost Tracking
+
+Track costs across providers:
+
+```typescript
+let totalCost = 0;
+
+const enrichment = createEnrichmentClient({
+  providers: { /* ... */ },
+  onCost: (provider, cost) => {
+    totalCost += cost;
+    console.log(`${provider}: $${cost.toFixed(4)}`);
+  },
+});
+
+// After enrichment
+console.log(`Total spent: $${totalCost.toFixed(4)}`);
+```
+
+Provider costs are also available as constants:
+
+```typescript
+import { PROVIDER_COSTS } from 'linkedin-secret-sauce';
+
+console.log(PROVIDER_COSTS);
+// {
+//   construct: 0,
+//   ldd: 0,
+//   smartprospect: 0,
+//   cosiall: 0,
+//   trykitt: 0,
+//   bounceban: 0.003,
+//   hunter: 0.015,
+//   snovio: 0.02,
+// }
+```
+
+---
+
+## Caching
+
+Implement a cache adapter to avoid duplicate lookups:
+
+```typescript
+const enrichment = createEnrichmentClient({
+  providers: { /* ... */ },
+  cache: {
+    async get(key: string) {
+      return redis.get(key);
+    },
+    async set(key: string, value: any, ttlMs?: number) {
+      await redis.set(key, value, 'PX', ttlMs || 30 * 24 * 60 * 60 * 1000);
+    },
+  },
+});
+```
+
+Default cache TTL is 30 days.
+
+---
+
+## Custom Provider Order
+
+Override the default 3-phase strategy:
+
+```typescript
+const enrichment = createEnrichmentClient({
+  providers: { /* ... */ },
+  options: {
+    providerOrder: [
+      'ldd',           // Check database first
+      'trykitt',       // Then AI finder
+      'construct',     // Pattern guess
+      'bounceban',     // Verify patterns
+      'hunter',        // Paid finder last
+    ],
+  },
+});
+```
+
+---
+
+## Error Handling
+
+The enrichment client handles errors gracefully - providers that fail return `null` and the next provider is tried. To get visibility:
+
+```typescript
+const enrichment = createEnrichmentClient({
+  providers: { /* ... */ },
+  logger: {
+    debug: (msg, data) => console.debug(msg, data),
+    info: (msg, data) => console.info(msg, data),
+    warn: (msg, data) => console.warn(msg, data),
+    error: (msg, data) => console.error(msg, data),
+  },
+});
+```
+
+---
+
+## LinkedIn Contact Matching
+
+For workflows that start with LinkedIn data:
+
+```typescript
+import {
+  getEmailsForLinkedInContact,
+  salesLeadToContact,
+  searchSalesLeads,
+} from 'linkedin-secret-sauce';
+
+// Search LinkedIn
+const results = await searchSalesLeads('cto fintech', { count: 25 });
+
+// Enrich each lead
+for (const lead of results.items) {
+  const contact = salesLeadToContact(lead);
+  
+  const emailResult = await getEmailsForLinkedInContact(contact, {
+    smartprospect: { email: '...', password: '...' },
+    trykitt: { apiKey: '...' },
+    bounceban: { apiKey: '...' },
+  });
+  
+  console.log(emailResult.bestEmail);
+  console.log(emailResult.emails);
+}
+```
+
+---
+
+## Troubleshooting
+
+### No emails found
+
+1. Ensure at least one provider is configured correctly
+2. Check that the candidate has enough data (name + domain/company)
+3. Try `enrichAll()` to see what each provider returns
+4. Check logs for provider errors
+
+### Low confidence scores
+
+1. Provide more candidate data (LinkedIn URL improves accuracy)
+2. Enable BounceBan verification for pattern-guessed emails
+3. Consider using TryKitt for AI-powered finding
+
+### High costs
+
+1. Ensure FREE providers are configured first
+2. Use `maxCostPerEmail` option to set a budget
+3. Use `confidenceThreshold` to accept lower confidence results from free providers
+4. Monitor costs with `onCost` callback
+
+### Rate limiting
+
+1. Use `delayMs` in batch operations
+2. Check provider-specific rate limits in their documentation
+3. Implement caching to avoid duplicate lookups