linkedin-secret-sauce 0.12.2 → 0.12.4

package/docs/PLAYGROUND.md

@@ -0,0 +1,553 @@
+# Playground
+
+A comprehensive UI + API server for testing and exploring the LinkedIn Secret Sauce library locally. The playground provides interactive interfaces for all library features, with detailed visualizations, side-by-side provider comparisons, and real-time monitoring.
+
+---
+
+## Quick Start
+
+### Starting the Playground
+
+From the repository root:
+
+```bash
+# Development mode (recommended)
+pnpm dev
+# or
+pnpm dev:playground
+
+# This will:
+# - Auto-detect free ports near 5175 (server) and 5173 (UI)
+# - Print the chosen ports on startup
+# - Open your browser automatically
+# - Enable hot-reload for both client and server
+```
+
+Alternative commands (from `apps/playground`):
+
+```bash
+# Run both client and server
+pnpm dev
+
+# UI only (Vite dev server)
+pnpm client
+
+# Server only (Express API)
+pnpm server
+```
+
+The playground will print URLs like:
+```
+Server: http://localhost:5175
+UI:     http://localhost:5173
+```
+
+---
+
+## Environment Setup
+
+Create `apps/playground/.env.local` or `.env` with your configuration:
+
+```bash
+# === Proxy Configuration (Optional) ===
+# Two formats supported:
+# 1. URL format: http://user:pass@host:port
+# 2. Colon format: host:port:user:pass
+PROXY_STRING=proxy.example.net:16888:user:pass
+
+# === LinkedIn Cookies (Required for most features) ===
+# Cookie pool for LinkedIn API requests
+# Format: accountId=cookieString, one per line
+# See main library docs for cookie extraction guide
+LINKEDIN_COOKIES="
+account1=li_at=...; JSESSIONID=...
+account2=li_at=...; JSESSIONID=...
+"
+
+# === LDD (LinkedIn Data Dump) ===
+LDD_API_URL=https://your-ldd-api.com
+LDD_API_TOKEN=your-ldd-token
+
+# === SmartLead/SmartProspect ===
+SMARTLEAD_EMAIL=your@email.com
+SMARTLEAD_PASSWORD=your_password
+
+# === Cosiall ===
+COSIALL_API_URL=https://api.cosiall.com
+COSIALL_API_KEY=your-cosiall-key
+
+# === TryKitt.ai ===
+TRYKITT_API_KEY=your-trykitt-key
+
+# === BounceBan ===
+BOUNCEBAN_API_KEY=your-bounceban-key
+
+# === Hunter.io ===
+HUNTER_API_KEY=your-hunter-key
+
+# === Snov.io ===
+SNOVIO_CLIENT_ID=your-snovio-client-id
+SNOVIO_CLIENT_SECRET=your-snovio-secret
+
+# === Port Configuration (Optional) ===
+# Auto-increments if ports are busy
+PLAYGROUND_SERVER_PORT=5175
+PLAYGROUND_UI_PORT=5173
+```
+
+**Note:** Most features work without paid API keys. The free tier includes:
+- LinkedIn search and lookups (requires cookies)
+- LDD lookups (requires LDD access)
+- SmartProspect matching (FREE for FlexIQ)
+- Cosiall lookups
+- TryKitt.ai email finding
+- Pattern guessing + MX validation
+
+---
+
+## Features Overview
+
+The playground is organized into 6 main tabs, each with specialized sub-features:
+
+### 1. LinkedIn Tab
+Test LinkedIn Sales Navigator and profile lookup features.
+
+#### Sub-tabs:
+- **Search** - Sales Navigator lead search with advanced filters
+- **Profile Lookup** - Look up profiles by vanity name or Sales Navigator ID
+- **Company Tools** - LinkedIn company lookup and data enrichment
+
+#### Key Features:
+- **Search Filters**: Industry, job title, location, seniority, company size, etc.
+- **Pagination**: Navigate through search results with prev/next buttons
+- **Card View**: Visual display of profile cards with key information
+- **Find Email**: Click "Find Email" on any profile to jump to email enrichment
+- **URL Copying**: Copy search URLs for sharing or automation
+- **Query Inspection**: View the raw GraphQL query sent to LinkedIn
+
+#### Use Cases:
+- Build and test search queries before using in production
+- Verify cookie health by running searches
+- Extract profile data for enrichment workflows
+- Test company lookup and website discovery
+
+---
+
+### 2. SmartLead Tab
+Access the SmartProspect contact database.
+
+#### Sub-tabs:
+- **Email Lookup** - Find verified emails by name + company/domain
+- **Contact Search** - Search SmartProspect database (SmartProspect filters)
+
+#### Key Features:
+- **Two Lookup Modes**:
+  - Search by Name: First/last name + company/domain → finds contact → fetches email
+  - Fetch by ID: Direct lookup with SmartProspect contact ID(s)
+- **Batch Support**: Look up up to 100 contact IDs at once
+- **Verification Status**: See email verification confidence (valid/catch-all/unverified)
+- **Contact Matching**: Automatic matching between LinkedIn and SmartProspect contacts
+
+#### Use Cases:
+- Verify if a contact exists in SmartProspect before doing expensive lookups
+- Bulk email fetching for known contact IDs
+- Test contact matching accuracy
+- FREE email lookups (no credits for FlexIQ customers)
+
+---
+
+### 3. LDD Tab
+Query your private LinkedIn Data Dump database.
+
+#### Features:
+- **Multiple Lookup Methods**:
+  - Username (e.g., `williamhgates`)
+  - Full LinkedIn URL (auto-extracts username)
+  - Source Record ID
+  - Numeric LinkedIn ID
+- **Data Retrieved**:
+  - Email addresses (with type classification)
+  - Phone numbers
+  - LinkedIn identifiers
+- **Performance**: Instant lookups from local/private database
+- **Cost**: 100% FREE - no external API calls
+
+#### Use Cases:
+- First-pass email lookup before trying paid providers
+- Phone number extraction
+- LinkedIn ID mapping and verification
+- Bulk lookups without API costs
+
+---
+
+### 4. Cosiall Tab
+Look up emails from the Cosiall FlexIQ database.
+
+#### Features:
+- **Lookup by**:
+  - LinkedIn vanity name (username)
+  - Full LinkedIn profile URL
+  - ObjectURN (most precise)
+- **URL Normalization**: Handles missing `https://` and trailing slashes
+- **Copy to Clipboard**: One-click copy for single or all emails
+- **Multiple Emails**: Returns all known emails for a profile
+
+#### Use Cases:
+- Alternative to LDD for email lookups
+- ObjectURN-based lookups from LinkedIn API responses
+- Cross-reference emails from multiple sources
+
+---
+
+### 5. Email Tools Tab
+Comprehensive email validation and verification utilities.
+
+#### Sub-tabs:
+- **Validate** - Email validation and classification
+- **Provider Testing** - Side-by-side provider comparison
+- **Matching Debugger** - Contact matching algorithm inspector
+
+#### Validate Features:
+- **Single Email Validation**:
+  - Syntax validation (RFC compliant)
+  - Personal vs. business classification
+  - Disposable email detection
+  - Role account detection (noreply@, info@, etc.)
+  - MX record verification
+- **Batch Validation**: Up to 100 emails at once
+- **Catch-All Detection**: Identify domains that accept all email addresses
+- **Email Existence Check**: SMTP RCPT TO verification (up to 20 emails)
+  - Configurable delay to avoid rate limiting (1-10 seconds)
+  - Real-time status updates
+
+#### Provider Testing Features:
+- **Compare Multiple Providers**:
+  - LDD, SmartProspect, TryKitt.ai (FREE)
+  - Pattern Guessing + MX validation (FREE)
+  - BounceBan (catch-all verification, $0.003/lookup)
+  - Hunter.io ($0.015/lookup)
+  - Snov.io ($0.02/lookup)
+- **Side-by-Side Results**: See speed, accuracy, and cost for each provider
+- **Configuration Status**: Know which providers are configured
+- **Cost Tracking**: See estimated cost per test run
+
+#### Matching Debugger Features:
+- **Field-by-Field Analysis**:
+  - Name matching (first, last, full name normalization)
+  - Title similarity (fuzzy matching, acronyms)
+  - Company matching (normalization, abbreviations)
+  - LinkedIn URL comparison
+- **Weighted Scoring**: See how each field contributes to overall confidence
+- **Match Quality Classification**: Exact, High, Medium, Low, None
+- **Recommendations**: Actionable advice based on match score
+
+#### Use Cases:
+- Validate email lists before sending campaigns
+- Identify catch-all domains that need special verification
+- Compare provider accuracy and speed for your use case
+- Debug why contacts aren't matching between LinkedIn and SmartProspect
+- Understand the matching algorithm to improve data quality
+
+---
+
+### 6. Secret Sauce Tab
+**Unified Email Lookup Workflow** - The complete email enrichment pipeline.
+
+#### Features:
+- **Intelligent Waterfall**: Queries providers in order until a verified email is found
+  1. **Phase 1 (FREE)**: LDD → SmartProspect → Cosiall → TryKitt.ai → Pattern Guessing (parallel)
+  2. **Phase 2 (Verification)**: BounceBan for catch-all domains
+  3. **Phase 3 (Paid)**: Hunter.io → Snov.io (only if free providers fail)
+- **Step-by-Step Visualization**: Watch each provider execute in real-time
+- **LinkedIn Company Lookup**: Auto-discovers company domain from LinkedIn Company URN
+- **Pattern Guessing**: Generates common email patterns (firstname.lastname@, etc.) + MX validation
+- **Catch-All Handling**: Automatically uses BounceBan when domain is catch-all
+- **Smart Thresholds**:
+  - Min Match Confidence (default 60%)
+  - Paid Provider Threshold (default 80%) - skip paid if confidence above this
+- **Pre-fill from Search**: Click "Find Email" on any LinkedIn search result to auto-populate
+
+#### Input Options:
+- **Form Mode**: Fill individual fields
+- **JSON Mode**: Paste entire LinkedIn contact object
+- **Examples**: Pre-built examples including Sales Nav format
+
+#### Contact Fields:
+- **LinkedIn Identifiers**: ObjectURN (for LDD/Cosiall)
+- **Profile Info**: First/last name, title, location
+- **Company Info**: Name, URN (for domain discovery), website
+
+#### Workflow Options:
+- Toggle individual providers on/off
+- Enable/disable paid providers
+- Adjust confidence thresholds
+- Configure SmartProspect matching sensitivity
+
+#### Use Cases:
+- Find emails for LinkedIn profiles with maximum accuracy
+- Test the full enrichment pipeline end-to-end
+- Compare free vs. paid provider results
+- Optimize cost by adjusting thresholds
+- Debug email lookup failures with detailed step logs
+
+---
+
+### 7. System Tab
+Monitor playground health, logs, and costs.
+
+#### Sub-tabs:
+- **Accounts** - LinkedIn cookie pool management
+- **Logs & Metrics** - Real-time logs, metrics, request history
+- **Cost Analytics** - Track spending and estimate budgets
+
+#### Accounts Features:
+- **Cookie Pool Status**: See all LinkedIn accounts and their health
+- **Health Indicators**: Healthy, Cooldown, Unhealthy
+- **Account Actions**:
+  - Apply Cooldown: Temporarily disable an account (15s test)
+  - Reset: Clear failures and restore health
+  - Simulate Auth Fail: Test failure handling
+- **Metadata**: Expiration dates, last used timestamps, failure counts
+- **Automatic Selection**: Test the round-robin account picker
+
+#### Logs & Metrics Features:
+- **Live Logs (SSE)**: Real-time streaming logs from the server
+  - Last 200 entries kept in memory
+  - Auto-scroll to newest
+  - Copy to clipboard
+- **Metrics Snapshot**: Current performance stats
+  - Request counts
+  - Error rates
+  - Cache hit rates
+  - Provider usage
+- **Request History**: Full log of API calls
+  - Paginated view (50 per page)
+  - Request/response details
+  - Clear history button
+
+#### Cost Analytics Features:
+- **Provider Cost Reference**: Cost per lookup for each provider
+- **Session Tracking**: Track costs since playground started
+- **Budget Calculator**:
+  - Set monthly budget
+  - Estimate lookups/month
+  - See projected costs
+  - Budget usage percentage
+- **Waterfall Strategy Diagram**: Visualize the provider pipeline
+- **Assumptions**: 40% free resolution rate, avg paid cost $0.008
+
+#### Use Cases:
+- Monitor cookie health and rotation
+- Debug API failures with real-time logs
+- Estimate monthly costs for your use case
+- Verify that free providers are being used first
+- Track session spending during testing
+
+---
+
+## Tips for Effective Use
+
+### 1. Start with Free Providers
+Before enabling paid APIs:
+- Test LDD, SmartProspect, and Cosiall
+- Use pattern guessing + MX validation
+- Check catch-all status with the Email Tools tab
+- Only enable paid providers if free ones have low success rates
+
+### 2. Use the Matching Debugger
+If SmartProspect matches seem inaccurate:
+- Copy LinkedIn contact to left panel
+- Copy SmartProspect contact to right panel
+- Review field-by-field scores
+- Adjust data normalization or matching logic
+
+### 3. Optimize Costs
+- Set `Paid Provider Threshold` to 80-90% to avoid redundant paid lookups
+- Use batch lookups when possible
+- Check catch-all status before pattern guessing
+- Monitor costs in the Cost Analytics tab
+
+### 4. Test Cookies Regularly
+- Run a simple LinkedIn search periodically
+- Check the Accounts tab for failures
+- Rotate or refresh cookies if seeing auth errors
+
+### 5. Use Examples
+- Every tab has "Fill Example" buttons
+- Examples demonstrate proper data formats
+- Sales Nav example tests Company URN → domain lookup
+
+### 6. Leverage URL Parameters
+The Secret Sauce tab accepts URL parameters for deep linking:
+```
+?tab=secret-sauce&firstName=John&lastName=Doe&companyName=Acme&objectUrn=urn:li:member:123
+```
+This enables "Find Email" buttons from search results.
+
+### 7. Monitor Performance
+- Watch step execution times in Unified Email Lookup
+- Check provider response times in Provider Testing
+- Review logs for slow queries or errors
+
+---
+
+## Troubleshooting
+
+### UI shows blank tab
+- **Solution**: Refresh the page. Vite HMR will reconnect automatically.
+
+### Ports already in use
+- **Solution**: The dev script auto-picks the next free port and logs it. Check console output.
+
+### No LinkedIn search results
+- **Causes**:
+  - Invalid/expired cookies
+  - Cookie doesn't have Sales Navigator access
+  - LinkedIn rate limiting
+- **Solutions**:
+  - Check Accounts tab for cookie health
+  - Try a different account (click "Simulate Selection")
+  - Refresh cookies from browser
+
+### Provider shows "Not configured"
+- **Solution**: Add the required environment variables to `.env.local` (see Environment Setup above)
+
+### Pattern guessing finds too many emails
+- **Cause**: Domain is catch-all (accepts all addresses)
+- **Solution**: Enable BounceBan verification or use Hunter.io for confirmation
+
+### SmartProspect matches are inaccurate
+- **Causes**:
+  - Name variations (nicknames, middle names)
+  - Company name differences (Inc. vs Corp, abbreviations)
+  - Outdated titles
+- **Solutions**:
+  - Use Matching Debugger to see field scores
+  - Lower min confidence threshold
+  - Include more fields (title, company)
+
+### Email validation fails
+- **Causes**:
+  - Invalid MX records
+  - Aggressive spam filtering
+  - Rate limiting on SMTP checks
+- **Solutions**:
+  - Increase delay in Email Existence Check (5-10s)
+  - Use smaller batches (5-10 emails)
+  - Check catch-all status first
+
+---
+
+## Development Notes
+
+### Project Structure
+```
+apps/playground/
+├── src/                 # React UI
+│   ├── ui/             # Tab components
+│   │   ├── App.tsx     # Main app with routing
+│   │   ├── *Tab.tsx    # Individual tab components
+│   │   └── layout/     # Layout components
+│   └── main.tsx        # React entry point
+├── server/             # Express API
+│   ├── index.ts        # Server entry point
+│   └── routes/         # API endpoints
+├── scripts/            # Dev utilities
+│   └── dev.mjs         # Concurrent dev server launcher
+└── public/             # Static assets
+```
+
+### API Endpoints
+The server exposes REST endpoints matching each tab's functionality:
+- `/api/linkedin/*` - Search, profiles, companies
+- `/api/smartlead/*` - SmartProspect lookups
+- `/api/ldd/*` - LDD lookups
+- `/api/cosiall/*` - Cosiall lookups
+- `/api/email/*` - Validation, verification
+- `/api/unified-email` - Full enrichment workflow
+- `/api/providers/*` - Provider testing
+- `/api/matching/*` - Contact matching
+- `/api/accounts/*` - Cookie management
+- `/api/logs/stream` - SSE log stream
+- `/api/costs/*` - Cost tracking
+
+### Adding a New Provider
+1. Add API client to library (`src/providers/`)
+2. Add endpoint to playground server (`apps/playground/server/`)
+3. Add UI component (`apps/playground/src/ui/`)
+4. Update provider list in `ProviderTestingTab.tsx`
+5. Update cost reference in `CostAnalyticsTab.tsx`
+6. Add to waterfall in unified email workflow
+
+---
+
+## Advanced Features
+
+### Custom Search Filters
+The LinkedIn Search tab supports all Sales Navigator filters:
+- Boolean operators in keywords
+- Nested location filters
+- Custom date ranges
+- Function + seniority combinations
+- Company headcount ranges
+
+### Batch Operations
+- SmartLead: Up to 100 contact IDs
+- Email validation: Up to 100 emails
+- Email existence: Up to 20 emails (to avoid SMTP blocks)
+
+### Real-time Monitoring
+- SSE (Server-Sent Events) for live logs
+- Auto-refresh for accounts/metrics (5s interval)
+- Progress tracking for long-running operations
+
+### Export Capabilities
+- Copy logs, metrics, search results to clipboard
+- Copy email addresses (single or batch)
+- Copy GraphQL queries for debugging
+- Copy LinkedIn URLs for sharing
+
+---
+
+## Additional Resources
+
+- **Library Docs**: See main [README.md](../README.md) for library usage
+- **API Docs**: TypeDoc generated docs at `/docs/api/`
+- **Code Search**: Use `pnpm search -- "pattern"` (wrapper for ripgrep)
+- **Tests**: Run `pnpm test` for Vitest unit tests
+
+---
+
+## Performance Tips
+
+### Optimize Search Speed
+- Use fewer filters to get faster responses
+- Limit result count (10-25 per page)
+- Enable caching on frequently-used searches
+
+### Reduce API Costs
+- Always try free providers first (default behavior)
+- Set high confidence thresholds for paid providers
+- Use batch operations when possible
+- Cache results locally
+
+### Avoid Rate Limiting
+- Spread requests across multiple cookies (automatic)
+- Use cooldowns when hitting 429 errors
+- Add delays between batch operations
+- Monitor logs for rate limit warnings
+
+---
+
+## Support
+
+For issues or questions:
+1. Check this documentation
+2. Review the library [README.md](../README.md)
+3. Inspect browser console and server logs
+4. Use the Matching Debugger for contact issues
+5. Check environment variable configuration
+
+---
+
+**Happy testing!** The playground is your sandbox for exploring LinkedIn Secret Sauce capabilities before integrating into production.