orangeslice 1.7.2 → 1.7.3

package/docs/b2b-docs/B2B_EMPLOYEE_SEARCH.md

@@ -0,0 +1,697 @@
+# B2B Employee Search Guide
+
+How to find employees by title at a given company using the B2B database.
+
+---
+
+## Available Fields
+
+When querying for employees, you can SELECT the following fields:
+
+### From `linkedin_profile` (alias: `lp`)
+
+| Field                   | Type    | Description                    |
+| ----------------------- | ------- | ------------------------------ |
+| `first_name`            | varchar | First name                     |
+| `last_name`             | varchar | Last name                      |
+| `formatted_name`        | varchar | Full name                      |
+| `headline`              | varchar | LinkedIn headline              |
+| `location_name`         | varchar | Full location string           |
+| `location_city`         | text    | City                           |
+| `location_region`       | text    | State/region                   |
+| `location_country`      | text    | Country name                   |
+| `location_country_code` | varchar | Country code (e.g., "US")      |
+| `connections`           | integer | Number of LinkedIn connections |
+| `public_profile_url`    | varchar | LinkedIn profile URL           |
+
+### From `linkedin_profile_position3` (alias: `pos`)
+
+| Field          | Type | Description                        |
+| -------------- | ---- | ---------------------------------- |
+| `title`        | text | Job title                          |
+| `company_name` | text | Company name (denormalized)        |
+| `start_date`   | date | Position start date                |
+| `end_date`     | date | Position end date (NULL = current) |
+| `summary`      | text | Role description/summary           |
+
+---
+
+## Quick Start
+
+```sql
+-- Find engineers at a company (fast: 10-60ms)
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371  -- Company ID (e.g., Stripe)
+  AND pos.end_date IS NULL               -- Current employees only
+  AND pos.title ILIKE '%engineer%'       -- Title filter
+LIMIT 50;
+```
+
+---
+
+## Key Tables
+
+| Table                        | Purpose                                    | Size             |
+| ---------------------------- | ------------------------------------------ | ---------------- |
+| `linkedin_profile_position3` | Work experience records                    | **2.6 billion**  |
+| `linkedin_profile`           | Profile details (name, headline, location) | **1.15 billion** |
+| `linkedin_company`           | Company lookup                             | Millions         |
+
+### Critical Indexes
+
+| Index                                               | Column                | Use                     |
+| --------------------------------------------------- | --------------------- | ----------------------- |
+| `ix_linkedin_profile_position3_linkedin_company_id` | `linkedin_company_id` | **Fast company lookup** |
+| `linkedin_profile_pkey`                             | `id`                  | Profile join            |
+
+**Note:** There is NO index on `title` - title filtering happens after the company index scan.
+
+---
+
+## Finding the Company ID
+
+Before searching for employees, you need the company's `linkedin_company_id`:
+
+```sql
+-- By universal_name (URL slug) - FASTEST
+SELECT id, company_name, employee_count
+FROM linkedin_company
+WHERE universal_name = 'stripe';
+
+-- By domain
+SELECT id, company_name, employee_count
+FROM linkedin_company
+WHERE domain = 'stripe.com';
+
+-- By slug with key64 (if using linkedin_company_slug)
+SELECT lc.id, lc.company_name
+FROM linkedin_company lc
+JOIN linkedin_company_slug lcs ON lcs.linkedin_company_id = lc.id
+WHERE lcs.slug_key64 = key64('stripe');
+```
+
+### Common Company IDs (for reference)
+
+| Company | ID       | Employee Count |
+| ------- | -------- | -------------- |
+| Amazon  | 1586     | 770K           |
+| Google  | 1441     | 330K           |
+| Stripe  | 2135371  | ~9K            |
+| OpenAI  | 11130470 | ~7K            |
+| Ramp    | 1406226  | ~3.5K          |
+
+---
+
+## Performance by Company Size
+
+### Tested Performance Results
+
+| Company Size         | Simple Query | DISTINCT ON | Aggregations |
+| -------------------- | ------------ | ----------- | ------------ |
+| **Massive (100K+)**  | 15-65ms      | TIMEOUT     | TIMEOUT      |
+| **Large (10K-100K)** | 10-40ms      | 500ms-1s    | 1-15s        |
+| **Medium (1K-10K)**  | 10-30ms      | 100-200ms   | 100-500ms    |
+| **Small (<1K)**      | 4-20ms       | 10-50ms     | 5-50ms       |
+
+### What This Means
+
+- **Amazon/Google/Microsoft**: Only use simple LIMIT queries
+- **Stripe/Meta/Salesforce**: DISTINCT ON works, avoid heavy aggregations
+- **Ramp/OpenAI**: Everything works reasonably fast
+- **Startups (<1K)**: All queries work instantly
+
+---
+
+## Query Patterns by Company Size
+
+### For MASSIVE Companies (>100K employees)
+
+**DO:**
+
+```sql
+-- Simple LIMIT query (15-65ms)
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  pos.title,
+  pos.company_name
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 1586  -- Amazon
+  AND pos.end_date IS NULL
+  AND pos.title ILIKE '%account executive%'
+LIMIT 50;
+```
+
+**DON'T:**
+
+```sql
+-- These will TIMEOUT (>30s):
+SELECT DISTINCT ON (lp.id) ...  -- NO
+SELECT COUNT(*) ...              -- NO
+GROUP BY pos.title ...           -- NO
+```
+
+### For LARGE Companies (10K-100K employees)
+
+```sql
+-- DISTINCT ON for deduplication (500ms-1s)
+SELECT DISTINCT ON (lp.id)
+  lp.first_name,
+  lp.last_name,
+  lp.formatted_name,
+  lp.headline,
+  lp.location_name,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371  -- Stripe
+  AND pos.end_date IS NULL
+  AND pos.title ILIKE '%vp %'
+ORDER BY lp.id, pos.start_date DESC
+LIMIT 100;
+```
+
+### For MEDIUM/SMALL Companies (<10K employees)
+
+```sql
+-- Full aggregations work (100-500ms)
+SELECT pos.title, COUNT(DISTINCT lp.id) as unique_count
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 1406226  -- Ramp
+  AND pos.end_date IS NULL
+GROUP BY pos.title
+ORDER BY unique_count DESC
+LIMIT 25;
+```
+
+---
+
+## Batch Processing 200+ Companies (Guaranteed No Timeout)
+
+When you need to find the same role type across many companies (including massive ones like Amazon/Google), use this pattern:
+
+### The Golden Rule
+
+**NEVER use DISTINCT ON or aggregations when batch processing companies of unknown size.**
+
+Use this simple pattern instead - it works for ANY company size:
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = :company_id   -- Parameterized
+  AND pos.end_date IS NULL
+  AND (/* your title filters here */)
+LIMIT 25;
+```
+
+### Tested Performance (All Sizes)
+
+| Query Type  | Amazon (770K) | Google (330K) | Stripe (9K) | Ramp (3.5K) | Cluely (86) |
+| ----------- | ------------- | ------------- | ----------- | ----------- | ----------- |
+| C-Suite     | **1.9s**      | **293ms**     | 1s          | 235ms       | 8ms         |
+| Head of Eng | **2.1s**      | **51ms**      | 36ms        | 61ms        | -           |
+| Founders    | **491ms**     | **433ms**     | 602ms       | 46ms        | 4ms         |
+
+**All queries complete successfully - no timeouts even on 770K employee companies.**
+
+### Ready-to-Use Query Templates
+
+#### C-Suite at Any Company
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.formatted_name,
+  lp.headline,
+  lp.location_name,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = :company_id
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE 'ceo%'
+       OR pos.title ILIKE 'cto%'
+       OR pos.title ILIKE 'cfo%'
+       OR pos.title ILIKE 'coo%'
+       OR pos.title ILIKE 'cmo%'
+       OR pos.title ILIKE '%chief exec%'
+       OR pos.title ILIKE '%chief tech%'
+       OR pos.title ILIKE '%chief fin%'
+       OR pos.title ILIKE '%chief oper%'
+       OR pos.title ILIKE '%chief market%')
+LIMIT 25;
+```
+
+#### Head of Engineering at Any Company
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = :company_id
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE '%head of engineer%'
+       OR pos.title ILIKE '%vp engineer%'
+       OR pos.title ILIKE '%vp of engineer%'
+       OR pos.title ILIKE '%director of engineer%'
+       OR pos.title ILIKE '%engineering lead%')
+LIMIT 25;
+```
+
+#### Founders at Any Company
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = :company_id
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE '%founder%'
+       OR pos.title ILIKE '%co-founder%'
+       OR pos.title ILIKE '%cofounder%')
+LIMIT 25;
+```
+
+#### Head of Sales at Any Company
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = :company_id
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE '%head of sales%'
+       OR pos.title ILIKE '%vp sales%'
+       OR pos.title ILIKE '%vp of sales%'
+       OR pos.title ILIKE '%chief revenue%'
+       OR pos.title ILIKE '%sales director%')
+LIMIT 25;
+```
+
+#### Recruiters at Any Company
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = :company_id
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE '%recruit%'
+       OR pos.title ILIKE '%talent acq%'
+       OR pos.title ILIKE '%sourcer%')
+LIMIT 25;
+```
+
+### Handling Duplicates in Application Code
+
+Since we can't use DISTINCT ON for massive companies, deduplicate in your application:
+
+```typescript
+// After fetching results
+const uniqueByName = new Map();
+for (const row of results) {
+   const key = `${row.first_name}-${row.last_name}-${row.title}`;
+   if (!uniqueByName.has(key)) {
+      uniqueByName.set(key, row);
+   }
+}
+const deduplicated = Array.from(uniqueByName.values());
+```
+
+### Why LIMIT 25?
+
+- Most role searches return <25 unique people anyway
+- Higher limits increase query time on massive companies
+- You can increase to 50 if needed, but test first on Amazon/Google
+
+---
+
+## Title Search Patterns
+
+### Role Categories
+
+| Role                     | ILIKE Patterns                                                          | Notes                                      |
+| ------------------------ | ----------------------------------------------------------------------- | ------------------------------------------ |
+| **Recruiters**           | `%recruit%`, `%talent%`, `%sourcer%`                                    | Also catches "Technical Recruiter"         |
+| **C-Suite**              | `ceo%`, `cto%`, `cfo%`, `coo%`, `cmo%`, `%chief%`                       | Watch for "Chief of Staff" false positives |
+| **VPs**                  | `%vp %`, `%vice president%`                                             | Space after "vp" avoids "MVP" matches      |
+| **Directors**            | `%director%`, `%head of%`                                               | Very broad - may need refinement           |
+| **Engineers**            | `%software engineer%`, `%engineer%`                                     | `%engineer%` catches all types             |
+| **Senior Engineers**     | `%senior%engineer%`, `%staff%engineer%`, `%principal%`                  | Senior IC roles                            |
+| **Engineering Managers** | `%engineering manager%`, `%eng manager%`                                | People managers                            |
+| **Sales**                | `%account executive%`, `%sales rep%`, `%ae %`                           | Space after "ae"                           |
+| **Sales Leadership**     | `%head of sales%`, `%sales director%`, `%vp sales%`                     | Sales leaders                              |
+| **SDRs/BDRs**            | `%sales development%`, `%business development%`, `%sdr%`, `%bdr%`       | Entry sales                                |
+| **Product Managers**     | `%product manager%`, `%product lead%`                                   | PM roles                                   |
+| **Marketing**            | `%marketing%`, `%growth%`, `%brand%`                                    | Broad category                             |
+| **HR/People**            | `%hr %`, `%human resources%`, `%people ops%`, `%people partner%`        | HR team                                    |
+| **Data/ML**              | `%data scientist%`, `%machine learning%`, `%ml engineer%`, `%research%` | Technical                                  |
+| **Design**               | `%designer%`, `%ux%`, `%ui%`                                            | Design team                                |
+| **Finance**              | `%finance%`, `%accountant%`, `%controller%`, `%fp&a%`                   | Finance team                               |
+
+### Example Queries by Role
+
+**Find Recruiters:**
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE '%recruit%'
+       OR pos.title ILIKE '%talent acq%'
+       OR pos.title ILIKE '%sourcer%')
+LIMIT 50;
+```
+
+**Find C-Suite:**
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.formatted_name,
+  lp.headline,
+  lp.location_name,
+  lp.connections,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE 'ceo%'
+       OR pos.title ILIKE 'cto%'
+       OR pos.title ILIKE 'cfo%'
+       OR pos.title ILIKE 'coo%'
+       OR pos.title ILIKE '%chief exec%'
+       OR pos.title ILIKE '%chief tech%'
+       OR pos.title ILIKE '%chief fin%'
+       OR pos.title ILIKE '%chief op%')
+LIMIT 30;
+```
+
+**Find Senior Engineers:**
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_city,
+  lp.location_region,
+  lp.location_country,
+  pos.title,
+  pos.company_name,
+  pos.start_date,
+  pos.summary
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE '%senior%engineer%'
+       OR pos.title ILIKE '%staff%engineer%'
+       OR pos.title ILIKE '%principal%engineer%'
+       OR pos.title ILIKE '%lead%engineer%')
+LIMIT 50;
+```
+
+**Find Sales Team:**
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  lp.location_country_code,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+  AND (pos.title ILIKE '%account exec%'
+       OR pos.title ILIKE '%sales rep%'
+       OR pos.title ILIKE '%business development%'
+       OR pos.title ILIKE '%ae %'
+       OR pos.title ILIKE '% ae')
+LIMIT 50;
+```
+
+---
+
+## Current vs Former Employees
+
+### Current Employees
+
+Use `end_date IS NULL` (recommended - more inclusive):
+
+```sql
+WHERE pos.end_date IS NULL
+```
+
+Or use `is_current = TRUE` (more conservative):
+
+```sql
+WHERE pos.is_current = TRUE
+```
+
+**Difference:** `end_date IS NULL` typically returns ~20% more results than `is_current = TRUE`.
+
+### Former Employees (Alumni)
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  pos.title,
+  pos.company_name,
+  pos.start_date,
+  pos.end_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NOT NULL
+LIMIT 50;
+```
+
+---
+
+## Handling Duplicates
+
+People often have multiple position records at the same company due to:
+
+- Title changes/promotions
+- Multiple data sources
+- Profile updates
+
+### Deduplication with DISTINCT ON
+
+```sql
+-- Get most recent title per person (works for <100K employee companies)
+SELECT DISTINCT ON (lp.id)
+  lp.first_name,
+  lp.last_name,
+  lp.formatted_name,
+  lp.headline,
+  lp.location_name,
+  pos.title,
+  pos.company_name,
+  pos.start_date
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+ORDER BY lp.id, pos.start_date DESC NULLS LAST
+LIMIT 100;
+```
+
+### For Massive Companies (No DISTINCT)
+
+For companies like Amazon/Google, deduplicate in application code:
+
+```sql
+-- Get raw results
+SELECT
+  lp.id,
+  lp.first_name,
+  lp.last_name,
+  lp.headline,
+  lp.location_name,
+  pos.title,
+  pos.company_name
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 1586
+  AND pos.end_date IS NULL
+  AND pos.title ILIKE '%engineer%'
+LIMIT 200;  -- Get more to account for duplicates
+```
+
+Then deduplicate by `lp.id` in your application.
+
+---
+
+## Full Employee Profile Query
+
+Get complete employee information with all available fields:
+
+```sql
+SELECT
+  lp.first_name,
+  lp.last_name,
+  lp.formatted_name,
+  lp.headline,
+  lp.location_name,
+  lp.location_city,
+  lp.location_region,
+  lp.location_country,
+  lp.location_country_code,
+  lp.connections,
+  lp.public_profile_url,
+  pos.title,
+  pos.company_name,
+  pos.start_date,
+  pos.end_date,
+  pos.summary
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+  AND pos.title ILIKE '%engineer%'
+LIMIT 50;
+```
+
+---
+
+## Role Breakdown Query
+
+Get title distribution at a company (works for <100K employee companies):
+
+```sql
+SELECT
+  pos.title,
+  COUNT(*) as position_count,
+  COUNT(DISTINCT pos.linkedin_profile_id) as unique_people
+FROM linkedin_profile_position3 pos
+WHERE pos.linkedin_company_id = 2135371  -- Stripe
+  AND pos.end_date IS NULL
+  AND pos.title IS NOT NULL
+GROUP BY pos.title
+ORDER BY unique_people DESC
+LIMIT 30;
+```
+
+---
+
+## Common Gotchas
+
+1. **Never use `linkedin_profile.linkedin_company_id`** - no index, will timeout
+2. **Always use `linkedin_profile_position3`** for company-based searches
+3. **Use `end_date IS NULL`** not `is_current = TRUE` for better coverage
+4. **Always include LIMIT** - especially for large companies
+5. **Avoid DISTINCT ON for 100K+ employee companies** - will timeout
+6. **Avoid COUNT/GROUP BY for 100K+ employee companies** - will timeout
+7. **Title matching is case-insensitive** with ILIKE but watch for variations
+8. **Some titles are NULL** - filter with `pos.title IS NOT NULL` if needed
+9. **Duplicates are normal** - same person may have multiple position records
+
+---
+
+## Performance Tips
+
+1. **Filter by company first** - uses the index
+2. **Add title filter after** - applied as a filter on indexed results
+3. **Use LIMIT early** - don't fetch more than needed
+4. **Avoid ORDER BY** on large result sets (except when needed for DISTINCT ON)
+5. **Check company size first** - adjust query strategy accordingly
+
+### Query Performance Reference
+
+| Query Type   | Small (<1K) | Medium (1K-10K) | Large (10K-100K) | Massive (100K+) |
+| ------------ | ----------- | --------------- | ---------------- | --------------- |
+| Simple LIMIT | 4-20ms      | 10-30ms         | 10-40ms          | 15-65ms         |
+| DISTINCT ON  | 10-50ms     | 100-200ms       | 500ms-1s         | **TIMEOUT**     |
+| GROUP BY     | 5-50ms      | 100-500ms       | 1-15s            | **TIMEOUT**     |
+| COUNT(\*)    | 5-50ms      | 100-500ms       | 1-15s            | **TIMEOUT**     |
+
+---
+
+## Related Documentation
+
+- **[B2B_SCHEMA.md](./B2B_SCHEMA.md)** - Full schema reference
+- **[B2B_DATABASE.md](./B2B_DATABASE.md)** - Database overview and examples