orangeslice 1.6.1 → 1.7.0

package/docs/B2B_EMPLOYEE_SEARCH.md

@@ -1,697 +0,0 @@
-# B2B Employee Search Guide
-
-How to find employees by title at a given company using the B2B database.
-
----
-
-## Available Fields
-
-When querying for employees, you can SELECT the following fields:
-
-### From `linkedin_profile` (alias: `lp`)
-
-| Field                   | Type    | Description                    |
-| ----------------------- | ------- | ------------------------------ |
-| `first_name`            | varchar | First name                     |
-| `last_name`             | varchar | Last name                      |
-| `formatted_name`        | varchar | Full name                      |
-| `headline`              | varchar | LinkedIn headline              |
-| `location_name`         | varchar | Full location string           |
-| `location_city`         | text    | City                           |
-| `location_region`       | text    | State/region                   |
-| `location_country`      | text    | Country name                   |
-| `location_country_code` | varchar | Country code (e.g., "US")      |
-| `connections`           | integer | Number of LinkedIn connections |
-| `public_profile_url`    | varchar | LinkedIn profile URL           |
-
-### From `linkedin_profile_position3` (alias: `pos`)
-
-| Field          | Type | Description                        |
-| -------------- | ---- | ---------------------------------- |
-| `title`        | text | Job title                          |
-| `company_name` | text | Company name (denormalized)        |
-| `start_date`   | date | Position start date                |
-| `end_date`     | date | Position end date (NULL = current) |
-| `summary`      | text | Role description/summary           |
-
----
-
-## Quick Start
-
-```sql
--- Find engineers at a company (fast: 10-60ms)
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371  -- Company ID (e.g., Stripe)
-  AND pos.end_date IS NULL               -- Current employees only
-  AND pos.title ILIKE '%engineer%'       -- Title filter
-LIMIT 50;
-```
-
----
-
-## Key Tables
-
-| Table                        | Purpose                                    | Size             |
-| ---------------------------- | ------------------------------------------ | ---------------- |
-| `linkedin_profile_position3` | Work experience records                    | **2.6 billion**  |
-| `linkedin_profile`           | Profile details (name, headline, location) | **1.15 billion** |
-| `linkedin_company`           | Company lookup                             | Millions         |
-
-### Critical Indexes
-
-| Index                                               | Column                | Use                     |
-| --------------------------------------------------- | --------------------- | ----------------------- |
-| `ix_linkedin_profile_position3_linkedin_company_id` | `linkedin_company_id` | **Fast company lookup** |
-| `linkedin_profile_pkey`                             | `id`                  | Profile join            |
-
-**Note:** There is NO index on `title` - title filtering happens after the company index scan.
-
----
-
-## Finding the Company ID
-
-Before searching for employees, you need the company's `linkedin_company_id`:
-
-```sql
--- By universal_name (URL slug) - FASTEST
-SELECT id, company_name, employee_count
-FROM linkedin_company
-WHERE universal_name = 'stripe';
-
--- By domain
-SELECT id, company_name, employee_count
-FROM linkedin_company
-WHERE domain = 'stripe.com';
-
--- By slug with key64 (if using linkedin_company_slug)
-SELECT lc.id, lc.company_name
-FROM linkedin_company lc
-JOIN linkedin_company_slug lcs ON lcs.linkedin_company_id = lc.id
-WHERE lcs.slug_key64 = key64('stripe');
-```
-
-### Common Company IDs (for reference)
-
-| Company | ID       | Employee Count |
-| ------- | -------- | -------------- |
-| Amazon  | 1586     | 770K           |
-| Google  | 1441     | 330K           |
-| Stripe  | 2135371  | ~9K            |
-| OpenAI  | 11130470 | ~7K            |
-| Ramp    | 1406226  | ~3.5K          |
-
----
-
-## Performance by Company Size
-
-### Tested Performance Results
-
-| Company Size         | Simple Query | DISTINCT ON | Aggregations |
-| -------------------- | ------------ | ----------- | ------------ |
-| **Massive (100K+)**  | 15-65ms      | TIMEOUT     | TIMEOUT      |
-| **Large (10K-100K)** | 10-40ms      | 500ms-1s    | 1-15s        |
-| **Medium (1K-10K)**  | 10-30ms      | 100-200ms   | 100-500ms    |
-| **Small (<1K)**      | 4-20ms       | 10-50ms     | 5-50ms       |
-
-### What This Means
-
-- **Amazon/Google/Microsoft**: Only use simple LIMIT queries
-- **Stripe/Meta/Salesforce**: DISTINCT ON works, avoid heavy aggregations
-- **Ramp/OpenAI**: Everything works reasonably fast
-- **Startups (<1K)**: All queries work instantly
-
----
-
-## Query Patterns by Company Size
-
-### For MASSIVE Companies (>100K employees)
-
-**DO:**
-
-```sql
--- Simple LIMIT query (15-65ms)
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  pos.title,
-  pos.company_name
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 1586  -- Amazon
-  AND pos.end_date IS NULL
-  AND pos.title ILIKE '%account executive%'
-LIMIT 50;
-```
-
-**DON'T:**
-
-```sql
--- These will TIMEOUT (>30s):
-SELECT DISTINCT ON (lp.id) ...  -- NO
-SELECT COUNT(*) ...              -- NO
-GROUP BY pos.title ...           -- NO
-```
-
-### For LARGE Companies (10K-100K employees)
-
-```sql
--- DISTINCT ON for deduplication (500ms-1s)
-SELECT DISTINCT ON (lp.id)
-  lp.first_name,
-  lp.last_name,
-  lp.formatted_name,
-  lp.headline,
-  lp.location_name,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371  -- Stripe
-  AND pos.end_date IS NULL
-  AND pos.title ILIKE '%vp %'
-ORDER BY lp.id, pos.start_date DESC
-LIMIT 100;
-```
-
-### For MEDIUM/SMALL Companies (<10K employees)
-
-```sql
--- Full aggregations work (100-500ms)
-SELECT pos.title, COUNT(DISTINCT lp.id) as unique_count
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 1406226  -- Ramp
-  AND pos.end_date IS NULL
-GROUP BY pos.title
-ORDER BY unique_count DESC
-LIMIT 25;
-```
-
----
-
-## Batch Processing 200+ Companies (Guaranteed No Timeout)
-
-When you need to find the same role type across many companies (including massive ones like Amazon/Google), use this pattern:
-
-### The Golden Rule
-
-**NEVER use DISTINCT ON or aggregations when batch processing companies of unknown size.**
-
-Use this simple pattern instead - it works for ANY company size:
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = :company_id   -- Parameterized
-  AND pos.end_date IS NULL
-  AND (/* your title filters here */)
-LIMIT 25;
-```
-
-### Tested Performance (All Sizes)
-
-| Query Type  | Amazon (770K) | Google (330K) | Stripe (9K) | Ramp (3.5K) | Cluely (86) |
-| ----------- | ------------- | ------------- | ----------- | ----------- | ----------- |
-| C-Suite     | **1.9s**      | **293ms**     | 1s          | 235ms       | 8ms         |
-| Head of Eng | **2.1s**      | **51ms**      | 36ms        | 61ms        | -           |
-| Founders    | **491ms**     | **433ms**     | 602ms       | 46ms        | 4ms         |
-
-**All queries complete successfully - no timeouts even on 770K employee companies.**
-
-### Ready-to-Use Query Templates
-
-#### C-Suite at Any Company
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.formatted_name,
-  lp.headline,
-  lp.location_name,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = :company_id
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE 'ceo%'
-       OR pos.title ILIKE 'cto%'
-       OR pos.title ILIKE 'cfo%'
-       OR pos.title ILIKE 'coo%'
-       OR pos.title ILIKE 'cmo%'
-       OR pos.title ILIKE '%chief exec%'
-       OR pos.title ILIKE '%chief tech%'
-       OR pos.title ILIKE '%chief fin%'
-       OR pos.title ILIKE '%chief oper%'
-       OR pos.title ILIKE '%chief market%')
-LIMIT 25;
-```
-
-#### Head of Engineering at Any Company
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = :company_id
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE '%head of engineer%'
-       OR pos.title ILIKE '%vp engineer%'
-       OR pos.title ILIKE '%vp of engineer%'
-       OR pos.title ILIKE '%director of engineer%'
-       OR pos.title ILIKE '%engineering lead%')
-LIMIT 25;
-```
-
-#### Founders at Any Company
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = :company_id
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE '%founder%'
-       OR pos.title ILIKE '%co-founder%'
-       OR pos.title ILIKE '%cofounder%')
-LIMIT 25;
-```
-
-#### Head of Sales at Any Company
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = :company_id
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE '%head of sales%'
-       OR pos.title ILIKE '%vp sales%'
-       OR pos.title ILIKE '%vp of sales%'
-       OR pos.title ILIKE '%chief revenue%'
-       OR pos.title ILIKE '%sales director%')
-LIMIT 25;
-```
-
-#### Recruiters at Any Company
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = :company_id
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE '%recruit%'
-       OR pos.title ILIKE '%talent acq%'
-       OR pos.title ILIKE '%sourcer%')
-LIMIT 25;
-```
-
-### Handling Duplicates in Application Code
-
-Since we can't use DISTINCT ON for massive companies, deduplicate in your application:
-
-```typescript
-// After fetching results
-const uniqueByName = new Map();
-for (const row of results) {
-   const key = `${row.first_name}-${row.last_name}-${row.title}`;
-   if (!uniqueByName.has(key)) {
-      uniqueByName.set(key, row);
-   }
-}
-const deduplicated = Array.from(uniqueByName.values());
-```
-
-### Why LIMIT 25?
-
-- Most role searches return <25 unique people anyway
-- Higher limits increase query time on massive companies
-- You can increase to 50 if needed, but test first on Amazon/Google
-
----
-
-## Title Search Patterns
-
-### Role Categories
-
-| Role                     | ILIKE Patterns                                                          | Notes                                      |
-| ------------------------ | ----------------------------------------------------------------------- | ------------------------------------------ |
-| **Recruiters**           | `%recruit%`, `%talent%`, `%sourcer%`                                    | Also catches "Technical Recruiter"         |
-| **C-Suite**              | `ceo%`, `cto%`, `cfo%`, `coo%`, `cmo%`, `%chief%`                       | Watch for "Chief of Staff" false positives |
-| **VPs**                  | `%vp %`, `%vice president%`                                             | Space after "vp" avoids "MVP" matches      |
-| **Directors**            | `%director%`, `%head of%`                                               | Very broad - may need refinement           |
-| **Engineers**            | `%software engineer%`, `%engineer%`                                     | `%engineer%` catches all types             |
-| **Senior Engineers**     | `%senior%engineer%`, `%staff%engineer%`, `%principal%`                  | Senior IC roles                            |
-| **Engineering Managers** | `%engineering manager%`, `%eng manager%`                                | People managers                            |
-| **Sales**                | `%account executive%`, `%sales rep%`, `%ae %`                           | Space after "ae"                           |
-| **Sales Leadership**     | `%head of sales%`, `%sales director%`, `%vp sales%`                     | Sales leaders                              |
-| **SDRs/BDRs**            | `%sales development%`, `%business development%`, `%sdr%`, `%bdr%`       | Entry sales                                |
-| **Product Managers**     | `%product manager%`, `%product lead%`                                   | PM roles                                   |
-| **Marketing**            | `%marketing%`, `%growth%`, `%brand%`                                    | Broad category                             |
-| **HR/People**            | `%hr %`, `%human resources%`, `%people ops%`, `%people partner%`        | HR team                                    |
-| **Data/ML**              | `%data scientist%`, `%machine learning%`, `%ml engineer%`, `%research%` | Technical                                  |
-| **Design**               | `%designer%`, `%ux%`, `%ui%`                                            | Design team                                |
-| **Finance**              | `%finance%`, `%accountant%`, `%controller%`, `%fp&a%`                   | Finance team                               |
-
-### Example Queries by Role
-
-**Find Recruiters:**
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE '%recruit%'
-       OR pos.title ILIKE '%talent acq%'
-       OR pos.title ILIKE '%sourcer%')
-LIMIT 50;
-```
-
-**Find C-Suite:**
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.formatted_name,
-  lp.headline,
-  lp.location_name,
-  lp.connections,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE 'ceo%'
-       OR pos.title ILIKE 'cto%'
-       OR pos.title ILIKE 'cfo%'
-       OR pos.title ILIKE 'coo%'
-       OR pos.title ILIKE '%chief exec%'
-       OR pos.title ILIKE '%chief tech%'
-       OR pos.title ILIKE '%chief fin%'
-       OR pos.title ILIKE '%chief op%')
-LIMIT 30;
-```
-
-**Find Senior Engineers:**
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_city,
-  lp.location_region,
-  lp.location_country,
-  pos.title,
-  pos.company_name,
-  pos.start_date,
-  pos.summary
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE '%senior%engineer%'
-       OR pos.title ILIKE '%staff%engineer%'
-       OR pos.title ILIKE '%principal%engineer%'
-       OR pos.title ILIKE '%lead%engineer%')
-LIMIT 50;
-```
-
-**Find Sales Team:**
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  lp.location_country_code,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NULL
-  AND (pos.title ILIKE '%account exec%'
-       OR pos.title ILIKE '%sales rep%'
-       OR pos.title ILIKE '%business development%'
-       OR pos.title ILIKE '%ae %'
-       OR pos.title ILIKE '% ae')
-LIMIT 50;
-```
-
----
-
-## Current vs Former Employees
-
-### Current Employees
-
-Use `end_date IS NULL` (recommended - more inclusive):
-
-```sql
-WHERE pos.end_date IS NULL
-```
-
-Or use `is_current = TRUE` (more conservative):
-
-```sql
-WHERE pos.is_current = TRUE
-```
-
-**Difference:** `end_date IS NULL` typically returns ~20% more results than `is_current = TRUE`.
-
-### Former Employees (Alumni)
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  pos.title,
-  pos.company_name,
-  pos.start_date,
-  pos.end_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NOT NULL
-LIMIT 50;
-```
-
----
-
-## Handling Duplicates
-
-People often have multiple position records at the same company due to:
-
-- Title changes/promotions
-- Multiple data sources
-- Profile updates
-
-### Deduplication with DISTINCT ON
-
-```sql
--- Get most recent title per person (works for <100K employee companies)
-SELECT DISTINCT ON (lp.id)
-  lp.first_name,
-  lp.last_name,
-  lp.formatted_name,
-  lp.headline,
-  lp.location_name,
-  pos.title,
-  pos.company_name,
-  pos.start_date
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NULL
-ORDER BY lp.id, pos.start_date DESC NULLS LAST
-LIMIT 100;
-```
-
-### For Massive Companies (No DISTINCT)
-
-For companies like Amazon/Google, deduplicate in application code:
-
-```sql
--- Get raw results
-SELECT
-  lp.id,
-  lp.first_name,
-  lp.last_name,
-  lp.headline,
-  lp.location_name,
-  pos.title,
-  pos.company_name
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 1586
-  AND pos.end_date IS NULL
-  AND pos.title ILIKE '%engineer%'
-LIMIT 200;  -- Get more to account for duplicates
-```
-
-Then deduplicate by `lp.id` in your application.
-
----
-
-## Full Employee Profile Query
-
-Get complete employee information with all available fields:
-
-```sql
-SELECT
-  lp.first_name,
-  lp.last_name,
-  lp.formatted_name,
-  lp.headline,
-  lp.location_name,
-  lp.location_city,
-  lp.location_region,
-  lp.location_country,
-  lp.location_country_code,
-  lp.connections,
-  lp.public_profile_url,
-  pos.title,
-  pos.company_name,
-  pos.start_date,
-  pos.end_date,
-  pos.summary
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NULL
-  AND pos.title ILIKE '%engineer%'
-LIMIT 50;
-```
-
----
-
-## Role Breakdown Query
-
-Get title distribution at a company (works for <100K employee companies):
-
-```sql
-SELECT
-  pos.title,
-  COUNT(*) as position_count,
-  COUNT(DISTINCT pos.linkedin_profile_id) as unique_people
-FROM linkedin_profile_position3 pos
-WHERE pos.linkedin_company_id = 2135371  -- Stripe
-  AND pos.end_date IS NULL
-  AND pos.title IS NOT NULL
-GROUP BY pos.title
-ORDER BY unique_people DESC
-LIMIT 30;
-```
-
----
-
-## Common Gotchas
-
-1. **Never use `linkedin_profile.linkedin_company_id`** - no index, will timeout
-2. **Always use `linkedin_profile_position3`** for company-based searches
-3. **Use `end_date IS NULL`** not `is_current = TRUE` for better coverage
-4. **Always include LIMIT** - especially for large companies
-5. **Avoid DISTINCT ON for 100K+ employee companies** - will timeout
-6. **Avoid COUNT/GROUP BY for 100K+ employee companies** - will timeout
-7. **Title matching is case-insensitive** with ILIKE but watch for variations
-8. **Some titles are NULL** - filter with `pos.title IS NOT NULL` if needed
-9. **Duplicates are normal** - same person may have multiple position records
-
----
-
-## Performance Tips
-
-1. **Filter by company first** - uses the index
-2. **Add title filter after** - applied as a filter on indexed results
-3. **Use LIMIT early** - don't fetch more than needed
-4. **Avoid ORDER BY** on large result sets (except when needed for DISTINCT ON)
-5. **Check company size first** - adjust query strategy accordingly
-
-### Query Performance Reference
-
-| Query Type   | Small (<1K) | Medium (1K-10K) | Large (10K-100K) | Massive (100K+) |
-| ------------ | ----------- | --------------- | ---------------- | --------------- |
-| Simple LIMIT | 4-20ms      | 10-30ms         | 10-40ms          | 15-65ms         |
-| DISTINCT ON  | 10-50ms     | 100-200ms       | 500ms-1s         | **TIMEOUT**     |
-| GROUP BY     | 5-50ms      | 100-500ms       | 1-15s            | **TIMEOUT**     |
-| COUNT(\*)    | 5-50ms      | 100-500ms       | 1-15s            | **TIMEOUT**     |
-
----
-
-## Related Documentation
-
-- **[B2B_SCHEMA.md](./B2B_SCHEMA.md)** - Full schema reference
-- **[B2B_DATABASE.md](./B2B_DATABASE.md)** - Database overview and examples