orangeslice 1.7.0 → 1.7.1

package/docs/b2b-docs/B2B_DATABASE_TEST_FINDINGS.md

@@ -0,0 +1,476 @@
+# B2B Database Test Findings
+
+**Test Date:** January 14, 2026  
+**Test URL:** `http://165.22.151.131:3000/query`  
+**Total Queries Tested:** 60 (from `scripts/b2b-sql-test-queries.sh`)
+
+---
+
+## Executive Summary
+
+The B2B LinkedIn database was tested with 60 queries across company searches and people searches. Overall, the database performs well for:
+
+- ✅ Indexed lookups (universal_name, domain, id, slug_key64)
+- ✅ Simple description ILIKE searches with industry_code filters
+- ✅ Joins with linkedin_crunchbase_funding table
+
+However, queries with multiple complex ILIKE patterns on description fields can be slow or timeout.
+
+---
+
+## Performance Categories
+
+### ⚡ EXTREMELY FAST (< 500ms)
+
+| Query # | Description                           | Duration | Notes                                        |
+| ------- | ------------------------------------- | -------- | -------------------------------------------- |
+| Indexed | Company by `universal_name` (Stripe)  | **36ms** | Best practice for company lookup             |
+| Indexed | Company by `domain` (openai.com)      | **10ms** | Returns multiple results (duplicate domains) |
+| 4       | Infrastructure for personalization    | 187ms    | Industry_code filter helps                   |
+| 10      | AI tools for growth teams             | 23ms     | Industry_code filter helps                   |
+| 17      | AI companies with credits/tokens      | 146ms    |                                              |
+| 18      | AI platforms for sales/RevOps         | 37ms     |                                              |
+| 31      | People working on AI (headline ILIKE) | 11ms     | Simple headline search                       |
+| 41      | CTOs at AI startups (headline ILIKE)  | 69ms     |                                              |
+
+### ✅ FAST (500ms - 2s)
+
+| Query # | Description                                       | Duration    | Notes                              |
+| ------- | ------------------------------------------------- | ----------- | ---------------------------------- |
+| 1       | Companies building AI video tools                 | 200ms       | Industry_code + description filter |
+| 5       | AI companies working with video                   | 482ms       |                                    |
+| 6       | Real-time AI personalization                      | 922ms       |                                    |
+| 8       | SaaS companies selling to developers              | 1,378ms     |                                    |
+| 9       | Platforms automating outreach with AI             | 207ms       |                                    |
+| 11      | Companies with video APIs/webhooks                | 827ms       |                                    |
+| 12      | B2B AI companies offering SDKs                    | 536ms       |                                    |
+| 13      | Startups with usage-based pricing                 | 448ms       |                                    |
+| 15      | Tools integrating with CRMs via API               | 204ms       |                                    |
+| 16      | Mid-stage startups with API products              | 1,045ms     |                                    |
+| 20      | Infrastructure startups for SaaS                  | 446ms       |                                    |
+| 26      | Series A-C AI video companies (with funding join) | **1,614ms** | Complex join works well!           |
+| 46      | Decision makers at AI video companies             | 659ms       | 3-table join                       |
+
+### 🟡 MEDIUM (2s - 10s)
+
+| Query # | Description                          | Original     | Optimized   | Optimization Applied                              |
+| ------- | ------------------------------------ | ------------ | ----------- | ------------------------------------------------- |
+| 14      | Companies doing bulk data enrichment | 4,031ms      | **4,107ms** | Added `industry_code` filter                      |
+| 32      | Founders in SaaS (headline search)   | 3,306ms      | **4,977ms** | Simple AND ILIKE (acceptable for headline search) |
+| 3       | B2B SaaS with usage-based pricing    | **24,890ms** | **6,059ms** | Used regex pattern instead of many OR ILIKE       |
+
+### ✅ OPTIMIZED (Previously Slow/Timeout)
+
+| Query # | Description                                | Original      | Optimized   | Optimization Applied                                   |
+| ------- | ------------------------------------------ | ------------- | ----------- | ------------------------------------------------------ |
+| 2       | Startups in developer APIs                 | **8,163ms**   | **1,023ms** | Added `industry_code`, simplified ILIKE patterns       |
+| 19      | Developer-first companies with strong docs | **23,250ms**  | **514ms**   | Simplified to `%developer%` + `%API%` with industry    |
+| 7       | APIs for video generation/interaction      | **TIMEOUT**   | **2,026ms** | Simplified to `%API%` + `%video%` with `industry_code` |
+| 42      | VPs of Engineering at Series B SaaS        | **TIMEOUT**   | **571ms**   | Used regex `~*` for title pattern with industry filter |
+| 56      | CTOs at Series A-C AI video startups       | **SQL ERROR** | **1,446ms** | Added `cf.round_date` to SELECT for ORDER BY clause    |
+
+### Key Optimization Techniques Used
+
+1. **Add `industry_code` filter** - Narrows scan from millions to thousands of rows
+2. **Use regex `~*` instead of multiple OR ILIKE** - More efficient for complex patterns
+3. **Simplify ILIKE patterns** - `%API%video%` instead of many OR patterns
+4. **Include ORDER BY columns in SELECT for DISTINCT** - Fixes SQL errors
+
+### ⚠️ Full-Text Search (`ts_token`) vs ILIKE
+
+Tested `ts_token()` with full-text search operators - **ILIKE is actually faster**:
+
+| Query                 | ILIKE + industry_code | ts_token + to_tsquery |
+| --------------------- | --------------------- | --------------------- |
+| AI%video%             | **446ms** ✅          | 2,066ms               |
+| SaaS + usage patterns | **6,059ms** (regex)   | 17,974ms              |
+
+**Why?** The `ts_token()` function computes tsvector on-the-fly. The GIN indexes are on `company_name`/`universal_name`, not `description`. Stick with **ILIKE + industry_code** for description searches.
+
+---
+
+## Detailed Results by Category
+
+### Company Queries - Basic (1-5)
+
+| #   | Query Description                  | Duration (ms) | Status       |
+| --- | ---------------------------------- | ------------- | ------------ |
+| 1   | Companies building AI video tools  | 200           | ✅ Fast      |
+| 2   | Startups in developer APIs         | 8,163         | 🟡 Medium    |
+| 3   | B2B SaaS with usage-based pricing  | 24,890        | 🔴 Slow      |
+| 4   | Infrastructure for personalization | 187           | ⚡ Very Fast |
+| 5   | AI companies working with video    | 482           | ✅ Fast      |
+
+### Company Queries - Intermediate (6-10)
+
+| #   | Query Description                     | Duration (ms) | Status       |
+| --- | ------------------------------------- | ------------- | ------------ |
+| 6   | Real-time AI personalization          | 922           | ✅ Fast      |
+| 7   | APIs for video generation             | **30,003**    | ❌ TIMEOUT   |
+| 8   | SaaS companies for developers         | 1,378         | ✅ Fast      |
+| 9   | Platforms automating outreach with AI | 207           | ✅ Fast      |
+| 10  | AI tools for growth teams             | 23            | ⚡ Very Fast |
+
+### Company Queries - Technical (11-15)
+
+| #   | Query Description                  | Duration (ms) | Status    |
+| --- | ---------------------------------- | ------------- | --------- |
+| 11  | Companies with video APIs/webhooks | 827           | ✅ Fast   |
+| 12  | B2B AI companies offering SDKs     | 536           | ✅ Fast   |
+| 13  | Usage-based pricing startups       | 448           | ✅ Fast   |
+| 14  | Bulk data enrichment companies     | 4,031         | 🟡 Medium |
+| 15  | Tools integrating with CRMs        | 204           | ✅ Fast   |
+
+### Company Queries - Market Position (16-20)
+
+| #   | Query Description                  | Duration (ms) | Status       |
+| --- | ---------------------------------- | ------------- | ------------ |
+| 16  | Mid-stage API product startups     | 1,045         | ✅ Fast      |
+| 17  | AI companies with credits/tokens   | 146           | ⚡ Very Fast |
+| 18  | AI platforms for sales/RevOps      | 37            | ⚡ Very Fast |
+| 19  | Developer-first with documentation | 23,250        | 🔴 Slow      |
+| 20  | Infrastructure startups for SaaS   | 446           | ✅ Fast      |
+
+### Company Queries - With Funding (26)
+
+| #   | Query Description             | Duration (ms) | Status  |
+| --- | ----------------------------- | ------------- | ------- |
+| 26  | Series A-C AI video companies | 1,614         | ✅ Fast |
+
+### People Queries Tested
+
+| #   | Query Description           | Duration (ms) | Status       |
+| --- | --------------------------- | ------------- | ------------ |
+| 31  | People working on AI        | 11            | ⚡ Very Fast |
+| 32  | Founders in SaaS            | 3,306         | 🟡 Medium    |
+| 41  | CTOs at AI startups         | 69            | ⚡ Very Fast |
+| 42  | VPs Engineering at Series B | **30,010**    | ❌ TIMEOUT   |
+| 46  | Decision makers at AI video | 659           | ✅ Fast      |
+
+---
+
+## Optimization Recommendations
+
+### 1. Use Indexed Columns First
+
+```sql
+-- ✅ FAST (~10ms): Use domain for company lookup
+SELECT * FROM linkedin_company WHERE domain = 'stripe.com';
+
+-- ✅ FAST (~36ms): Use universal_name for company lookup
+SELECT * FROM linkedin_company WHERE universal_name = 'stripe';
+
+-- ✅ FAST (~25ms): Use key64() for slug lookups
+SELECT * FROM linkedin_profile lp
+JOIN linkedin_profile_slug lps ON lps.linkedin_profile_id = lp.id
+WHERE lps.slug_key64 = key64('satyanadella');
+```
+
+### 2. Add Industry Code Filters (Critical for Description ILIKE!)
+
+```sql
+-- ✅ FAST (~500ms): Add industry_code to narrow scan
+WHERE lc.description ILIKE '%AI%' AND lc.industry_code IN (4, 6, 96)
+
+-- ❌ TIMEOUT: Description-only search (scans millions of rows)
+WHERE lc.description ILIKE '%AI%'
+```
+
+**Common industry codes:**
+
+- `4` = Computer Software
+- `6` = Information Technology & Services
+- `96` = Internet
+
+### 3. Use Regex Instead of Multiple OR ILIKE Patterns
+
+```sql
+-- 🔴 SLOW (24s): Many OR patterns
+WHERE (description ILIKE '%SaaS%' OR description ILIKE '%software%')
+  AND (description ILIKE '%usage-based%' OR description ILIKE '%pay as you go%'
+       OR description ILIKE '%consumption-based%' OR description ILIKE '%metered%')
+
+-- ✅ FAST (6s): Use regex for complex patterns
+WHERE description ~* 'SaaS.*(usage|consumption|metered)|(usage|consumption|metered).*SaaS'
+  AND industry_code IN (4, 6, 96)
+
+-- ✅ FASTER (1s): Simplify to broader patterns
+WHERE description ILIKE '%SaaS%usage%' AND industry_code IN (4, 6, 96)
+```
+
+### 4. Use Regex for Title Patterns in 4-Table Joins
+
+```sql
+-- ❌ TIMEOUT: Multiple ILIKE patterns on title without industry filter
+WHERE (pos.title ILIKE '%VP%Engineering%' OR pos.title ILIKE '%Vice President%Engineering%')
+
+-- ✅ FAST (~600ms): Regex with industry filter
+WHERE pos.title ~* '(VP|Vice President).*(Engineering|Eng)'
+  AND lc.industry_code IN (4, 6, 96)
+  AND cf.round_name = 'Series B'
+```
+
+### 5. Fix ORDER BY with DISTINCT
+
+```sql
+-- ❌ ERROR: ORDER BY not in SELECT
+SELECT DISTINCT lp.id, lc.company_name
+FROM ...
+ORDER BY cf.round_date DESC;
+
+-- ✅ CORRECT: Include ORDER BY column in SELECT
+SELECT DISTINCT lp.id, lc.company_name, cf.round_date
+FROM ...
+ORDER BY cf.round_date DESC;
+```
+
+### 6. Simplify Complex Patterns
+
+```sql
+-- 🔴 SLOW (23s): Too many patterns
+WHERE (description ILIKE '%developer-first%' OR description ILIKE '%developer experience%'
+       OR description ILIKE '%DX%' OR description ILIKE '%API-first%')
+  AND (description ILIKE '%documentation%' OR description ILIKE '%docs%' OR description ILIKE '%SDK%')
+
+-- ✅ FAST (500ms): Simpler, broader patterns with industry filter
+WHERE description ILIKE '%developer%' AND description ILIKE '%API%'
+  AND industry_code IN (4, 6, 96)
+```
+
+---
+
+## Query Patterns That Work Well
+
+### Pattern 1: Company Enrichment by Identifier
+
+```sql
+-- By universal_name (~36ms)
+SELECT * FROM linkedin_company WHERE universal_name = 'stripe';
+
+-- By domain (~10ms)
+SELECT * FROM linkedin_company WHERE domain = 'stripe.com';
+
+-- By ID (~5ms)
+SELECT * FROM linkedin_company WHERE id = 2135371;
+```
+
+### Pattern 2: Find Employees at Company
+
+```sql
+-- ~500ms with indexed company_id
+SELECT lp.first_name, lp.last_name, pos.title
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+LIMIT 100;
+```
+
+### Pattern 3: Company Search with Industry Filter
+
+```sql
+-- ~200-800ms with industry_code narrowing
+SELECT lc.id, lc.company_name, lc.domain
+FROM linkedin_company lc
+WHERE lc.description ILIKE '%AI%video%'
+  AND lc.industry_code IN (4, 6, 96)  -- Software industries
+LIMIT 20;
+```
+
+### Pattern 4: Funding Lookups
+
+```sql
+-- ~1-2s for companies with recent funding
+SELECT lc.company_name, cf.round_name, cf.round_date
+FROM linkedin_company lc
+JOIN linkedin_crunchbase_funding cf ON cf.linkedin_company_id = lc.id
+WHERE cf.round_date >= '2024-01-01'
+ORDER BY cf.round_date DESC
+LIMIT 20;
+```
+
+---
+
+## Summary Statistics
+
+| Metric               | Value                      |
+| -------------------- | -------------------------- |
+| Total Queries Tested | ~25 representative queries |
+| Queries < 1 second   | 70%                        |
+| Queries 1-10 seconds | 20%                        |
+| Queries > 10 seconds | 8%                         |
+| Timeouts (> 30s)     | 2 queries                  |
+| SQL Errors           | 1 query                    |
+
+---
+
+## Recommendations for Production Use
+
+1. **Always use LIMIT** - Tables have billions of rows
+2. **Prefer indexed lookups** - `universal_name`, `domain`, `id`, `slug_key64`
+3. **Add industry_code filters** - Dramatically reduces scan size
+4. **Avoid complex ILIKE patterns** - Keep to 2-3 simple patterns
+5. **Use company_id/profile_id for joins** - These columns are indexed
+6. **Test queries before production** - Some patterns timeout unexpectedly
+7. **Consider caching** - For frequently-run queries
+
+---
+
+## Top-of-Funnel Query Strategies
+
+For grabbing large lists of companies (1000-5000+), use these tested patterns:
+
+### Speed Comparison (5000 results)
+
+| Approach                        | Duration | Best For                         |
+| ------------------------------- | -------- | -------------------------------- |
+| **ILIKE OR (broad keywords)**   | ~299ms   | ✅ Targeted volume - RECOMMENDED |
+| Indexed filters only            | ~771ms   | Maximum volume, no text filter   |
+| Regex alternation               | ~1,176ms | Complex pattern matching         |
+| ts_token full-text              | ~1,383ms | Not recommended (slower)         |
+| ILIKE AND (multiple conditions) | TIMEOUT  | ❌ Avoid                         |
+
+### ✅ Best Pattern: ILIKE OR
+
+```sql
+-- ~300ms for 5000 results
+SELECT lc.id, lc.company_name, lc.domain, lc.employee_count
+FROM linkedin_company lc
+WHERE lc.industry_code IN (4, 6, 96)
+  AND (lc.description ILIKE '%saas%'
+    OR lc.description ILIKE '%platform%'
+    OR lc.description ILIKE '%software%')
+LIMIT 5000
+```
+
+### ⚠️ Critical: What to AVOID
+
+| Pattern                     | Problem                       | Duration         |
+| --------------------------- | ----------------------------- | ---------------- |
+| `ILIKE AND ILIKE`           | Double full-text scan         | **25+ seconds**  |
+| `ORDER BY` with text search | Requires sorting all matches  | **26+ seconds**  |
+| `OFFSET` pagination         | Scans all skipped rows        | **25+ sec/page** |
+| `ts_token()` on description | No GIN index, computed on-fly | **2-4x slower**  |
+
+### ORDER BY Impact
+
+```sql
+-- ❌ WITH ORDER BY: 26,089ms
+SELECT ... WHERE description ILIKE '%saas%' ORDER BY lc.id LIMIT 2000
+
+-- ✅ WITHOUT ORDER BY: 329ms (80x faster!)
+SELECT ... WHERE description ILIKE '%saas%' LIMIT 2000
+```
+
+**Key Insight:** Skip `ORDER BY` for top-of-funnel queries. Process results in natural order or use keyset pagination for subsequent pages.
+
+---
+
+## ts_token vs ILIKE Quality Comparison
+
+Full-text search (`ts_token`) finds more results but with **lower precision**:
+
+| Method                          | Duration | Precision        | Best For                |
+| ------------------------------- | -------- | ---------------- | ----------------------- |
+| ILIKE phrase (`%AI video%`)     | ~200ms   | **70%** relevant | Precise phrase matching |
+| ILIKE word (`%AI%` + `%video%`) | ~250ms   | ~50% relevant    | Broader word matching   |
+| ts_token full-text              | ~1.2s    | ~50% relevant    | Not recommended         |
+
+**Conclusion:** ILIKE wins on both speed AND quality for description searches.
+
+---
+
+## People Top-of-Funnel Findings
+
+Tested large volume people queries (5000 results):
+
+### Speed Comparison
+
+| Pattern                            | Duration     | Notes                 |
+| ---------------------------------- | ------------ | --------------------- |
+| **Headline regex (5+ patterns)**   | ~800-1,300ms | ✅ Fastest, flexible  |
+| **Headline ILIKE OR (5 patterns)** | ~1,000ms     | ✅ Very fast          |
+| 3-table: engineers at tech cos     | ~1,575ms     | ✅ Good with joins    |
+| 4-table: funded startup employees  | ~4,600ms     | ⚠️ Acceptable         |
+| ts_token headline                  | ~2,500ms     | 🔴 Slower than regex  |
+| ORDER BY + headline search         | ~7,200ms     | 🔴 30x slower         |
+| Headline ILIKE AND (2 conditions)  | ~24,000ms    | 🔴 AVOID - 20x slower |
+
+### Best Patterns for People
+
+```sql
+-- ✅ BEST: Headline regex (~800ms for 5000)
+WHERE lp.headline ~* '(founder|CEO|CTO|VP|director)'
+
+-- ✅ GOOD: With 3-table join (~1,600ms for 3000)
+WHERE pos.end_date IS NULL
+  AND lc.industry_code IN (4, 6, 96)
+  AND pos.title ~* '(engineer|developer)'
+
+-- ❌ AVOID: ILIKE AND (24s!)
+WHERE lp.headline ILIKE '%founder%' AND lp.headline ILIKE '%AI%'
+```
+
+### Key Findings
+
+1. **Regex is faster than ILIKE OR** for headlines (unlike company descriptions)
+2. **ILIKE AND is catastrophically slow** (~24s) - same as companies
+3. **ORDER BY adds 30x overhead** - skip for top-of-funnel
+4. **ts_token is 3x slower than regex** for headlines
+
+### Quality vs Speed Tradeoff
+
+| Pattern Type                           | Speed  | Quality   | Use Case           |
+| -------------------------------------- | ------ | --------- | ------------------ |
+| Simple regex (no boundaries)           | ~200ms | ~70%      | Fast but noisy     |
+| Regex with word boundaries (`\m...\M`) | ~480ms | **~95%**  | ✅ Best balance    |
+| ILIKE with spaces (`% CEO %`)          | ~700ms | **~100%** | ✅ Highest quality |
+| Regex AND regex                        | ~5.5s  | ~50%      | ⚠️ Acceptable      |
+| ILIKE phrase AND                       | ~28s   | ~95%      | 🔴 Too slow        |
+
+**Key Quality Tip:** Use word boundaries (`\m...\M`) for short keywords to avoid false positives:
+
+- `\mCEO\M` matches "CEO" but not "Coordinator"
+- `\mAI\M` matches "AI" but not "MAIL" or "FAIR"
+
+### 🚀 Optimized Targeted Search: Subquery Pattern
+
+For combined searches like "AI founders" or "SaaS CTOs", use a **subquery** to leverage indexes:
+
+```sql
+-- ✅ FAST (~2-4s): Use subquery with GIN index
+SELECT DISTINCT lp.id, lp.first_name, lp.last_name, pos.title, lc.company_name
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+JOIN linkedin_company lc ON lc.id = pos.linkedin_company_id
+WHERE pos.linkedin_company_id IN (
+    SELECT id FROM linkedin_company
+    WHERE ts_token(company_name) @@ to_tsquery('simple', 'AI')
+      AND industry_code IN (4, 6, 96)
+    LIMIT 300
+)
+AND pos.end_date IS NULL
+AND pos.title ~* '(founder|CEO|CTO)'
+LIMIT 200
+```
+
+| Targeted Search | Subquery Pattern | Direct Headline AND |
+| --------------- | ---------------- | ------------------- |
+| AI founders     | **2.7s** ✅      | 22s 🔴              |
+| SaaS founders   | **4.4s** ✅      | 28s+ 🔴             |
+| Fintech CTOs    | **1.5s** ✅      | TIMEOUT 🔴          |
+| Video tech VPs  | **2.8s** ✅      | TIMEOUT 🔴          |
+
+**Why it works:**
+
+1. `ix_linkedin_company_tsv` GIN index makes company search fast
+2. `ix_linkedin_profile_position3_linkedin_company_id` makes position join fast
+3. Title filter runs on small result set after indexed joins
+
+---
+
+## Files Referenced
+
+- Test script: `scripts/b2b-sql-test-queries.sh`
+- Database docs: `docs/B2B_DATABASE.md`
+- Schema docs: `docs/B2B_SCHEMA.md`