orangeslice 1.6.0 → 1.7.0

package/docs/strategies.md

@@ -0,0 +1,250 @@
+# Research Strategies
+
+Patterns for prospecting, enrichment, and social listening.
+
+---
+
+## Prospecting: Two Approaches
+
+### 1. Direct Query with Filters (Preferred)
+
+Use when criteria is directly searchable:
+
+- **Google dorking** — `"AI CRM" site:linkedin.com/company`
+- **B2B database** — industry, company size, funding, job titles
+
+```typescript
+// Direct: Companies in fintech with 50-500 employees
+const companies = await orangeslice.b2b.sql(`
+  SELECT * FROM linkedin_company 
+  WHERE industry ILIKE '%fintech%'
+    AND employee_count BETWEEN 50 AND 500
+  LIMIT 100
+`);
+```
+
+### 2. Search → Enrich → Qualify
+
+Use when criteria can't be searched directly:
+
+- "Companies that recently switched CRMs"
+- "Are they actively hiring for this role?"
+- "Do they use [specific tool]?"
+
+```typescript
+// 1. Broad search
+const { results } = await orangeslice.serp.search(
+  `"switched to Salesforce" site:linkedin.com/posts`
+);
+
+// 2. Enrich each result
+for (const result of results) {
+  const companyName = extractCompanyFromPost(result);
+  const company = await orangeslice.b2b.sql(`
+    SELECT * FROM linkedin_company WHERE company_name ILIKE '%${companyName}%'
+  `);
+  
+  // 3. Qualify
+  if (company[0]?.employee_count > 100) {
+    // Add to qualified list
+  }
+}
+```
+
+---
+
+## Data Enrichment Pattern
+
+**Standard: Search → Scrape → Extract**
+
+```typescript
+// 1. Find relevant pages
+const { results } = await orangeslice.serp.search(
+  `site:${domain} "practice areas" "medical malpractice"`
+);
+
+// 2. Scrape the top result
+const { markdown } = await orangeslice.firecrawl.scrape(results[0].link);
+
+// 3. Extract structured data
+const data = await orangeslice.generateObject.generate({
+  prompt: `Does this law firm handle medical malpractice cases?\n\n${markdown}`,
+  schema: {
+    type: "object",
+    properties: {
+      handles_med_mal: { type: "boolean" },
+      practice_areas: { type: "array", items: { type: "string" } }
+    }
+  }
+});
+```
+
+### When to Use Each Tool
+
+| Use Search → Scrape → Extract | Use `browser.execute` |
+|------------------------------|----------------------|
+| Data spread across unknown pages | Same template across pages |
+| Varied/unknown page structure | Need specific CSS selectors |
+| One-off enrichment | Scraping lists/tables |
+
+---
+
+## Social Listening
+
+Find posts mentioning topics, brands, or competitors.
+
+### Finding Posts: Use Dorking
+
+```
+# LinkedIn posts mentioning topic
+"AI sales tools" site:linkedin.com/posts
+
+# Twitter/X posts
+"competitor name" site:x.com inurl:status
+
+# Reddit discussions
+"product name" site:reddit.com
+```
+
+### Common Problem: Sellers vs. Complainers
+
+Users want people **complaining about** tools. Searches return mostly **people selling** alternatives.
+
+**Solution: Filter with verification**
+
+```typescript
+const { results } = await orangeslice.serp.search(
+  `"hate Salesforce" OR "frustrated with Salesforce" site:linkedin.com/posts`
+);
+
+for (const post of results) {
+  // Get author info
+  const authorUrl = extractAuthorUrl(post.link);
+  const profile = await orangeslice.b2b.sql(`...`);
+  
+  // Filter out salespeople
+  if (!profile.headline?.toLowerCase().includes('sales')) {
+    // This is likely a real complainer
+  }
+}
+```
+
+---
+
+## Company Research Checklist
+
+```typescript
+async function researchCompany(domain: string) {
+  // 1. Basic company info
+  const company = await orangeslice.b2b.sql(`
+    SELECT * FROM linkedin_company WHERE domain = '${domain}'
+  `);
+  
+  // 2. Leadership team
+  const leadership = await orangeslice.b2b.sql(`
+    SELECT lp.first_name, lp.last_name, pos.title
+    FROM linkedin_profile lp
+    JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+    WHERE pos.linkedin_company_id = ${company[0].id}
+      AND pos.end_date IS NULL
+      AND (pos.title ILIKE 'ceo%' OR pos.title ILIKE 'cto%' OR pos.title ILIKE '%founder%')
+    LIMIT 10
+  `);
+  
+  // 3. Funding history
+  const funding = await orangeslice.b2b.sql(`
+    SELECT * FROM linkedin_crunchbase_funding 
+    WHERE linkedin_company_id = ${company[0].id}
+    ORDER BY announced_date DESC
+  `);
+  
+  // 4. Recent news
+  const news = await orangeslice.serp.search(
+    `"${company[0].company_name}" funding OR acquisition`,
+    { tbs: "qdr:m" }
+  );
+  
+  // 5. Website content
+  const about = await orangeslice.firecrawl.scrape(`https://${domain}/about`);
+  
+  // 6. Current job openings
+  const jobs = await orangeslice.b2b.sql(`
+    SELECT title, locality FROM linkedin_job
+    WHERE linkedin_company_id = ${company[0].id}
+      AND closed_since IS NULL
+    LIMIT 20
+  `);
+  
+  return { company, leadership, funding, news, about, jobs };
+}
+```
+
+---
+
+## Person Research Checklist
+
+```typescript
+async function researchPerson(linkedinUrl: string) {
+  // Extract profile ID from URL
+  const parts = linkedinUrl.split('/in/');
+  const linkedinUserId = parts[1]?.split('/')[0];
+  
+  // 1. Profile info
+  const profile = await orangeslice.b2b.sql(`
+    SELECT * FROM linkedin_profile 
+    WHERE linkedin_user_id = '${linkedinUserId}'
+  `);
+  
+  // 2. Work history
+  const positions = await orangeslice.b2b.sql(`
+    SELECT pos.*, lc.company_name, lc.domain
+    FROM linkedin_profile_position3 pos
+    JOIN linkedin_company lc ON lc.id = pos.linkedin_company_id
+    WHERE pos.linkedin_profile_id = ${profile[0].id}
+    ORDER BY pos.start_date DESC
+  `);
+  
+  // 3. Recent activity (posts, mentions)
+  const activity = await orangeslice.serp.search(
+    `"${profile[0].first_name} ${profile[0].last_name}" site:linkedin.com`,
+    { tbs: "qdr:m" }
+  );
+  
+  return { profile, positions, activity };
+}
+```
+
+---
+
+## Batch Processing Pattern
+
+For large lists, process in controlled batches:
+
+```typescript
+async function processBatch<T, R>(
+  items: T[],
+  processor: (item: T) => Promise<R>,
+  batchSize = 10
+): Promise<R[]> {
+  const results: R[] = [];
+  
+  for (let i = 0; i < items.length; i += batchSize) {
+    const batch = items.slice(i, i + batchSize);
+    const batchResults = await Promise.all(batch.map(processor));
+    results.push(...batchResults);
+    
+    // Rate limiting is handled by orangeslice, but add delay between batches
+    if (i + batchSize < items.length) {
+      await new Promise(r => setTimeout(r, 1000));
+    }
+  }
+  
+  return results;
+}
+
+// Usage
+const domains = ["stripe.com", "ramp.com", "brex.com", ...];
+const companies = await processBatch(domains, async (domain) => {
+  return orangeslice.b2b.sql(`SELECT * FROM linkedin_company WHERE domain = '${domain}'`);
+});
+```

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "orangeslice",
-  "version": "1.6.0",
-  "description": "Turn any AI agent into a B2B sales research assistant with 1B+ LinkedIn profiles",
+  "version": "1.7.0",
+  "description": "AI agent toolkit: B2B database, SERP, web scraping, browser automation, structured AI output, Apify actors, geocoding",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
   "bin": {

package/docs/B2B_CROSS_TABLE_TEST_FINDINGS.md

@@ -1,255 +0,0 @@
-# B2B Cross-Table Query Test Findings
-
-Comprehensive performance comparison between normalized tables (`linkedin_profile`, `linkedin_company`) and denormalized views (`lkd_profile`, `lkd_company`) for cross-table queries.
-
----
-
-## Executive Summary
-
-| Pattern                            | Normalized | Denormalized | Winner       | Speedup |
-| ---------------------------------- | ---------- | ------------ | ------------ | ------- |
-| **Company ID lookup → employees**  | 48ms       | 279ms        | Normalized   | 5.8x    |
-| **Company name (org) search**      | 274ms      | 8,600ms      | Normalized   | 31x     |
-| **GIN-indexed org ILIKE**          | 430ms      | 29,409ms     | Normalized   | 68x     |
-| **Title ILIKE (common term)**      | 64ms       | 313ms        | Normalized   | 4.9x    |
-| **updated_at filter**              | 4ms        | 14ms         | Normalized   | 3.5x    |
-| **Company ID direct lookup**       | 4ms        | 31ms         | Normalized   | 7.8x    |
-| **Headline (rare term)**           | 2,530ms    | 1,258ms      | Denormalized | 2x      |
-| **Skill array search**             | 216ms      | 169ms        | Denormalized | 1.3x    |
-| **Industry + employee_count**      | 742ms      | 202ms        | Denormalized | 3.7x    |
-| **Headline + company size (JOIN)** | 20,205ms   | 217ms        | Denormalized | 93x     |
-| **Multi-skill + company size**     | 28,173ms   | 1,281ms      | Denormalized | 22x     |
-| **Skill + company industry**       | TIMEOUT    | 3,553ms      | Denormalized | ∞       |
-| **Complex multi-filter + company** | TIMEOUT    | 4,947ms      | Denormalized | ∞       |
-| **AI company + SF location**       | TIMEOUT    | 11,061ms     | Denormalized | ∞       |
-
-**Key Finding**: When combining profile text filters (headline, skills) with company constraints (employee_count, industry), **denormalized JOINs are 20-90x faster** and often the only option that completes within timeout.
-
----
-
-## Critical Pattern: Profile + Company Combined Filters
-
-The most important discovery: **cross-table queries with text filters perform dramatically better with denormalized tables**.
-
-### Normalized Multi-JOIN (Often Fails)
-
-```sql
--- ❌ TIMEOUT or 20+ seconds
-SELECT lp.id, lp.first_name, lp.headline, lc.company_name
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-JOIN linkedin_company lc ON lc.id = pos.linkedin_company_id
-WHERE pos.end_date IS NULL
-  AND lp.headline ILIKE '%engineer%'
-  AND lc.employee_count > 1000
-LIMIT 50
--- Result: 20,205ms
-```
-
-### Denormalized JOIN (Fast)
-
-```sql
--- ✅ 217ms - 93x faster
-SELECT lkd.profile_id, lkd.first_name, lkd.headline, lkdc.name
-FROM lkd_profile lkd
-JOIN lkd_company lkdc ON lkdc.linkedin_company_id = lkd.linkedin_company_id
-WHERE lkd.headline ILIKE '%engineer%'
-  AND lkdc.employee_count > 1000
-LIMIT 50
--- Result: 217ms
-```
-
----
-
-## Test Results by Category
-
-### A. Company-First Queries
-
-| Test | Query                           | Normalized | Denormalized | Winner            |
-| ---- | ------------------------------- | ---------- | ------------ | ----------------- |
-| A1   | Employees at company ID         | **48ms**   | 279ms        | Normalized (5.8x) |
-| A2   | Employees by company name (org) | **274ms**  | 8,600ms      | Normalized (31x)  |
-| A3   | Engineers at large companies    | **96ms**   | 234ms        | Normalized (2.4x) |
-
-**Conclusion**: For company-first queries, normalized tables win due to indexed lookups.
-
-### B. Profile-First Queries
-
-| Test | Query                      | Normalized | Denormalized | Winner              |
-| ---- | -------------------------- | ---------- | ------------ | ------------------- |
-| B1   | Python developers          | 216ms      | **169ms**    | Denormalized (1.3x) |
-| B2   | US Data Scientists         | 644ms      | **557ms**    | Denormalized (1.2x) |
-| B3   | Senior engineers + company | 4,535ms    | **196ms**    | Denormalized (23x)  |
-
-**Conclusion**: Simple profile queries are similar; profile + company queries favor denormalized.
-
-### C. Complex Prospecting Queries
-
-| Test | Query                                     | Normalized  | Denormalized | Winner            |
-| ---- | ----------------------------------------- | ----------- | ------------ | ----------------- |
-| C1   | Decision makers at funded startups        | **1,198ms** | 3,124ms      | Normalized (2.6x) |
-| C2   | AI company employees in SF                | TIMEOUT     | **11,061ms** | Denormalized (∞)  |
-| C3   | Hybrid (normalized profile + lkd_company) | 9,631ms     | -            | -                 |
-
-**Conclusion**: When funding table is used (indexed JOIN), normalized wins. When text filters span tables, denormalized wins.
-
-### D. Company Lookups
-
-| Test | Query                      | Normalized | Denormalized | Winner              |
-| ---- | -------------------------- | ---------- | ------------ | ------------------- |
-| D1   | Company by ID              | **4ms**    | 31ms         | Normalized (7.8x)   |
-| D2   | Industry + employee filter | 742ms      | **202ms**    | Denormalized (3.7x) |
-
-### E. Edge Cases
-
-| Test | Query                 | Normalized | Denormalized | Winner              |
-| ---- | --------------------- | ---------- | ------------ | ------------------- |
-| E1   | Headline (blockchain) | 713ms      | **384ms**    | Denormalized (1.9x) |
-| E2   | Company description   | 144ms      | 152ms        | Tie                 |
-
-### F. Verification Tests
-
-| Test | Query                        | Normalized | Denormalized | Winner              |
-| ---- | ---------------------------- | ---------- | ------------ | ------------------- |
-| F1   | Multi-skill + company size   | 28,173ms   | **1,281ms**  | Denormalized (22x)  |
-| F2   | Country + org (GIN)          | **990ms**  | 4,594ms      | Normalized (4.6x)   |
-| F3   | Title regex + company filter | 434ms      | **227ms**    | Denormalized (1.9x) |
-
-### G. Index Pattern Tests
-
-| Test | Query                     | Normalized | Denormalized | Winner            |
-| ---- | ------------------------- | ---------- | ------------ | ----------------- |
-| G1   | org ILIKE (GIN indexed)   | **430ms**  | 29,409ms     | Normalized (68x)  |
-| G2   | headline ILIKE (no index) | 2,530ms    | **1,258ms**  | Denormalized (2x) |
-| G3   | title ILIKE               | **64ms**   | 313ms        | Normalized (4.9x) |
-| G4   | updated_at filter         | **4ms**    | 14ms         | Normalized (3.5x) |
-
-### H. Cross-Table JOIN Patterns
-
-| Test | Query                     | Normalized | Denormalized | Winner             |
-| ---- | ------------------------- | ---------- | ------------ | ------------------ |
-| H1   | Headline + employee_count | 20,205ms   | **217ms**    | Denormalized (93x) |
-| H2   | Skill + company industry  | TIMEOUT    | **3,553ms**  | Denormalized (∞)   |
-| H3   | Multi-filter + company    | TIMEOUT    | **4,947ms**  | Denormalized (∞)   |
-
----
-
-## Decision Rules for Cross-Table Queries
-
-### Use Normalized (`linkedin_profile` + `linkedin_company` JOINs) When:
-
-1. **Company-first lookup** - Start with company ID, get employees
-2. **GIN-indexed field** - Searching `linkedin_profile.org` (company name)
-3. **Indexed lookups** - `updated_at`, company ID, profile ID
-4. **Title field search** - `linkedin_profile.title` is faster
-5. **Indexed JOIN tables** - `linkedin_crunchbase_funding`, `linkedin_profile_position3` by company
-
-### Use Denormalized (`lkd_profile` JOIN `lkd_company`) When:
-
-1. **Headline + company filter** - 93x faster
-2. **Skill + company constraint** - Normalized times out
-3. **Multi-filter combinations** - 22x faster
-4. **Industry + employee_count** - 3.7x faster
-5. **Text filter spanning profile + company** - Often only option
-
-### Never Use:
-
-1. `lkd_profile.company_name` ILIKE - Use `linkedin_profile.org` (68x faster)
-2. Normalized multi-JOIN with headline filter - Will timeout or be 20s+
-
----
-
-## Recommended Query Patterns
-
-### Pattern 1: Find Employees at Company by Name
-
-```sql
--- ✅ BEST: Use GIN-indexed org field
-SELECT id, first_name, title, headline, org
-FROM linkedin_profile
-WHERE org ILIKE '%Google%'
-LIMIT 50
--- Result: 274ms
-```
-
-### Pattern 2: Find Engineers at Large Companies
-
-```sql
--- ✅ BEST: Denormalized JOIN (93x faster)
-SELECT lkd.profile_id, lkd.first_name, lkd.headline, lkdc.name, lkdc.employee_count
-FROM lkd_profile lkd
-JOIN lkd_company lkdc ON lkdc.linkedin_company_id = lkd.linkedin_company_id
-WHERE lkd.headline ILIKE '%engineer%'
-  AND lkdc.employee_count > 1000
-LIMIT 50
--- Result: 217ms
-```
-
-### Pattern 3: Find People with Skills at Specific Company Types
-
-```sql
--- ✅ BEST: Denormalized (normalized times out)
-SELECT lkd.profile_id, lkd.first_name, lkd.headline, lkdc.name
-FROM lkd_profile lkd
-JOIN lkd_company lkdc ON lkdc.linkedin_company_id = lkd.linkedin_company_id
-WHERE 'Python' = ANY(lkd.skills)
-  AND 'SQL' = ANY(lkd.skills)
-  AND lkdc.employee_count BETWEEN 100 AND 5000
-LIMIT 50
--- Result: 1,281ms (normalized: 28,173ms)
-```
-
-### Pattern 4: Prospecting Query (Profile Criteria + Company Criteria)
-
-```sql
--- ✅ BEST: Denormalized for multi-filter
-SELECT lkd.profile_id, lkd.first_name, lkd.title, lkdc.name, lkdc.employee_count
-FROM lkd_profile lkd
-JOIN lkd_company lkdc ON lkdc.linkedin_company_id = lkd.linkedin_company_id
-WHERE lkd.title ~* '(manager|director|lead)'
-  AND lkdc.employee_count BETWEEN 100 AND 1000
-LIMIT 50
--- Result: 227ms (normalized: 434ms)
-```
-
-### Pattern 5: Decision Makers at Funded Startups
-
-```sql
--- ✅ BEST: Normalized when using indexed funding table
-SELECT DISTINCT lp.id, lp.first_name, lp.title, lc.company_name
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-JOIN linkedin_company lc ON lc.id = pos.linkedin_company_id
-JOIN linkedin_crunchbase_funding cf ON cf.linkedin_company_id = lc.id
-WHERE pos.end_date IS NULL
-  AND lp.title ~* '(CEO|CTO|VP|Director|Head)'
-  AND lc.employee_count BETWEEN 10 AND 500
-LIMIT 50
--- Result: 1,198ms
-```
-
----
-
-## Summary: The Cross-Table Golden Rules
-
-1. **Company name search** → Always use `linkedin_profile.org` (GIN indexed, 68x faster)
-2. **Headline/skill + company constraint** → Always use denormalized JOIN (20-93x faster, normalized often times out)
-3. **Company-first lookups** → Use normalized (5-8x faster)
-4. **Indexed table JOINs (funding, positions)** → Normalized is fine
-5. **Multi-filter profile + company** → Denormalized is the only option that works
-
-### Quick Decision:
-
-```
-Need to search by company name?
-  └─ YES → Use linkedin_profile.org
-
-Need profile text filter (headline/skills) + company constraint?
-  └─ YES → Use lkd_profile JOIN lkd_company
-
-Need company ID lookup or indexed JOIN?
-  └─ YES → Use normalized tables
-
-Default for prospecting queries:
-  └─ Use lkd_profile JOIN lkd_company
-```