orangeslice 1.6.0 → 1.7.0

package/docs/apify.md

@@ -0,0 +1,133 @@
+# Apify Actors
+
+Pre-built web scrapers for social media, Google Maps, e-commerce, and more.
+
+```typescript
+import { orangeslice } from 'orangeslice';
+
+// Run an actor
+const results = await orangeslice.apify.run("username/actor-name", { input: "params" });
+
+// Search for actors
+const { actors } = await orangeslice.apify.search("linkedin scraper");
+
+// Get actor input schema (what params it accepts)
+const schema = await orangeslice.apify.getInputSchema("apify/web-scraper");
+```
+
+---
+
+## Workflow
+
+1. **Search** for an actor that does what you need
+2. **Get input schema** to understand required params
+3. **Run** the actor with your inputs
+4. Results are returned when the actor completes
+
+---
+
+## Searching for Actors
+
+```typescript
+const { actors, total } = await orangeslice.apify.search("google maps reviews", 10);
+
+// actors = [{
+//   actorId: "compass/crawler-google-places",
+//   title: "Google Maps Scraper",
+//   description: "Scrape Google Maps...",
+//   stats: { totalRuns: 1000000 },
+//   pricing: { ... }
+// }, ...]
+```
+
+---
+
+## Getting Input Schema
+
+Before running, check what params the actor needs:
+
+```typescript
+const schema = await orangeslice.apify.getInputSchema("compass/crawler-google-places");
+
+console.log(schema.inputProperties);
+// {
+//   searchStringsArray: { type: "array", description: "Search queries" },
+//   maxReviews: { type: "integer", description: "Max reviews per place" },
+//   language: { type: "string", default: "en" },
+//   ...
+// }
+```
+
+---
+
+## Running Actors
+
+```typescript
+// Google Maps reviews
+const reviews = await orangeslice.apify.run("compass/crawler-google-places", {
+  searchStringsArray: ["restaurants in NYC"],
+  maxReviews: 20,
+  language: "en"
+});
+
+// With dataset params (limit results)
+const results = await orangeslice.apify.run("apify/web-scraper", 
+  { startUrls: [{ url: "https://example.com" }] },
+  { limit: 100 }  // Only return first 100 items
+);
+```
+
+---
+
+## Popular Actors
+
+| Use Case | Actor | Example Input |
+|----------|-------|---------------|
+| Google Maps | `compass/crawler-google-places` | `{ searchStringsArray: ["cafes SF"] }` |
+| Google Search | `apify/google-search-scraper` | `{ queries: "site:linkedin.com CEO" }` |
+| Instagram | `apify/instagram-scraper` | `{ directUrls: ["https://instagram.com/user"] }` |
+| Twitter/X | `apidojo/tweet-scraper` | `{ searchTerms: ["#startup"] }` |
+| LinkedIn | `anchor/linkedin-profile-scraper` | `{ profileUrls: [...] }` |
+| YouTube | `streamers/youtube-scraper` | `{ searchKeywords: ["tech reviews"] }` |
+| TikTok | `clockworks/tiktok-scraper` | `{ profiles: ["@username"] }` |
+| Amazon | `junglee/amazon-scraper` | `{ keyword: "laptop stand" }` |
+| Yelp | `yin/yelp-scraper` | `{ searchTerms: "plumber", location: "NYC" }` |
+
+---
+
+## Dataset Params
+
+Control the results returned:
+
+```typescript
+await orangeslice.apify.run(actor, input, {
+  limit: 100,           // Max items to return
+  offset: 0,            // Skip first N items
+  clean: true,          // Remove empty fields
+  fields: ["name", "url"], // Only these fields
+  unwind: "reviews"     // Flatten nested array
+});
+```
+
+---
+
+## Timeouts
+
+- Actors run asynchronously and are polled for completion
+- Max wait: 5 minutes
+- Large scrapes may timeout — use smaller batches
+
+---
+
+## Response Format
+
+Returns the dataset items directly as an array:
+
+```typescript
+const results = await orangeslice.apify.run(...);
+// results = [
+//   { name: "...", address: "...", rating: 4.5 },
+//   { name: "...", address: "...", rating: 4.2 },
+//   ...
+// ]
+```

package/docs/b2b.md

@@ -0,0 +1,178 @@
+# B2B Database
+
+**Scale:** 1.15B profiles, 2.6B positions, millions of companies. Naive queries timeout.
+
+```typescript
+import { orangeslice } from 'orangeslice';
+
+// Returns rows directly
+const companies = await orangeslice.b2b.sql("SELECT * FROM linkedin_company WHERE domain = 'stripe.com'");
+
+// Returns { rows, rowCount, duration_ms }
+const result = await orangeslice.b2b.query("SELECT * FROM linkedin_company WHERE domain = 'stripe.com'");
+```
+
+---
+
+## Fast Lookups (Indexed)
+
+```sql
+-- Company by domain (FAST)
+SELECT * FROM linkedin_company WHERE domain = 'stripe.com';
+
+-- Company by universal_name (FAST)
+SELECT * FROM linkedin_company WHERE universal_name = 'stripe';
+
+-- Employees at company (FAST - filter by company ID)
+SELECT lp.first_name, lp.last_name, pos.title
+FROM linkedin_profile lp
+JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
+WHERE pos.linkedin_company_id = 2135371
+  AND pos.end_date IS NULL
+LIMIT 50;
+```
+
+---
+
+## Slow Queries (Will Timeout)
+
+```sql
+-- ❌ Text search on names (no index)
+WHERE company_name ILIKE '%stripe%'
+
+-- ❌ Headline search without company filter
+WHERE headline ILIKE '%sales%'
+
+-- ❌ COUNT on huge companies
+SELECT COUNT(*) FROM ... WHERE linkedin_company_id = 1586
+```
+
+---
+
+## Indexed Columns
+
+| Table | Indexed Columns |
+|-------|-----------------|
+| `linkedin_company` | `id`, `universal_name`, `domain` |
+| `linkedin_profile` | `id`, `linkedin_user_id` |
+| `linkedin_profile_position3` | `linkedin_profile_id`, `linkedin_company_id` |
+| `linkedin_job` | `linkedin_company_id`, `title_id` |
+| `linkedin_crunchbase_funding` | `linkedin_company_id` |
+
+---
+
+## Company Size Performance
+
+| Company Size | Simple Query | Aggregations |
+|--------------|--------------|--------------|
+| Small (<1K) | 4-20ms | 5-50ms |
+| Medium (1K-10K) | 10-30ms | 100-500ms |
+| Large (10K-100K) | 10-40ms | 1-15s |
+| Massive (100K+) | 15-65ms | **TIMEOUT** |
+
+**For Amazon/Google/Meta:** Only use simple `LIMIT` queries. No aggregations.
+
+---
+
+## Common Company IDs
+
+| Company | ID | Employees |
+|---------|----------|-----------|
+| Amazon | 1586 | 770K |
+| Google | 1441 | 330K |
+| Meta | 10667 | 70K |
+| Stripe | 2135371 | ~9K |
+| OpenAI | 11130470 | ~7K |
+| Ramp | 1406226 | ~3.5K |
+
+---
+
+## Title Search Patterns
+
+| Role | ILIKE Pattern |
+|------|---------------|
+| C-Suite | `ceo%`, `cto%`, `cfo%`, `%chief%` |
+| VPs | `%vp %`, `%vice president%` |
+| Directors | `%director%`, `%head of%` |
+| Sales | `%account exec%`, `%sales rep%`, `%ae %` |
+| SDRs | `%sales development%`, `%sdr%`, `%bdr%` |
+| Engineering | `%engineer%`, `%developer%` |
+| Recruiters | `%recruit%`, `%talent%`, `%sourcer%` |
+| Legal | `%lawyer%`, `%attorney%`, `%counsel%` |
+
+---
+
+## Check If Company Is Hiring
+
+```sql
+-- Filter for companies with active job postings
+EXISTS (
+  SELECT 1 FROM linkedin_job j 
+  WHERE j.linkedin_company_id = lc.id
+    AND j.closed_since IS NULL
+    AND (j.valid_until IS NULL OR j.valid_until > NOW())
+    AND j.posted_date >= CURRENT_DATE - INTERVAL '90 days'
+)
+```
+
+---
+
+## Query Strategy
+
+**Timeout?** Immediately fall back to SERP:
+```
+site:linkedin.com/company [query]
+```
+
+**Complex criteria?** Decompose:
+1. Simple indexed query → get IDs
+2. Enrich with additional data
+3. Filter/qualify results
+
+---
+
+## Key Tables
+
+### linkedin_company
+```sql
+SELECT id, company_name, universal_name, domain, 
+       employee_count, industry, locality, country,
+       description, specialties, founded_year
+FROM linkedin_company
+WHERE domain = 'example.com';
+```
+
+### linkedin_profile
+```sql
+SELECT id, first_name, last_name, headline, 
+       locality, country, summary, linkedin_user_id
+FROM linkedin_profile
+WHERE id = 12345;
+```
+
+### linkedin_profile_position3
+```sql
+SELECT linkedin_profile_id, linkedin_company_id,
+       title, start_date, end_date, description
+FROM linkedin_profile_position3
+WHERE linkedin_company_id = 2135371
+  AND end_date IS NULL;  -- Current employees only
+```
+
+### linkedin_job
+```sql
+SELECT id, linkedin_company_id, title, 
+       locality, posted_date, closed_since
+FROM linkedin_job
+WHERE linkedin_company_id = 2135371
+  AND closed_since IS NULL;
+```
+
+### linkedin_crunchbase_funding
+```sql
+SELECT linkedin_company_id, funding_type, 
+       money_raised, announced_date, investor_names
+FROM linkedin_crunchbase_funding
+WHERE linkedin_company_id = 2135371
+ORDER BY announced_date DESC;
+```

package/docs/browser.md

@@ -0,0 +1,173 @@
+# Browser Automation
+
+Execute Playwright code with a `page` object in scope. For dynamic/JS-heavy pages.
+
+```typescript
+import { orangeslice } from 'orangeslice';
+
+const response = await orangeslice.browser.execute(`
+  await page.goto("https://example.com", { waitUntil: 'domcontentloaded' });
+  return await page.evaluate(() => document.title);
+`);
+// { success: true, result: "Example Domain" }
+```
+
+---
+
+## When to Use
+
+| Use `firecrawl` | Use `browser` |
+|-----------------|---------------|
+| Static HTML | JavaScript-rendered content |
+| Simple content extraction | Complex interactions |
+| Fast, one-shot scraping | Login flows, clicking |
+| Getting social URLs | Bot-protected sites |
+
+---
+
+## Basic Patterns
+
+### Extract List Items
+
+```typescript
+const response = await orangeslice.browser.execute(`
+  await page.goto("https://example.com/products", { waitUntil: 'domcontentloaded' });
+  return await page.evaluate(() => {
+    return [...document.querySelectorAll('.product')].map(el => ({
+      name: el.querySelector('h2')?.textContent?.trim(),
+      price: el.querySelector('.price')?.textContent?.trim(),
+      url: el.querySelector('a')?.href
+    }));
+  });
+`);
+```
+
+### Click and Wait
+
+```typescript
+const response = await orangeslice.browser.execute(`
+  await page.goto(url, { waitUntil: 'domcontentloaded' });
+  await page.click('button.load-more');
+  await page.waitForSelector('.new-content');
+  return await page.evaluate(() => document.body.innerText);
+`);
+```
+
+### Fill Form
+
+```typescript
+const response = await orangeslice.browser.execute(`
+  await page.goto(url, { waitUntil: 'domcontentloaded' });
+  await page.fill('input[name="search"]', 'query');
+  await page.click('button[type="submit"]');
+  await page.waitForSelector('.results');
+  return await page.evaluate(() => 
+    [...document.querySelectorAll('.result')].map(e => e.textContent)
+  );
+`);
+```
+
+---
+
+## Workflow: Discover → Extract
+
+### Step 1: Analyze Page Structure
+
+```typescript
+const response = await orangeslice.browser.execute(`
+  await page.goto(url, { waitUntil: 'domcontentloaded' });
+  return await page._snapshotForAI();
+`);
+// Returns page structure for selector discovery
+```
+
+### Step 2: Extract with Discovered Selectors
+
+```typescript
+const response = await orangeslice.browser.execute(`
+  await page.goto(url, { waitUntil: 'domcontentloaded' });
+  return await page.evaluate(() => {
+    return [...document.querySelectorAll('.discovered-selector')].map(e => ({
+      title: e.querySelector('h2')?.textContent?.trim(),
+      description: e.querySelector('p')?.textContent?.trim()
+    }));
+  });
+`);
+```
+
+---
+
+## Bot Protection: Single Session
+
+For bot-protected sites, do all navigation in ONE session:
+
+```typescript
+const response = await orangeslice.browser.execute(`
+  // 1. Navigate to entry page (passes bot check once)
+  await page.goto(entryUrl, { waitUntil: 'domcontentloaded' });
+  
+  // 2. Get all URLs to visit
+  const urls = await page.evaluate(() => 
+    [...document.querySelectorAll('a.link')].map(a => a.href)
+  );
+  
+  // 3. Visit each IN THE SAME SESSION
+  const results = [];
+  for (const url of urls.slice(0, 10)) {
+    await page.goto(url, { waitUntil: 'domcontentloaded' });
+    const data = await page.evaluate(() => ({
+      title: document.querySelector('h1')?.textContent?.trim(),
+      content: document.querySelector('.main')?.textContent?.trim()
+    }));
+    results.push(data);
+  }
+  return results;
+`);
+```
+
+---
+
+## Options
+
+```typescript
+await orangeslice.browser.execute(code, {
+  timeout_sec: 60,              // Execution timeout (default: 30)
+  acquire_timeout_seconds: 30   // Browser acquisition timeout
+});
+```
+
+---
+
+## Rules
+
+1. **Always use `{ waitUntil: 'domcontentloaded' }`** — Prevents hanging on slow resources
+2. **Check `response.success`** — Don't just destructure `result`
+3. **Analyze before extracting** — Use `_snapshotForAI()` to find selectors
+4. **Return objects, not HTML** — Use `page.evaluate()` for structured data
+5. **3 minute hard limit** — Plan multi-page scrapes accordingly
+
+---
+
+## Response Format
+
+```typescript
+interface BrowserResponse {
+  success: boolean;
+  result?: any;           // Your return value
+  error?: string;         // Error message if failed
+  console_logs?: string[];
+  browser_live_view_url?: string;  // Debug URL
+}
+```
+
+---
+
+## Convenience Methods
+
+```typescript
+// Get page snapshot for analysis
+const snapshot = await orangeslice.browser.snapshot(url);
+
+// Get just the text content
+const text = await orangeslice.browser.text(url);
+```

package/docs/serp.md

@@ -0,0 +1,167 @@
+# Google SERP
+
+Google search with advanced dorking. Fast but **requires verification**.
+
+```typescript
+import { orangeslice } from 'orangeslice';
+
+// Basic search
+const { results } = await orangeslice.serp.search("AI CRM software");
+
+// With options
+const { results } = await orangeslice.serp.search("Stripe hiring", {
+  tbs: "qdr:m",           // Past month
+  page: 0,                // First page
+  advance_search: true    // Include snippets
+});
+
+// Convenience: just organic results
+const results = await orangeslice.serp.organic("query");
+```
+
+---
+
+## Dorking Cheatsheet
+
+### Core Operators
+
+| Operator | Example | Effect |
+|----------|---------|--------|
+| `"..."` | `"exact phrase"` | Match exact text |
+| `OR` | `CEO OR Founder` | Match either term |
+| `-` | `startup -jobs` | Exclude term |
+| `site:` | `site:linkedin.com` | Restrict to domain |
+| `inurl:` | `inurl:status` | URL must contain |
+| `intitle:` | `intitle:"series A"` | Title must contain |
+
+### Platform Dorks
+
+| Goal | Dork |
+|------|------|
+| LinkedIn profiles | `site:linkedin.com/in "query"` |
+| LinkedIn companies | `site:linkedin.com/company "query"` |
+| LinkedIn posts | `site:linkedin.com/posts "query"` |
+| Twitter/X posts | `site:x.com inurl:status "query"` |
+| Twitter/X profiles | `site:x.com -inurl:status "query"` |
+| Reddit threads | `site:reddit.com "query"` |
+| Crunchbase | `site:crunchbase.com/organization "query"` |
+| GitHub | `site:github.com "query"` |
+
+---
+
+## B2B Prospecting Dorks
+
+```
+# Find employees at company
+"Stripe" site:linkedin.com/in
+
+# Find leadership
+"Acme Corp" CEO OR Founder OR "Co-founder" site:linkedin.com/in
+
+# Find by title
+"VP Sales" "Series A" site:linkedin.com/in
+
+# Find company pages by criteria
+"YC W24" site:linkedin.com/company
+"Series B" fintech site:linkedin.com/company
+
+# Find companies by product category
+"AI CRM" OR "AI-powered CRM" site:linkedin.com/company
+```
+
+---
+
+## Time Filters
+
+| Value | Period |
+|-------|--------|
+| `qdr:h` | Past hour |
+| `qdr:d` | Past 24h |
+| `qdr:w` | Past week |
+| `qdr:m` | Past month |
+| `qdr:y` | Past year |
+
+```typescript
+// Recent news only
+orangeslice.serp.search("Stripe funding", { tbs: "qdr:m" });
+```
+
+---
+
+## Query Permutation Strategy
+
+SERP is cheap. Run 10-30 variations in parallel:
+
+| Dimension | Variations |
+|-----------|------------|
+| Name | Full name, initials, nicknames |
+| Company | Full name, abbreviation, domain |
+| Title | CEO/Founder/Chief, VP/Director, formal/informal |
+| Location | City, metro area, state |
+
+```typescript
+const queries = [
+  `"John Smith" "Acme" site:linkedin.com/in`,
+  `"J. Smith" Acme site:linkedin.com/in`,
+  `"John Smith" CEO site:linkedin.com/in`,
+];
+const results = await Promise.all(queries.map(q => orangeslice.serp.search(q)));
+```
+
+---
+
+## Verification Required
+
+**Dorking returns false positives.** Always verify:
+
+1. **Enrich via B2B database** — Get actual company/person data
+2. **Scrape website** — Check product page, about page
+3. **AI classification** — "Based on [data], does this match [criteria]?"
+
+### Example: Find AI Companies
+
+```typescript
+// 1. Dork for candidates
+const { results } = await orangeslice.serp.search(
+  `"AI-powered" site:linkedin.com/company`
+);
+
+// 2. Extract LinkedIn URLs
+const linkedinUrls = results
+  .map(r => r.link)
+  .filter(url => url.includes('linkedin.com/company'));
+
+// 3. Enrich via B2B database
+for (const url of linkedinUrls) {
+  const universalName = url.split('/company/')[1]?.split('/')[0];
+  if (universalName) {
+    const company = await orangeslice.b2b.sql(`
+      SELECT * FROM linkedin_company 
+      WHERE universal_name = '${universalName}'
+    `);
+    // 4. Verify with actual description
+    if (company[0]?.description?.toLowerCase().includes('artificial intelligence')) {
+      // Confirmed AI company
+    }
+  }
+}
+```
+
+---
+
+## Response Format
+
+```typescript
+interface SerpResult {
+  title: string;
+  link: string;
+  snippet?: string;
+  displayed_link?: string;
+  position?: number;
+}
+
+interface SerpResponse {
+  results: SerpResult[];
+  related_questions?: Array<{ question: string; snippet: string }>;
+}
+```