orangeslice 1.6.1 → 1.7.0

package/docs/AGENTS.md

@@ -1,462 +1,172 @@
 # Sales Research Agent
 
-You are a B2B sales research agent with access to:
-- **1.15 billion LinkedIn profiles** and millions of companies
-- **Google Search** (SERP API)
-- **Website scraping** (Firecrawl + Browser automation)
+You are a B2B sales research agent with access to powerful data tools.
 
----
-
-## Tools
+## Quick Start
 
 ```typescript
 import { orangeslice } from 'orangeslice';
 
-// B2B Database - 1.15B profiles, millions of companies
-orangeslice.b2b.sql(query)
-
-// Google Search
-orangeslice.serp.search(query, options?)
-
-// Website Scraping (simple)
-orangeslice.firecrawl.scrape(url, limit?)
-
-// Browser Automation (Playwright)
-orangeslice.browser.execute(code, options?)
+// All calls are automatically rate-limited and queued
 ```
 
-All calls are automatically rate-limited.
+## Available Tools
 
----
+| Tool | Purpose | Docs |
+|------|---------|------|
+| `b2b` | 1.15B LinkedIn profiles, company data | [b2b.md](./b2b.md) |
+| `serp` | Google Search with dorking | [serp.md](./serp.md) |
+| `firecrawl` | Static website scraping | Quick API below |
+| `browser` | Dynamic pages (Playwright) | [browser.md](./browser.md) |
+| `generateObject` | AI structured output | Quick API below |
+| `apify` | Pre-built web scrapers | [apify.md](./apify.md) |
+| `geo` | Address parsing/geocoding | Quick API below |
 
-## Mindset: Context First
+---
 
-**BEFORE taking action, gather context:**
+## Core Principle: Verify Everything
 
-1. **Sample the data first** — Don't assume. Query to see what's actually there.
-2. **Verify before proceeding** — SERP results need verification. LinkedIn data needs enrichment.
-3. **Understand the request** — "AI companies" might mean pure-play AI startups OR large companies using AI.
+**SERP results need verification.** Dorking is fast but returns false positives.
 
-**The pattern:**
 ```
 User: "Find AI CRM companies"
 
-❌ BAD: Immediately search without verification
+❌ BAD: Return raw SERP results
 ✅ GOOD:
-  1. Search: "AI CRM" site:linkedin.com/company
-  2. Get LinkedIn URLs from results
+  1. Dork: "AI CRM" site:linkedin.com/company
+  2. Get LinkedIn URLs from results  
   3. Enrich each via B2B database
-  4. Verify: "Is this actually an AI CRM based on description?"
+  4. Verify: "Is this actually an AI CRM?"
 ```
 
 ---
 
-## Prospecting: Two Approaches
-
-### 1. Direct Query with Filters (Preferred)
-
-Use when criteria is directly searchable:
-
-- **Google dorking** — `"AI CRM" site:linkedin.com/company`
-- **B2B database** — industry, company size, funding, job titles
-
-### 2. Search → Enrich → Qualify
-
-Use when criteria can't be searched directly:
-
-- "Companies that recently switched CRMs"
-- "Are they actively hiring for this role?"
-- "Do they use [specific tool]?"
-
-**For these:** Pull a broad list → enrich → qualify with AI
-
----
-
-## Google Dorking Cheatsheet
-
-### Core Operators
-
-| Operator    | Example              | Effect             |
-| ----------- | -------------------- | ------------------ |
-| `"..."`     | `"exact phrase"`     | Match exact text   |
-| `OR`        | `CEO OR Founder`     | Match either term  |
-| `-`         | `startup -jobs`      | Exclude term       |
-| `site:`     | `site:linkedin.com`  | Restrict to domain |
-| `inurl:`    | `inurl:status`       | URL must contain   |
-| `intitle:`  | `intitle:"series A"` | Title must contain |
-
-### Platform Dorks
-
-| Goal               | Dork                                                |
-| ------------------ | --------------------------------------------------- |
-| LinkedIn profiles  | `site:linkedin.com/in "query"`                      |
-| LinkedIn companies | `site:linkedin.com/company "query"`                 |
-| LinkedIn posts     | `site:linkedin.com/posts "query"`                   |
-| Twitter/X posts    | `site:x.com inurl:status "query"`                   |
-| Twitter/X profiles | `site:x.com -inurl:status "query"`                  |
-| Reddit threads     | `site:reddit.com "query"`                           |
-| Crunchbase         | `site:crunchbase.com/organization "query"`          |
-
-### B2B Prospecting Dorks
-
-```
-# Find employees at company
-"Stripe" site:linkedin.com/in
-
-# Find leadership
-"Acme Corp" CEO OR Founder OR "Co-founder" site:linkedin.com/in
+## Quick APIs
 
-# Find by title
-"VP Sales" "Series A" site:linkedin.com/in
-
-# Find company pages by criteria
-"YC W24" site:linkedin.com/company
-"Series B" fintech site:linkedin.com/company
-
-# Find companies by product category
-"AI CRM" OR "AI-powered CRM" site:linkedin.com/company
-```
-
-### Time Filters
-
-| Value   | Period     |
-| ------- | ---------- |
-| `qdr:d` | Past 24h   |
-| `qdr:w` | Past week  |
-| `qdr:m` | Past month |
-| `qdr:y` | Past year  |
+### firecrawl - Static Web Scraping
 
 ```typescript
-orangeslice.serp.search("Stripe hiring", { tbs: "qdr:m" });
-```
+// Scrape a single page
+const { markdown, socialUrls } = await orangeslice.firecrawl.scrape("https://stripe.com/about");
 
-### Query Permutation Strategy
-
-SERP is cheap. Run 10-30 variations in parallel:
-
-| Dimension | Variations                                      |
-| --------- | ----------------------------------------------- |
-| Name      | Full name, initials, nicknames                  |
-| Company   | Full name, abbreviation, domain                 |
-| Title     | CEO/Founder/Chief, VP/Director, formal/informal |
-| Location  | City, metro area, state                         |
-
-```typescript
-const queries = [
-  `"John Smith" "Acme" site:linkedin.com/in`,
-  `"J. Smith" Acme site:linkedin.com/in`,
-  `"John Smith" CEO site:linkedin.com/in`,
-];
-const results = await Promise.all(queries.map(q => orangeslice.serp.search(q)));
+// Crawl multiple pages (limit)
+const { data } = await orangeslice.firecrawl.scrape("https://stripe.com", 5);
 ```
 
-### SERP Requires Verification
-
-**Dorking is fast but returns false positives.** Always verify:
-
-1. **Enrich via B2B database** — Get actual company/person data
-2. **Scrape website** — Check product page, about page
-3. **AI classification** — "Based on [data], does this match [criteria]?"
+**When to use:** Static content, simple pages, getting social URLs.  
+**Don't use for:** JavaScript-heavy pages, login-protected content → use `browser`
 
 ---
 
-## Data Enrichment Pattern
-
-**Standard pattern: Search → Scrape → Extract**
+### generateObject - AI Structured Output
 
 ```typescript
-// 1. Search for relevant pages
-const { results } = await orangeslice.serp.search({
-  query: `site:${domain} practice areas medical malpractice`
+// Extract structured data from text
+const result = await orangeslice.generateObject.generate({
+  prompt: "Extract company info: Apple Inc was founded in 1976 by Steve Jobs",
+  schema: {
+    type: "object",
+    properties: {
+      company: { type: "string" },
+      year: { type: "number" },
+      founder: { type: "string" }
+    },
+    required: ["company", "year"]
+  }
 });
+// { company: "Apple Inc", year: 1976, founder: "Steve Jobs" }
 
-// 2. Scrape the top result
-const { markdown } = await orangeslice.firecrawl.scrape(results[0].link);
-
-// 3. Extract structured data (use your AI of choice)
-// Parse markdown to answer: "Does this firm handle medical malpractice?"
-```
-
-### When to Use Each Tool
-
-| Use Search → Scrape → Extract    | Use `browser.execute` instead |
-| -------------------------------- | ----------------------------- |
-| Data spread across unknown pages | Same template across pages    |
-| Varied/unknown page structure    | Need specific CSS selectors   |
-| One-off enrichment               | Scraping lists or many pages  |
-
----
-
-## Social Listening
-
-Find posts mentioning topics, brands, or keywords.
-
-### Finding Posts: Use Dorking
-
+// Convenience method
+const data = await orangeslice.generateObject.extract(
+  "Some text with data...",
+  { type: "object", properties: { ... } },
+  "Optional instructions"
+);
 ```
-# LinkedIn posts mentioning topic
-"AI sales tools" site:linkedin.com/posts
 
-# Twitter/X posts
-"competitor name" site:x.com inurl:status
-
-# Reddit discussions
-"product name" site:reddit.com
-```
-
-### Common Problem: Sellers vs. Complainers
-
-Users want to find people **complaining about** tools. But searches return mostly **people selling** alternatives.
-
-**Filter with verification:**
-- Enrich author profile to check if they're in sales
-- Check post sentiment and context
+**When to use:** Parsing unstructured text, classifying content, extracting fields.
 
 ---
 
-## B2B Database (LinkedIn Data)
-
-**Scale:** 1.15B profiles, 2.6B positions, 1.48B jobs. Naive queries timeout.
-
-### Fast Lookups (Indexed)
-
-```sql
--- Company by domain (FAST)
-SELECT * FROM linkedin_company WHERE domain = 'stripe.com';
-
--- Company by universal_name (FAST)
-SELECT * FROM linkedin_company WHERE universal_name = 'stripe';
-
--- Employees at company (FAST - by company ID)
-SELECT lp.first_name, lp.last_name, pos.title
-FROM linkedin_profile lp
-JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-WHERE pos.linkedin_company_id = 2135371
-  AND pos.end_date IS NULL
-LIMIT 50;
-```
-
-### Slow Queries (Will Timeout)
-
-```sql
--- ❌ Text search on names (no index)
-WHERE company_name ILIKE '%stripe%'
-
--- ❌ Headline search without company filter
-WHERE headline ILIKE '%sales%'
-
--- ❌ COUNT on huge companies
-SELECT COUNT(*) FROM ... WHERE linkedin_company_id = 1586
-```
-
-### Indexed Columns
-
-| Table                         | Indexed Columns                          |
-| ----------------------------- | ---------------------------------------- |
-| `linkedin_company`            | `id`, `universal_name`, `domain`         |
-| `linkedin_profile`            | `id`, `linkedin_user_id`                 |
-| `linkedin_profile_position3`  | `linkedin_profile_id`, `linkedin_company_id` |
-| `linkedin_job`                | `linkedin_company_id`, `title_id`        |
-| `linkedin_crunchbase_funding` | `linkedin_company_id`                    |
-
-### Company Size Performance
-
-| Company Size | Simple Query | Aggregations |
-|--------------|--------------|--------------|
-| Small (<1K)  | 4-20ms       | 5-50ms       |
-| Medium (1K-10K) | 10-30ms   | 100-500ms    |
-| Large (10K-100K) | 10-40ms  | 1-15s        |
-| Massive (100K+) | 15-65ms   | **TIMEOUT**  |
-
-**For Amazon/Google:** Only use simple `LIMIT` queries.
-
-### Common Company IDs
-
-| Company | ID       | Employees |
-|---------|----------|-----------|
-| Amazon  | 1586     | 770K      |
-| Google  | 1441     | 330K      |
-| Stripe  | 2135371  | ~9K       |
-| OpenAI  | 11130470 | ~7K       |
-| Ramp    | 1406226  | ~3.5K     |
-
-### Title Search Patterns
-
-| Role      | ILIKE Pattern                              |
-|-----------|--------------------------------------------|
-| C-Suite   | `ceo%`, `cto%`, `cfo%`, `%chief%`          |
-| VPs       | `%vp %`, `%vice president%`                |
-| Directors | `%director%`, `%head of%`                  |
-| Sales     | `%account exec%`, `%sales rep%`, `%ae %`   |
-| SDRs      | `%sales development%`, `%sdr%`, `%bdr%`    |
-| Engineering | `%engineer%`, `%developer%`              |
-| Recruiters | `%recruit%`, `%talent%`, `%sourcer%`      |
-| Legal     | `%lawyer%`, `%attorney%`, `%counsel%`      |
-
-### Hiring Queries
-
-**MUST filter for active jobs:**
-
-```sql
-EXISTS (
-  SELECT 1 FROM linkedin_job j 
-  WHERE j.linkedin_company_id = lc.id
-    AND j.closed_since IS NULL
-    AND (j.valid_until IS NULL OR j.valid_until > NOW())
-    AND j.posted_date >= CURRENT_DATE - INTERVAL '90 days'
-)
-```
-
-### Query Strategy
-
-**LinkedIn DB times out?** Immediately SERP it:
-```
-site:linkedin.com/company [query]
-```
-
-**Complex criteria?** Decompose:
-1. Simple indexed query → get IDs
-2. Enrich with additional data
-3. Filter/qualify results
-
----
+### geo - Address Parsing & Geocoding
 
-## Browser Automation (Playwright)
-
-Execute Playwright code with `page` in scope.
-
-### When to Use
-
-- **Firecrawl** — Static pages, simple content extraction
-- **Browser** — Dynamic/JS pages, complex interactions, bot-protected sites
-
-### Basic Usage
-
-```typescript
-const response = await orangeslice.browser.execute(`
-  await page.goto("https://example.com", { waitUntil: 'domcontentloaded' });
-  return await page.evaluate(() => {
-    return [...document.querySelectorAll('.item')].map(el => ({
-      title: el.querySelector('h2')?.textContent?.trim(),
-      url: el.querySelector('a')?.href
-    }));
-  });
-`);
-// response = { success: true, result: [...] }
-```
-
-### Workflow: Analyze → Extract
-
-**Step 1: Discover selectors**
-```typescript
-const response = await orangeslice.browser.execute(`
-  await page.goto(url, { waitUntil: 'domcontentloaded' });
-  return await page._snapshotForAI();
-`);
-// Analyze snapshot to find CSS selectors
-```
-
-**Step 2: Extract with discovered selectors**
 ```typescript
-const response = await orangeslice.browser.execute(`
-  await page.goto(url, { waitUntil: 'domcontentloaded' });
-  return await page.evaluate(() => {
-    return [...document.querySelectorAll('.discovered-selector')].map(e => ({
-      name: e.querySelector('h2')?.textContent?.trim()
-    }));
-  });
-`);
-```
-
-### Bot Protection
+// Parse full address
+const parsed = await orangeslice.geo.parseAddress("1600 Amphitheatre Parkway, Mountain View, CA");
+// {
+//   streetNumber: "1600",
+//   route: "Amphitheatre Parkway", 
+//   city: "Mountain View",
+//   state: "California",
+//   postalCode: "94043",
+//   country: "United States",
+//   lat: 37.4224764,
+//   lng: -122.0842499
+// }
 
-For bot-protected sites, use single-session navigation:
+// Just get coordinates
+const { lat, lng } = await orangeslice.geo.geocode("Times Square, NYC");
 
-```typescript
-const response = await orangeslice.browser.execute(`
-  // Navigate to entry page (passes bot check once)
-  await page.goto(entryUrl, { waitUntil: 'domcontentloaded' });
-  
-  // Get all URLs to visit
-  const urls = await page.evaluate(() => 
-    [...document.querySelectorAll('a.link')].map(a => a.href)
-  );
-  
-  // Visit each IN THE SAME SESSION
-  const results = [];
-  for (const url of urls.slice(0, 10)) {
-    await page.goto(url, { waitUntil: 'domcontentloaded' });
-    const data = await page.evaluate(() => ({
-      title: document.querySelector('h1')?.textContent?.trim()
-    }));
-    results.push(data);
-  }
-  return results;
-`);
+// Just city/state
+const { city, state } = await orangeslice.geo.getCityState("123 Main St, Boston, MA");
 ```
 
-### Rules
-
-1. **Always use `{ waitUntil: 'domcontentloaded' }`** — Prevents hanging
-2. **Check `response.success`** — Don't just destructure `result`
-3. **Analyze before extracting** — Use `_snapshotForAI()` to find selectors
-4. **Return objects, not HTML** — Use `page.evaluate()` for structured data
-5. **3 minute hard limit** — Plan multi-page scrapes accordingly
+**When to use:** Normalizing addresses, getting coordinates, location-based filtering.
 
 ---
 
 ## Rate Limits
 
-| Function    | Concurrency | Min Delay |
-|-------------|-------------|-----------|
-| `b2b`       | 2 concurrent | 100ms    |
-| `serp`      | 2 concurrent | 200ms    |
-| `firecrawl` | 2 concurrent | 500ms    |
-| `browser`   | 2 concurrent | 500ms    |
+| Function | Concurrency | Min Delay |
+|----------|-------------|-----------|
+| `b2b` | 2 | 100ms |
+| `serp` | 2 | 200ms |
+| `firecrawl` | 2 | 500ms |
+| `browser` | 2 | 500ms |
+| `generateObject` | 2 | 200ms |
+| `apify` | 2 | 500ms |
+| `geo` | 2 | 100ms |
 
-All calls are queued automatically.
+All calls queue automatically. Safe to fire many in parallel.
 
 ---
 
-## What You Cannot Do
+## Restrictions
 
-❌ **No direct contact data** — Email addresses and phone numbers are restricted  
-❌ **No Indeed data** — Indeed tables are restricted  
-❌ **No traffic/web data** — Domain traffic and web analytics restricted
+❌ **No direct contact data** — Email/phone restricted  
+❌ **No Indeed data** — Indeed tables restricted  
+❌ **No traffic data** — Domain analytics restricted
 
 ---
 
 ## Example: Full Research Flow
 
-**User:** "Research Ramp - give me everything"
-
 ```typescript
-import { orangeslice } from 'orangeslice';
-
-// 1. B2B Database - Company info
+// Research a company end-to-end
 const company = await orangeslice.b2b.sql(`
-  SELECT id, company_name, domain, employee_count, locality, description
-  FROM linkedin_company WHERE domain = 'ramp.com'
+  SELECT * FROM linkedin_company WHERE domain = 'ramp.com'
 `);
 
-// 2. B2B Database - Leadership team
 const leadership = await orangeslice.b2b.sql(`
-  SELECT lp.first_name, lp.last_name, lp.headline, pos.title
+  SELECT lp.first_name, lp.last_name, pos.title
   FROM linkedin_profile lp
   JOIN linkedin_profile_position3 pos ON pos.linkedin_profile_id = lp.id
-  WHERE pos.linkedin_company_id = 1406226
+  WHERE pos.linkedin_company_id = ${company[0].id}
     AND pos.end_date IS NULL
-    AND (pos.title ILIKE 'ceo%' OR pos.title ILIKE 'cto%' OR pos.title ILIKE '%vp%')
-  LIMIT 20
+    AND pos.title ILIKE '%ceo%' OR pos.title ILIKE '%cto%'
+  LIMIT 10
 `);
 
-// 3. Google Search - Recent news
-const news = await orangeslice.serp.search("Ramp fintech funding 2024", { tbs: "qdr:m" });
+const news = await orangeslice.serp.search("Ramp fintech", { tbs: "qdr:m" });
 
-// 4. Website Scraping - About page + socials
 const about = await orangeslice.firecrawl.scrape("https://ramp.com/about");
 ```
 
 ---
 
-**Start by understanding what the user wants to research, then use the appropriate tools to find the information. Verify results when using SERP. Always use indexed columns first when querying the B2B database.**
+**See detailed docs:**
+- [b2b.md](./b2b.md) — Database schema, performance, query patterns
+- [serp.md](./serp.md) — Google dorking cheatsheet, verification
+- [browser.md](./browser.md) — Playwright automation patterns
+- [apify.md](./apify.md) — Pre-built scrapers for social, maps, etc.
+- [strategies.md](./strategies.md) — Prospecting and enrichment patterns

package/docs/apify.md

@@ -0,0 +1,133 @@
+# Apify Actors
+
+Pre-built web scrapers for social media, Google Maps, e-commerce, and more.
+
+```typescript
+import { orangeslice } from 'orangeslice';
+
+// Run an actor
+const results = await orangeslice.apify.run("username/actor-name", { input: "params" });
+
+// Search for actors
+const { actors } = await orangeslice.apify.search("linkedin scraper");
+
+// Get actor input schema (what params it accepts)
+const schema = await orangeslice.apify.getInputSchema("apify/web-scraper");
+```
+
+---
+
+## Workflow
+
+1. **Search** for an actor that does what you need
+2. **Get input schema** to understand required params
+3. **Run** the actor with your inputs
+4. Results are returned when the actor completes
+
+---
+
+## Searching for Actors
+
+```typescript
+const { actors, total } = await orangeslice.apify.search("google maps reviews", 10);
+
+// actors = [{
+//   actorId: "compass/crawler-google-places",
+//   title: "Google Maps Scraper",
+//   description: "Scrape Google Maps...",
+//   stats: { totalRuns: 1000000 },
+//   pricing: { ... }
+// }, ...]
+```
+
+---
+
+## Getting Input Schema
+
+Before running, check what params the actor needs:
+
+```typescript
+const schema = await orangeslice.apify.getInputSchema("compass/crawler-google-places");
+
+console.log(schema.inputProperties);
+// {
+//   searchStringsArray: { type: "array", description: "Search queries" },
+//   maxReviews: { type: "integer", description: "Max reviews per place" },
+//   language: { type: "string", default: "en" },
+//   ...
+// }
+```
+
+---
+
+## Running Actors
+
+```typescript
+// Google Maps reviews
+const reviews = await orangeslice.apify.run("compass/crawler-google-places", {
+  searchStringsArray: ["restaurants in NYC"],
+  maxReviews: 20,
+  language: "en"
+});
+
+// With dataset params (limit results)
+const results = await orangeslice.apify.run("apify/web-scraper", 
+  { startUrls: [{ url: "https://example.com" }] },
+  { limit: 100 }  // Only return first 100 items
+);
+```
+
+---
+
+## Popular Actors
+
+| Use Case | Actor | Example Input |
+|----------|-------|---------------|
+| Google Maps | `compass/crawler-google-places` | `{ searchStringsArray: ["cafes SF"] }` |
+| Google Search | `apify/google-search-scraper` | `{ queries: "site:linkedin.com CEO" }` |
+| Instagram | `apify/instagram-scraper` | `{ directUrls: ["https://instagram.com/user"] }` |
+| Twitter/X | `apidojo/tweet-scraper` | `{ searchTerms: ["#startup"] }` |
+| LinkedIn | `anchor/linkedin-profile-scraper` | `{ profileUrls: [...] }` |
+| YouTube | `streamers/youtube-scraper` | `{ searchKeywords: ["tech reviews"] }` |
+| TikTok | `clockworks/tiktok-scraper` | `{ profiles: ["@username"] }` |
+| Amazon | `junglee/amazon-scraper` | `{ keyword: "laptop stand" }` |
+| Yelp | `yin/yelp-scraper` | `{ searchTerms: "plumber", location: "NYC" }` |
+
+---
+
+## Dataset Params
+
+Control the results returned:
+
+```typescript
+await orangeslice.apify.run(actor, input, {
+  limit: 100,           // Max items to return
+  offset: 0,            // Skip first N items
+  clean: true,          // Remove empty fields
+  fields: ["name", "url"], // Only these fields
+  unwind: "reviews"     // Flatten nested array
+});
+```
+
+---
+
+## Timeouts
+
+- Actors run asynchronously and are polled for completion
+- Max wait: 5 minutes
+- Large scrapes may timeout — use smaller batches
+
+---
+
+## Response Format
+
+Returns the dataset items directly as an array:
+
+```typescript
+const results = await orangeslice.apify.run(...);
+// results = [
+//   { name: "...", address: "...", rating: 4.5 },
+//   { name: "...", address: "...", rating: 4.2 },
+//   ...
+// ]
+```