scrapercity 1.0.15 → 1.0.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT_DOCS.txt +41 -3
- package/README.md +1 -0
- package/SKILL.md +1 -1
- package/bin/cli.mjs +50 -0
- package/bin/mcp.mjs +40 -0
- package/lib/catalog.generated.mjs +6 -1
- package/lib/client.mjs +6 -1
- package/package.json +1 -1
package/AGENT_DOCS.txt
CHANGED
|
@@ -50,8 +50,12 @@ apollo
|
|
|
50
50
|
|
|
51
51
|
maps
|
|
52
52
|
Body: { searchStringsArray: ["plumbers"], locationQuery: "Denver, CO", maxCrawledPlacesPerSearch: 500 }
|
|
53
|
-
Returns: { runId, datasetId, priceMicro }
|
|
53
|
+
Returns: { runId, datasetId, priceMicro, local_business_database? }
|
|
54
|
+
local_business_database (when there are matches): { matches, url, api } - businesses for the
|
|
55
|
+
same search already in the Local Business Database (ones you don't have), a dashboard link,
|
|
56
|
+
and the /api/v1/database/local-businesses call that returns them now.
|
|
54
57
|
Cost: $0.01/place | Delivery: 5-30 min
|
|
58
|
+
To search by category and location without running a scrape, use the Local Business Database (below).
|
|
55
59
|
|
|
56
60
|
email-validator
|
|
57
61
|
Body: { emails: ["user@example.com", "test@company.com"] }
|
|
@@ -214,9 +218,39 @@ GET /api/v1/database/leads?title=CTO&country=United States&hasEmail=true&page=1&
|
|
|
214
218
|
503: the search took too long to run. Narrow it (fewer titles or keywords, add a location
|
|
215
219
|
or company size) and retry.
|
|
216
220
|
|
|
217
|
-
GET /api/v1/database/local-businesses
|
|
221
|
+
GET /api/v1/database/local-businesses?category=Dentist&country=US&hasContact=true&limit=100
|
|
222
|
+
Local businesses ($149/mo plan and up, 100k new businesses/day), same pattern as above.
|
|
223
|
+
Search the way you would on Google Maps:
|
|
224
|
+
keyword=thai restaurant (repeat for several; matches business names and categories, any keyword;
|
|
225
|
+
a place inside a keyword, "plumbers in Austin TX", is the location when none is sent)
|
|
226
|
+
location=Houston TX (repeat for several, up to 2,000: cities, states, countries or postal codes)
|
|
227
|
+
The response adds searched_as: { keywords: [{ text, words, categories, matches_nothing }],
|
|
228
|
+
places: [{ text, read_as, kind, not_found? }] }. A search that finds nothing adds
|
|
229
|
+
lead_database: { count, read_as, filters, dashboard_url } when the Lead Database has 100+
|
|
230
|
+
people for it (keywords read as job title or company keywords, whichever has more).
|
|
231
|
+
Filters: category (repeat for several), country (two-letter code), state, city, postalCode,
|
|
232
|
+
query (business name and description), title (business name, comma separated for several),
|
|
233
|
+
minRating, maxRating, minReviews, maxReviews,
|
|
234
|
+
hasEmail, hasPhone, hasWebsite, hasContact (only businesses with a named contact person).
|
|
235
|
+
box=minLat,minLng,maxLat,maxLng: only businesses inside that map area (a box that can't be
|
|
236
|
+
read returns 400).
|
|
237
|
+
Exclusions: notCategory (repeat for several)
|
|
238
|
+
url=<Google Maps search URL>: use that search's keyword and place (other params override).
|
|
239
|
+
The response adds search_translation: { applied, approximated, not_supported }.
|
|
240
|
+
A URL with nothing the database can use returns 400.
|
|
241
|
+
POST /api/v1/database/local-businesses with a JSON body takes the same parameters (arrays as
|
|
242
|
+
JSON arrays). Use it for a long url or long lists: a query string over ~16KB is refused.
|
|
243
|
+
Optional: excludeDelivered=true, after=<id> (as above).
|
|
244
|
+
Returns: { data: [...businesses], pagination: { page, limit, total, totalPages, has_more, next_after },
|
|
245
|
+
rate_limit, search_translation?, searched_as?, lead_database? }
|
|
246
|
+
Business fields include title, category_name, address, city, state, postal_code, country_code,
|
|
247
|
+
phone, website, emails, total_score, reviews_count, lat, lng, and contact_first_name,
|
|
248
|
+
contact_last_name, contact_job_title, contact_email, contact_mobile when there is a named contact.
|
|
249
|
+
503: the search took too long to run. Narrow it (add a category, a city, a state or a postal
|
|
250
|
+
code) and retry.
|
|
251
|
+
|
|
218
252
|
GET /api/v1/database/ecommerce?...
|
|
219
|
-
Same pattern (including excludeDelivered and after), different
|
|
253
|
+
Same pattern (including excludeDelivered and after), different dataset.
|
|
220
254
|
|
|
221
255
|
────────────────────────────────────────
|
|
222
256
|
ERROR CODES
|
|
@@ -251,3 +285,7 @@ WORKFLOW EXAMPLES
|
|
|
251
285
|
|
|
252
286
|
6. REAL ESTATE PIPELINE:
|
|
253
287
|
Zillow Agents + Property Lookup → download CSVs → enrich with email-finder
|
|
288
|
+
|
|
289
|
+
7. LOCAL BUSINESS DATABASE (instant):
|
|
290
|
+
GET /api/v1/database/local-businesses?keyword=dentist&location=Austin%2C+TX&hasEmail=true&page=1
|
|
291
|
+
Loop pages until page > totalPages
|
package/README.md
CHANGED
|
@@ -105,6 +105,7 @@ curl -O https://app.scrapercity.com/api/downloads/RUN_ID \
|
|
|
105
105
|
| **BuiltWith** | All sites using a technology | $4.99/search |
|
|
106
106
|
| **Criminal Records** | Background check by name | $1.00 if found |
|
|
107
107
|
| **Lead Database** | 3M+ B2B contacts, instant query ($149/mo plan and up) | Included |
|
|
108
|
+
| **Local Business Database** | Local businesses with phones, emails and ratings, instant query ($149/mo plan and up) | Included |
|
|
108
109
|
|
|
109
110
|
## How It Works
|
|
110
111
|
|
package/SKILL.md
CHANGED
|
@@ -79,7 +79,7 @@ Apollo scrapes take **up to 4 days** to deliver. Do NOT poll in a loop.
|
|
|
79
79
|
| GET | `/api/v1/scrape/logs/{runId}` | Run logs |
|
|
80
80
|
| GET | `/api/v1/apollo-status` | Apollo service health: `{status: "url-based" or "legacy", message, timestamp}` |
|
|
81
81
|
| GET | `/api/v1/database/leads?title=CTO&country=United%20States&hasEmail=true&limit=100` | Lead DB ($149/mo plan and up, 100k new leads/day). Optional: `excludeDelivered=true` (only leads you don't have yet), `after=0` then `pagination.next_after` (cursor paging). Company filters: `keywords`, `revenueMin`/`revenueMax`, `companyCountry`/`companyState`/`companyCity`. Exclusions: `notTitle`, `notKeywords`, `notIndustry`. `url=<people-search URL>` searches with that URL's filters. POST with a JSON body takes the same parameters (use it for a long `url`) |
|
|
82
|
-
| GET | `/api/v1/database/local-businesses
|
|
82
|
+
| GET | `/api/v1/database/local-businesses?category=Dentist&country=US&hasContact=true&limit=100` | Local biz DB ($149/mo plan and up, 100k new businesses/day). Search like Google Maps: `keyword` (repeat; names and categories, any keyword) and `location` (repeat; cities, states, countries, postal codes), with `searched_as` in the response and `lead_database` when nothing is found but the Lead Database has 100+ people. Filters: `category`, `country`, `state`, `city`, `postalCode`, `query`, `minRating`/`maxRating`, `minReviews`/`maxReviews`, `hasEmail`, `hasPhone`, `hasWebsite`, `hasContact` (only businesses with a named contact person), `title` (business name), `box=minLat,minLng,maxLat,maxLng` (a map area). Exclusions: `notCategory`. `url=<Google Maps search URL>` searches with that URL's keyword and place. POST with a JSON body takes the same parameters (use it for a long `url`). Same optional `excludeDelivered` / `after` |
|
|
83
83
|
| GET | `/api/v1/database/ecommerce?...` | Ecommerce DB ($149/mo plan and up). Same optional `excludeDelivered` / `after` |
|
|
84
84
|
|
|
85
85
|
## Status Values
|
package/bin/cli.mjs
CHANGED
|
@@ -402,6 +402,45 @@ async function main() {
|
|
|
402
402
|
break
|
|
403
403
|
}
|
|
404
404
|
|
|
405
|
+
// ── Database: Local businesses ($149/mo plan and up) ──
|
|
406
|
+
case 'db-local': {
|
|
407
|
+
const params = {}
|
|
408
|
+
const keyFor = { '--category': 'category', '--country': 'country', '--state': 'state', '--city': 'city',
|
|
409
|
+
'--postal-code': 'postalCode', '--query': 'query', '--min-rating': 'minRating', '--max-rating': 'maxRating',
|
|
410
|
+
'--min-reviews': 'minReviews', '--max-reviews': 'maxReviews', '--page': 'page', '--limit': 'limit',
|
|
411
|
+
'--after': 'after', '--url': 'url', '--title': 'title', '--not-category': 'notCategory',
|
|
412
|
+
'--box': 'box', '--keyword': 'keyword', '--location': 'location' }
|
|
413
|
+
// List filters take commas: --category "Dentist,Orthodontist" sends both. Locations take semicolons, since a
|
|
414
|
+
// place has commas of its own: --location "Houston, TX; Austin, TX"
|
|
415
|
+
// (--keyword goes as typed: the server splits a list on commas and keeps "plumbers in Austin, TX" whole)
|
|
416
|
+
const lists = new Set(['--category', '--country', '--not-category'])
|
|
417
|
+
for (const f of Object.keys(keyFor)) {
|
|
418
|
+
const v = flag(f)
|
|
419
|
+
if (v === undefined) continue
|
|
420
|
+
params[keyFor[f]] = lists.has(f) ? String(v).split(',').map(s => s.trim()).filter(Boolean)
|
|
421
|
+
: f === '--location' ? String(v).split(';').map(s => s.trim()).filter(Boolean) : v
|
|
422
|
+
}
|
|
423
|
+
if (flagBool('--has-email')) params.hasEmail = 'true'
|
|
424
|
+
if (flagBool('--has-phone')) params.hasPhone = 'true'
|
|
425
|
+
if (flagBool('--has-website')) params.hasWebsite = 'true'
|
|
426
|
+
if (flagBool('--has-contact')) params.hasContact = 'true'
|
|
427
|
+
if (flagBool('--exclude-delivered')) params.excludeDelivered = 'true'
|
|
428
|
+
const r = await sc.dbLocalBusinesses(params)
|
|
429
|
+
console.log(`${r.pagination?.total || '?'} total businesses, page ${r.pagination?.page || 1} of ${r.pagination?.totalPages || '?'}`)
|
|
430
|
+
if (r.pagination?.next_after) console.log(`Next page: --after ${r.pagination.next_after}`)
|
|
431
|
+
if (r.search_translation?.not_supported?.length) console.log(`Not supported from the search URL: ${r.search_translation.not_supported.join(', ')}`)
|
|
432
|
+
if (r.searched_as) {
|
|
433
|
+
const places = (r.searched_as.places || []).map(p => p.not_found ? `${p.text} (not found)` : p.read_as)
|
|
434
|
+
if (places.length) console.log(`Places: ${places.slice(0, 10).join('; ')}${places.length > 10 ? ` and ${places.length - 10} more` : ''}`)
|
|
435
|
+
const none = (r.searched_as.keywords || []).filter(k => k.matches_nothing).map(k => k.text)
|
|
436
|
+
if (none.length) console.log(`Keywords no business name or category has: ${none.join(', ')}`)
|
|
437
|
+
}
|
|
438
|
+
if (r.lead_database) console.log(`Nothing here, but the Lead Database has ${r.lead_database.count} people for this search by ${r.lead_database.read_as}: ${r.lead_database.dashboard_url}`)
|
|
439
|
+
console.log(json(r.data?.slice(0, 3) || r))
|
|
440
|
+
if (r.data?.length > 3) console.log(`... and ${r.data.length - 3} more`)
|
|
441
|
+
break
|
|
442
|
+
}
|
|
443
|
+
|
|
405
444
|
// ── Poll (convenience) ────────────────────────────────
|
|
406
445
|
case 'poll': {
|
|
407
446
|
if (!args[0]) die('Usage: scrapercity poll <runId> [--interval 15]')
|
|
@@ -461,6 +500,17 @@ ScraperCity CLI - B2B lead generation from your terminal
|
|
|
461
500
|
--url "<people-search URL>" Search with that URL's filters
|
|
462
501
|
--exclude-delivered Only leads you don't already have
|
|
463
502
|
--after <id> Cursor paging (start with 0, then use the printed next id)
|
|
503
|
+
scrapercity db-local [filters] Query local business database
|
|
504
|
+
--keyword "thai restaurant,sushi" What the businesses do or are called (any of them)
|
|
505
|
+
--location "Houston, TX; Austin, TX" Cities, states, countries or postal codes (separate with ;)
|
|
506
|
+
--category --country --state --city --postal-code --query --title
|
|
507
|
+
--min-rating --max-rating --min-reviews --max-reviews
|
|
508
|
+
--has-email --has-phone --has-website --has-contact (only businesses with a named contact person)
|
|
509
|
+
--not-category
|
|
510
|
+
--box "minLat,minLng,maxLat,maxLng" Only businesses inside this map area
|
|
511
|
+
--url "<Google Maps search URL>" Search with that URL's keyword and place
|
|
512
|
+
--exclude-delivered Only businesses you don't already have
|
|
513
|
+
--after <id> Cursor paging (start with 0, then use the printed next id)
|
|
464
514
|
|
|
465
515
|
Env: SCRAPERCITY_API_KEY=... or scrapercity login
|
|
466
516
|
Docs: https://scrapercity.com/agents
|
package/bin/mcp.mjs
CHANGED
|
@@ -81,6 +81,36 @@ const UTILITY_TOOLS = [
|
|
|
81
81
|
excludeDelivered: { type: 'boolean', description: 'Skip leads this account already has (from the API or unlocked in the dashboard). With this on, keep page at 1 or use after.' },
|
|
82
82
|
after: { type: 'string', description: 'Cursor paging: return leads after this lead id. Use "0" to start, then the pagination.next_after from each response.' }
|
|
83
83
|
} }
|
|
84
|
+
},
|
|
85
|
+
{
|
|
86
|
+
name: 'query_local_business_database',
|
|
87
|
+
description: 'Query the local business database directly. Returns businesses with phone numbers, emails, websites, addresses and ratings, many with a named contact person. Up to 100 per request. Included with the $149/mo plan (100,000 new businesses a day; businesses you already have do not count again). Set hasContact=true for only businesses with a named contact person. For a daily pull of only new businesses set excludeDelivered=true. For big pulls page with after: send after="0" first, then the pagination.next_after from each response until it is null. Search the way you would on Google Maps: keyword (what the businesses do or are called, e.g. ["thai restaurant"]) and location (cities, states, countries or postal codes, e.g. ["Houston TX", "Austin TX"]); searched_as in the response says how each was read. A search that finds nothing also returns lead_database: the same search in the Lead Database when it has 100+ people. Have a Google Maps search URL? Pass it as url to search with its keyword and place right away (search_translation in the response says what was applied).',
|
|
88
|
+
inputSchema: { type: 'object', properties: {
|
|
89
|
+
keyword: { type: 'array', items: { type: 'string' }, description: 'What the businesses do or are called, as typed on Google Maps, e.g. ["thai restaurant", "sushi"]. Matches business names and categories; any keyword. A place inside a keyword ("plumbers in Austin TX") is used as the location when no location is given.' },
|
|
90
|
+
location: { type: 'array', items: { type: 'string' }, description: 'Cities, states, countries or postal codes, as typed, e.g. ["Houston TX", "78701", "Ontario", "Germany"] (up to 2,000; any of them).' },
|
|
91
|
+
url: { type: 'string', description: 'A Google Maps search URL (the address of a search on Google Maps). Its keyword and place become the search; any other filter you pass overrides the matching one.' },
|
|
92
|
+
category: { type: 'array', items: { type: 'string' }, description: 'Business categories, e.g. ["Dentist"]. A word also finds categories that contain it ("restaurant" finds "Mexican restaurant")' },
|
|
93
|
+
country: { type: 'array', items: { type: 'string' }, description: 'Two-letter country codes, e.g. ["US"]' },
|
|
94
|
+
state: { type: 'string', description: 'State or region, comma separated for several (US abbreviations work, e.g. "TX")' },
|
|
95
|
+
city: { type: 'string', description: 'City (partial match)' },
|
|
96
|
+
postalCode: { type: 'string', description: 'Postal or zip code' },
|
|
97
|
+
box: { type: 'string', description: 'A map area as "minLat,minLng,maxLat,maxLng" (e.g. "30.1,-97.9,30.5,-97.5"). Only businesses inside it.' },
|
|
98
|
+
query: { type: 'string', description: 'Search business names and descriptions' },
|
|
99
|
+
title: { type: 'string', description: 'Business name, comma separated for several (partial match)' },
|
|
100
|
+
notCategory: { type: 'array', items: { type: 'string' }, description: 'Exclude these business categories' },
|
|
101
|
+
minRating: { type: 'number', description: 'Minimum Google rating (1-5)' },
|
|
102
|
+
maxRating: { type: 'number', description: 'Maximum Google rating (1-5)' },
|
|
103
|
+
minReviews: { type: 'number', description: 'Minimum number of reviews' },
|
|
104
|
+
maxReviews: { type: 'number', description: 'Maximum number of reviews' },
|
|
105
|
+
hasEmail: { type: 'boolean', description: 'Only businesses with an email address' },
|
|
106
|
+
hasPhone: { type: 'boolean', description: 'Only businesses with a phone number' },
|
|
107
|
+
hasWebsite: { type: 'boolean', description: 'Only businesses with a website' },
|
|
108
|
+
hasContact: { type: 'boolean', description: 'Only businesses with a named contact person' },
|
|
109
|
+
page: { type: 'number', description: 'Page number (default 1)', default: 1 },
|
|
110
|
+
limit: { type: 'number', description: 'Results per page (max 100)', default: 50 },
|
|
111
|
+
excludeDelivered: { type: 'boolean', description: 'Skip businesses this account already has (from the API or unlocked in the dashboard). With this on, keep page at 1 or use after.' },
|
|
112
|
+
after: { type: 'string', description: 'Cursor paging: return businesses after this id. Use "0" to start, then the pagination.next_after from each response.' }
|
|
113
|
+
} }
|
|
84
114
|
}
|
|
85
115
|
]
|
|
86
116
|
|
|
@@ -116,6 +146,16 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
|
|
|
116
146
|
result = await sc.dbLeads(params)
|
|
117
147
|
break
|
|
118
148
|
}
|
|
149
|
+
case 'query_local_business_database': {
|
|
150
|
+
const params = { ...args }
|
|
151
|
+
for (const k of ['hasEmail', 'hasPhone', 'hasWebsite', 'hasContact', 'excludeDelivered']) {
|
|
152
|
+
if (params[k]) params[k] = 'true'
|
|
153
|
+
else delete params[k]
|
|
154
|
+
}
|
|
155
|
+
if (params.after !== undefined && params.after !== null) params.after = String(params.after)
|
|
156
|
+
result = await sc.dbLocalBusinesses(params)
|
|
157
|
+
break
|
|
158
|
+
}
|
|
119
159
|
default:
|
|
120
160
|
return { content: [{ type: 'text', text: `Unknown tool: ${name}` }], isError: true }
|
|
121
161
|
}
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
// AUTO-GENERATED by scripts/generate.mjs — DO NOT EDIT BY HAND.
|
|
2
2
|
// Source of truth: scraperConfigs.ts (+ src/config/scrapers.ts). Regenerate: npm run generate
|
|
3
|
-
// Generated: 2026-10-
|
|
3
|
+
// Generated: 2026-10-09T16:22:32.237Z
|
|
4
4
|
export const TOOLS = [
|
|
5
5
|
{
|
|
6
6
|
"name": "scrape_apollo",
|
|
@@ -169,6 +169,10 @@ export const TOOLS = [
|
|
|
169
169
|
"maxCrawledPlacesPerSearch": {
|
|
170
170
|
"type": "number",
|
|
171
171
|
"description": "Maximum results per search term"
|
|
172
|
+
},
|
|
173
|
+
"url": {
|
|
174
|
+
"type": "string",
|
|
175
|
+
"description": "A Google Maps search link, instead of searchStringsArray and locationQuery. Its search words become the keywords and the place in them the location (\"thai restaurants vegas\" runs \"thai restaurants\" in Las Vegas, Nevada). With no place in the words, the map area the link shows is searched. The response says what it ran in search_translation. searchStringsArray, locationQuery or customGeolocation sent with it are used instead of the matching part of the link."
|
|
172
176
|
}
|
|
173
177
|
}
|
|
174
178
|
}
|
|
@@ -655,6 +659,7 @@ export const ENDPOINTS = {
|
|
|
655
659
|
"buy-credits": "/api/v1/buy-credits",
|
|
656
660
|
"database-leads": "/api/v1/database/leads",
|
|
657
661
|
"database-enrichment": "/api/v1/enrich",
|
|
662
|
+
"database-local-business-enrichment": "/api/v1/enrich/local-businesses",
|
|
658
663
|
"database-local-businesses": "/api/v1/database/local-businesses",
|
|
659
664
|
"database-ecommerce": "/api/v1/database/ecommerce"
|
|
660
665
|
}
|
package/lib/client.mjs
CHANGED
|
@@ -183,14 +183,19 @@ export const dbLeads = (params) => {
|
|
|
183
183
|
return get(`${ENDPOINTS['database-leads']}?${qstr}`)
|
|
184
184
|
}
|
|
185
185
|
|
|
186
|
+
// Same as dbLeads: a long request (a big Google Maps URL, long category lists) goes as a POST with a JSON body.
|
|
186
187
|
export const dbLocalBusinesses = (params) => {
|
|
187
188
|
const qs = new URLSearchParams()
|
|
189
|
+
const body = {}
|
|
188
190
|
for (const [k, v] of Object.entries(params)) {
|
|
189
191
|
if (v === undefined || v === null || v === '') continue
|
|
190
192
|
if (Array.isArray(v)) v.forEach(x => qs.append(k, x))
|
|
191
193
|
else qs.append(k, String(v))
|
|
194
|
+
body[k] = v
|
|
192
195
|
}
|
|
193
|
-
|
|
196
|
+
const qstr = qs.toString()
|
|
197
|
+
if (qstr.length > LONG_QUERY) return post(ENDPOINTS['database-local-businesses'], body)
|
|
198
|
+
return get(`${ENDPOINTS['database-local-businesses']}?${qstr}`)
|
|
194
199
|
}
|
|
195
200
|
|
|
196
201
|
export const dbEcommerce = (params) => {
|