scrapercity 1.0.16 → 1.0.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT_DOCS.txt +34 -2
- package/README.md +1 -0
- package/SKILL.md +1 -1
- package/bin/cli.mjs +22 -5
- package/bin/mcp.mjs +7 -1
- package/lib/catalog.generated.mjs +6 -1
- package/lib/client.mjs +6 -1
- package/package.json +1 -1
package/AGENT_DOCS.txt
CHANGED
|
@@ -50,8 +50,12 @@ apollo
|
|
|
50
50
|
|
|
51
51
|
maps
|
|
52
52
|
Body: { searchStringsArray: ["plumbers"], locationQuery: "Denver, CO", maxCrawledPlacesPerSearch: 500 }
|
|
53
|
-
Returns: { runId, datasetId, priceMicro }
|
|
53
|
+
Returns: { runId, datasetId, priceMicro, local_business_database? }
|
|
54
|
+
local_business_database (when there are matches): { matches, url, api } - businesses for the
|
|
55
|
+
same search already in the Local Business Database (ones you don't have), a dashboard link,
|
|
56
|
+
and the /api/v1/database/local-businesses call that returns them now.
|
|
54
57
|
Cost: $0.01/place | Delivery: 5-30 min
|
|
58
|
+
To search by category and location without running a scrape, use the Local Business Database (below).
|
|
55
59
|
|
|
56
60
|
email-validator
|
|
57
61
|
Body: { emails: ["user@example.com", "test@company.com"] }
|
|
@@ -216,10 +220,34 @@ GET /api/v1/database/leads?title=CTO&country=United States&hasEmail=true&page=1&
|
|
|
216
220
|
|
|
217
221
|
GET /api/v1/database/local-businesses?category=Dentist&country=US&hasContact=true&limit=100
|
|
218
222
|
Local businesses ($149/mo plan and up, 100k new businesses/day), same pattern as above.
|
|
223
|
+
Search the way you would on Google Maps:
|
|
224
|
+
keyword=thai restaurant (repeat for several; matches business names and categories, any keyword;
|
|
225
|
+
a place inside a keyword, "plumbers in Austin TX", is the location when none is sent)
|
|
226
|
+
location=Houston TX (repeat for several, up to 2,000: cities, states, countries or postal codes)
|
|
227
|
+
The response adds searched_as: { keywords: [{ text, words, categories, matches_nothing }],
|
|
228
|
+
places: [{ text, read_as, kind, not_found? }] }. A search that finds nothing adds
|
|
229
|
+
lead_database: { count, read_as, filters, dashboard_url } when the Lead Database has 100+
|
|
230
|
+
people for it (keywords read as job title or company keywords, whichever has more).
|
|
219
231
|
Filters: category (repeat for several), country (two-letter code), state, city, postalCode,
|
|
220
|
-
query (business name and description),
|
|
232
|
+
query (business name and description), title (business name, comma separated for several),
|
|
233
|
+
minRating, maxRating, minReviews, maxReviews,
|
|
221
234
|
hasEmail, hasPhone, hasWebsite, hasContact (only businesses with a named contact person).
|
|
235
|
+
box=minLat,minLng,maxLat,maxLng: only businesses inside that map area (a box that can't be
|
|
236
|
+
read returns 400).
|
|
237
|
+
Exclusions: notCategory (repeat for several)
|
|
238
|
+
url=<Google Maps search URL>: use that search's keyword and place (other params override).
|
|
239
|
+
The response adds search_translation: { applied, approximated, not_supported }.
|
|
240
|
+
A URL with nothing the database can use returns 400.
|
|
241
|
+
POST /api/v1/database/local-businesses with a JSON body takes the same parameters (arrays as
|
|
242
|
+
JSON arrays). Use it for a long url or long lists: a query string over ~16KB is refused.
|
|
222
243
|
Optional: excludeDelivered=true, after=<id> (as above).
|
|
244
|
+
Returns: { data: [...businesses], pagination: { page, limit, total, totalPages, has_more, next_after },
|
|
245
|
+
rate_limit, search_translation?, searched_as?, lead_database? }
|
|
246
|
+
Business fields include title, category_name, address, city, state, postal_code, country_code,
|
|
247
|
+
phone, website, emails, total_score, reviews_count, lat, lng, and contact_first_name,
|
|
248
|
+
contact_last_name, contact_job_title, contact_email, contact_mobile when there is a named contact.
|
|
249
|
+
503: the search took too long to run. Narrow it (add a category, a city, a state or a postal
|
|
250
|
+
code) and retry.
|
|
223
251
|
|
|
224
252
|
GET /api/v1/database/ecommerce?...
|
|
225
253
|
Same pattern (including excludeDelivered and after), different dataset.
|
|
@@ -257,3 +285,7 @@ WORKFLOW EXAMPLES
|
|
|
257
285
|
|
|
258
286
|
6. REAL ESTATE PIPELINE:
|
|
259
287
|
Zillow Agents + Property Lookup → download CSVs → enrich with email-finder
|
|
288
|
+
|
|
289
|
+
7. LOCAL BUSINESS DATABASE (instant):
|
|
290
|
+
GET /api/v1/database/local-businesses?keyword=dentist&location=Austin%2C+TX&hasEmail=true&page=1
|
|
291
|
+
Loop pages until page > totalPages
|
package/README.md
CHANGED
|
@@ -105,6 +105,7 @@ curl -O https://app.scrapercity.com/api/downloads/RUN_ID \
|
|
|
105
105
|
| **BuiltWith** | All sites using a technology | $4.99/search |
|
|
106
106
|
| **Criminal Records** | Background check by name | $1.00 if found |
|
|
107
107
|
| **Lead Database** | 3M+ B2B contacts, instant query ($149/mo plan and up) | Included |
|
|
108
|
+
| **Local Business Database** | Local businesses with phones, emails and ratings, instant query ($149/mo plan and up) | Included |
|
|
108
109
|
|
|
109
110
|
## How It Works
|
|
110
111
|
|
package/SKILL.md
CHANGED
|
@@ -79,7 +79,7 @@ Apollo scrapes take **up to 4 days** to deliver. Do NOT poll in a loop.
|
|
|
79
79
|
| GET | `/api/v1/scrape/logs/{runId}` | Run logs |
|
|
80
80
|
| GET | `/api/v1/apollo-status` | Apollo service health: `{status: "url-based" or "legacy", message, timestamp}` |
|
|
81
81
|
| GET | `/api/v1/database/leads?title=CTO&country=United%20States&hasEmail=true&limit=100` | Lead DB ($149/mo plan and up, 100k new leads/day). Optional: `excludeDelivered=true` (only leads you don't have yet), `after=0` then `pagination.next_after` (cursor paging). Company filters: `keywords`, `revenueMin`/`revenueMax`, `companyCountry`/`companyState`/`companyCity`. Exclusions: `notTitle`, `notKeywords`, `notIndustry`. `url=<people-search URL>` searches with that URL's filters. POST with a JSON body takes the same parameters (use it for a long `url`) |
|
|
82
|
-
| GET | `/api/v1/database/local-businesses?category=Dentist&country=US&hasContact=true&limit=100` | Local biz DB ($149/mo plan and up, 100k new businesses/day). Filters: `category`, `country`, `state`, `city`, `postalCode`, `query`, `minRating`/`maxRating`, `minReviews`/`maxReviews`, `hasEmail`, `hasPhone`, `hasWebsite`, `hasContact` (only businesses with a named contact person). Same optional `excludeDelivered` / `after` |
|
|
82
|
+
| GET | `/api/v1/database/local-businesses?category=Dentist&country=US&hasContact=true&limit=100` | Local biz DB ($149/mo plan and up, 100k new businesses/day). Search like Google Maps: `keyword` (repeat; names and categories, any keyword) and `location` (repeat; cities, states, countries, postal codes), with `searched_as` in the response and `lead_database` when nothing is found but the Lead Database has 100+ people. Filters: `category`, `country`, `state`, `city`, `postalCode`, `query`, `minRating`/`maxRating`, `minReviews`/`maxReviews`, `hasEmail`, `hasPhone`, `hasWebsite`, `hasContact` (only businesses with a named contact person), `title` (business name), `box=minLat,minLng,maxLat,maxLng` (a map area). Exclusions: `notCategory`. `url=<Google Maps search URL>` searches with that URL's keyword and place. POST with a JSON body takes the same parameters (use it for a long `url`). Same optional `excludeDelivered` / `after` |
|
|
83
83
|
| GET | `/api/v1/database/ecommerce?...` | Ecommerce DB ($149/mo plan and up). Same optional `excludeDelivered` / `after` |
|
|
84
84
|
|
|
85
85
|
## Status Values
|
package/bin/cli.mjs
CHANGED
|
@@ -408,13 +408,17 @@ async function main() {
|
|
|
408
408
|
const keyFor = { '--category': 'category', '--country': 'country', '--state': 'state', '--city': 'city',
|
|
409
409
|
'--postal-code': 'postalCode', '--query': 'query', '--min-rating': 'minRating', '--max-rating': 'maxRating',
|
|
410
410
|
'--min-reviews': 'minReviews', '--max-reviews': 'maxReviews', '--page': 'page', '--limit': 'limit',
|
|
411
|
-
'--after': 'after'
|
|
412
|
-
|
|
413
|
-
|
|
411
|
+
'--after': 'after', '--url': 'url', '--title': 'title', '--not-category': 'notCategory',
|
|
412
|
+
'--box': 'box', '--keyword': 'keyword', '--location': 'location' }
|
|
413
|
+
// List filters take commas: --category "Dentist,Orthodontist" sends both. Locations take semicolons, since a
|
|
414
|
+
// place has commas of its own: --location "Houston, TX; Austin, TX"
|
|
415
|
+
// (--keyword goes as typed: the server splits a list on commas and keeps "plumbers in Austin, TX" whole)
|
|
416
|
+
const lists = new Set(['--category', '--country', '--not-category'])
|
|
414
417
|
for (const f of Object.keys(keyFor)) {
|
|
415
418
|
const v = flag(f)
|
|
416
419
|
if (v === undefined) continue
|
|
417
|
-
params[keyFor[f]] = lists.has(f) ? String(v).split(',').map(s => s.trim()).filter(Boolean)
|
|
420
|
+
params[keyFor[f]] = lists.has(f) ? String(v).split(',').map(s => s.trim()).filter(Boolean)
|
|
421
|
+
: f === '--location' ? String(v).split(';').map(s => s.trim()).filter(Boolean) : v
|
|
418
422
|
}
|
|
419
423
|
if (flagBool('--has-email')) params.hasEmail = 'true'
|
|
420
424
|
if (flagBool('--has-phone')) params.hasPhone = 'true'
|
|
@@ -424,6 +428,14 @@ async function main() {
|
|
|
424
428
|
const r = await sc.dbLocalBusinesses(params)
|
|
425
429
|
console.log(`${r.pagination?.total || '?'} total businesses, page ${r.pagination?.page || 1} of ${r.pagination?.totalPages || '?'}`)
|
|
426
430
|
if (r.pagination?.next_after) console.log(`Next page: --after ${r.pagination.next_after}`)
|
|
431
|
+
if (r.search_translation?.not_supported?.length) console.log(`Not supported from the search URL: ${r.search_translation.not_supported.join(', ')}`)
|
|
432
|
+
if (r.searched_as) {
|
|
433
|
+
const places = (r.searched_as.places || []).map(p => p.not_found ? `${p.text} (not found)` : p.read_as)
|
|
434
|
+
if (places.length) console.log(`Places: ${places.slice(0, 10).join('; ')}${places.length > 10 ? ` and ${places.length - 10} more` : ''}`)
|
|
435
|
+
const none = (r.searched_as.keywords || []).filter(k => k.matches_nothing).map(k => k.text)
|
|
436
|
+
if (none.length) console.log(`Keywords no business name or category has: ${none.join(', ')}`)
|
|
437
|
+
}
|
|
438
|
+
if (r.lead_database) console.log(`Nothing here, but the Lead Database has ${r.lead_database.count} people for this search by ${r.lead_database.read_as}: ${r.lead_database.dashboard_url}`)
|
|
427
439
|
console.log(json(r.data?.slice(0, 3) || r))
|
|
428
440
|
if (r.data?.length > 3) console.log(`... and ${r.data.length - 3} more`)
|
|
429
441
|
break
|
|
@@ -489,9 +501,14 @@ ScraperCity CLI - B2B lead generation from your terminal
|
|
|
489
501
|
--exclude-delivered Only leads you don't already have
|
|
490
502
|
--after <id> Cursor paging (start with 0, then use the printed next id)
|
|
491
503
|
scrapercity db-local [filters] Query local business database
|
|
492
|
-
--
|
|
504
|
+
--keyword "thai restaurant,sushi" What the businesses do or are called (any of them)
|
|
505
|
+
--location "Houston, TX; Austin, TX" Cities, states, countries or postal codes (separate with ;)
|
|
506
|
+
--category --country --state --city --postal-code --query --title
|
|
493
507
|
--min-rating --max-rating --min-reviews --max-reviews
|
|
494
508
|
--has-email --has-phone --has-website --has-contact (only businesses with a named contact person)
|
|
509
|
+
--not-category
|
|
510
|
+
--box "minLat,minLng,maxLat,maxLng" Only businesses inside this map area
|
|
511
|
+
--url "<Google Maps search URL>" Search with that URL's keyword and place
|
|
495
512
|
--exclude-delivered Only businesses you don't already have
|
|
496
513
|
--after <id> Cursor paging (start with 0, then use the printed next id)
|
|
497
514
|
|
package/bin/mcp.mjs
CHANGED
|
@@ -84,14 +84,20 @@ const UTILITY_TOOLS = [
|
|
|
84
84
|
},
|
|
85
85
|
{
|
|
86
86
|
name: 'query_local_business_database',
|
|
87
|
-
description: 'Query the local business database directly. Returns businesses with phone numbers, emails, websites, addresses and ratings, many with a named contact person. Up to 100 per request. Included with the $149/mo plan (100,000 new businesses a day; businesses you already have do not count again). Set hasContact=true for only businesses with a named contact person. For a daily pull of only new businesses set excludeDelivered=true. For big pulls page with after: send after="0" first, then the pagination.next_after from each response until it is null.',
|
|
87
|
+
description: 'Query the local business database directly. Returns businesses with phone numbers, emails, websites, addresses and ratings, many with a named contact person. Up to 100 per request. Included with the $149/mo plan (100,000 new businesses a day; businesses you already have do not count again). Set hasContact=true for only businesses with a named contact person. For a daily pull of only new businesses set excludeDelivered=true. For big pulls page with after: send after="0" first, then the pagination.next_after from each response until it is null. Search the way you would on Google Maps: keyword (what the businesses do or are called, e.g. ["thai restaurant"]) and location (cities, states, countries or postal codes, e.g. ["Houston TX", "Austin TX"]); searched_as in the response says how each was read. A search that finds nothing also returns lead_database: the same search in the Lead Database when it has 100+ people. Have a Google Maps search URL? Pass it as url to search with its keyword and place right away (search_translation in the response says what was applied).',
|
|
88
88
|
inputSchema: { type: 'object', properties: {
|
|
89
|
+
keyword: { type: 'array', items: { type: 'string' }, description: 'What the businesses do or are called, as typed on Google Maps, e.g. ["thai restaurant", "sushi"]. Matches business names and categories; any keyword. A place inside a keyword ("plumbers in Austin TX") is used as the location when no location is given.' },
|
|
90
|
+
location: { type: 'array', items: { type: 'string' }, description: 'Cities, states, countries or postal codes, as typed, e.g. ["Houston TX", "78701", "Ontario", "Germany"] (up to 2,000; any of them).' },
|
|
91
|
+
url: { type: 'string', description: 'A Google Maps search URL (the address of a search on Google Maps). Its keyword and place become the search; any other filter you pass overrides the matching one.' },
|
|
89
92
|
category: { type: 'array', items: { type: 'string' }, description: 'Business categories, e.g. ["Dentist"]. A word also finds categories that contain it ("restaurant" finds "Mexican restaurant")' },
|
|
90
93
|
country: { type: 'array', items: { type: 'string' }, description: 'Two-letter country codes, e.g. ["US"]' },
|
|
91
94
|
state: { type: 'string', description: 'State or region, comma separated for several (US abbreviations work, e.g. "TX")' },
|
|
92
95
|
city: { type: 'string', description: 'City (partial match)' },
|
|
93
96
|
postalCode: { type: 'string', description: 'Postal or zip code' },
|
|
97
|
+
box: { type: 'string', description: 'A map area as "minLat,minLng,maxLat,maxLng" (e.g. "30.1,-97.9,30.5,-97.5"). Only businesses inside it.' },
|
|
94
98
|
query: { type: 'string', description: 'Search business names and descriptions' },
|
|
99
|
+
title: { type: 'string', description: 'Business name, comma separated for several (partial match)' },
|
|
100
|
+
notCategory: { type: 'array', items: { type: 'string' }, description: 'Exclude these business categories' },
|
|
95
101
|
minRating: { type: 'number', description: 'Minimum Google rating (1-5)' },
|
|
96
102
|
maxRating: { type: 'number', description: 'Maximum Google rating (1-5)' },
|
|
97
103
|
minReviews: { type: 'number', description: 'Minimum number of reviews' },
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
// AUTO-GENERATED by scripts/generate.mjs — DO NOT EDIT BY HAND.
|
|
2
2
|
// Source of truth: scraperConfigs.ts (+ src/config/scrapers.ts). Regenerate: npm run generate
|
|
3
|
-
// Generated: 2026-10-
|
|
3
|
+
// Generated: 2026-10-09T16:22:32.237Z
|
|
4
4
|
export const TOOLS = [
|
|
5
5
|
{
|
|
6
6
|
"name": "scrape_apollo",
|
|
@@ -169,6 +169,10 @@ export const TOOLS = [
|
|
|
169
169
|
"maxCrawledPlacesPerSearch": {
|
|
170
170
|
"type": "number",
|
|
171
171
|
"description": "Maximum results per search term"
|
|
172
|
+
},
|
|
173
|
+
"url": {
|
|
174
|
+
"type": "string",
|
|
175
|
+
"description": "A Google Maps search link, instead of searchStringsArray and locationQuery. Its search words become the keywords and the place in them the location (\"thai restaurants vegas\" runs \"thai restaurants\" in Las Vegas, Nevada). With no place in the words, the map area the link shows is searched. The response says what it ran in search_translation. searchStringsArray, locationQuery or customGeolocation sent with it are used instead of the matching part of the link."
|
|
172
176
|
}
|
|
173
177
|
}
|
|
174
178
|
}
|
|
@@ -655,6 +659,7 @@ export const ENDPOINTS = {
|
|
|
655
659
|
"buy-credits": "/api/v1/buy-credits",
|
|
656
660
|
"database-leads": "/api/v1/database/leads",
|
|
657
661
|
"database-enrichment": "/api/v1/enrich",
|
|
662
|
+
"database-local-business-enrichment": "/api/v1/enrich/local-businesses",
|
|
658
663
|
"database-local-businesses": "/api/v1/database/local-businesses",
|
|
659
664
|
"database-ecommerce": "/api/v1/database/ecommerce"
|
|
660
665
|
}
|
package/lib/client.mjs
CHANGED
|
@@ -183,14 +183,19 @@ export const dbLeads = (params) => {
|
|
|
183
183
|
return get(`${ENDPOINTS['database-leads']}?${qstr}`)
|
|
184
184
|
}
|
|
185
185
|
|
|
186
|
+
// Same as dbLeads: a long request (a big Google Maps URL, long category lists) goes as a POST with a JSON body.
|
|
186
187
|
export const dbLocalBusinesses = (params) => {
|
|
187
188
|
const qs = new URLSearchParams()
|
|
189
|
+
const body = {}
|
|
188
190
|
for (const [k, v] of Object.entries(params)) {
|
|
189
191
|
if (v === undefined || v === null || v === '') continue
|
|
190
192
|
if (Array.isArray(v)) v.forEach(x => qs.append(k, x))
|
|
191
193
|
else qs.append(k, String(v))
|
|
194
|
+
body[k] = v
|
|
192
195
|
}
|
|
193
|
-
|
|
196
|
+
const qstr = qs.toString()
|
|
197
|
+
if (qstr.length > LONG_QUERY) return post(ENDPOINTS['database-local-businesses'], body)
|
|
198
|
+
return get(`${ENDPOINTS['database-local-businesses']}?${qstr}`)
|
|
194
199
|
}
|
|
195
200
|
|
|
196
201
|
export const dbEcommerce = (params) => {
|