lacspace-leads 1.6.0 → 1.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +165 -1
- package/dist/cli.js +269 -8
- package/dist/lib.cjs +214 -1
- package/dist/lib.d.cts +155 -29
- package/dist/lib.d.ts +155 -29
- package/dist/lib.js +204 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -11,6 +11,7 @@ That opens a browser, searches Maps for *"restaurants in Baneshwor, Kathmandu"*,
|
|
|
11
11
|
## Why it's different
|
|
12
12
|
|
|
13
13
|
- **Free & keyless** — uses a real browser (via [Playwright](https://playwright.dev)), not a paid Places API.
|
|
14
|
+
- **Ask for a number, get that number** — one Google search stops at ~120 results; `--target 500` keeps searching (every area you name, then tiles of the map) until it has 500 unique businesses.
|
|
14
15
|
- **Sweep a whole city** — comma-separate areas and it runs each search, then **merges and de-duplicates** into one list: `--area "Thamel,Baneshwor,Patan"`.
|
|
15
16
|
- **Search by radius** — centre on a coordinate and keep only what's within range: `--near "27.72,85.32" --radius 2km`. Each lead gets a `distanceKm`, sorted nearest-first.
|
|
16
17
|
- **Accumulate a master list** — `--append` merges each run into your existing file and de-duplicates, so daily runs build one clean database.
|
|
@@ -27,6 +28,132 @@ That opens a browser, searches Maps for *"restaurants in Baneshwor, Kathmandu"*,
|
|
|
27
28
|
- **Robust & polite** — `--proxy`, `--retries`, `--jitter`, per-listing delays and a permission-first prompt before it opens a browser.
|
|
28
29
|
- **Library too** — `import { searchLeads, searchLeadsBatch, searchLeadsDetailed } from "lacspace-leads"`.
|
|
29
30
|
|
|
31
|
+
## How to use it
|
|
32
|
+
|
|
33
|
+
### 1. Run it and answer the questions
|
|
34
|
+
|
|
35
|
+
No flags to learn first. Run it bare and it walks you through business type, city, areas, how many leads you want and the file format — every question has a default, so you can press Enter through it:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
npx lacspace-leads
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Type more than ~120 at the "how many" question and it automatically switches to a full sweep, because a single Google search cannot return more than that.
|
|
42
|
+
|
|
43
|
+
### 2. Or say it in one line
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
npx lacspace-leads restaurants --city Kathmandu --area Baneshwor -f xlsx
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
A real Chromium window opens, walks the map and writes the file into the folder you ran it from. Add `--headless` once you trust it and the window stays hidden.
|
|
50
|
+
|
|
51
|
+
### 3. Ask for the number of leads you actually want
|
|
52
|
+
|
|
53
|
+
`-n / --limit` is the cap for **one search**, and Google itself stops serving a single search at roughly **120 results** — which is why `--limit 500` comes back with about 114. `--target` is the flag that goes past that:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
# 500 unique restaurants, however many searches that takes
|
|
57
|
+
npx lacspace-leads restaurants --city Kathmandu --target 500 -f xlsx
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
It expands coverage in this order, stopping the moment it has enough:
|
|
61
|
+
|
|
62
|
+
1. **Every place you named** — each city x area x type is its own search.
|
|
63
|
+
2. **Map tiles** — the same query re-centred on a grid of points walking outwards from the city centre. The centre comes from the map itself, so no geocoding service and no API key.
|
|
64
|
+
|
|
65
|
+
Name your neighbourhoods to put the areas you care about first, and to finish sooner:
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
npx lacspace-leads restaurants --city Kathmandu \
|
|
69
|
+
--areas "Baneshwor, Thamel, Patan, Lazimpat" --target 400
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Tune the grid with `--step` (spacing between tiles — tighter finds more in a dense city) and `--tiles` (how far out to go):
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
npx lacspace-leads cafes --city Lalitpur --target 600 --step 1.5km --tiles 60
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
If the map genuinely runs out before the target, it tells you instead of pretending:
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
! The map ran out of new results at 214 — that is everything Google lists here.
|
|
82
|
+
Widen it: more areas, more cities, a bigger --step, or related --types.
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### 4. Several cities, several areas, several business types
|
|
86
|
+
|
|
87
|
+
Comma-separate any of them. Every combination becomes its own search and the results merge into one de-duplicated list:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
npx lacspace-leads \
|
|
91
|
+
--type "restaurant, cafe" \
|
|
92
|
+
--cities "Kathmandu, Lalitpur, Bhaktapur" \
|
|
93
|
+
--areas "Baneshwor, Thamel" \
|
|
94
|
+
--target 600 -f xlsx
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Add `--split city` to also write one file per city (or `area`, or `type`) beside the master export:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
npx lacspace-leads gyms --cities "Kathmandu, Pokhara" --target 300 --split city -f xlsx
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Long sweeps are resumable — `--resume` checkpoints after every search, so an interruption continues instead of re-scraping.
|
|
104
|
+
|
|
105
|
+
### 5. Turn listings into contacts
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
# Emails + social profiles from each business website
|
|
109
|
+
npx lacspace-leads clinics --city Pokhara --target 200 --enrich -f xlsx
|
|
110
|
+
|
|
111
|
+
# Only addresses whose domain really accepts mail
|
|
112
|
+
npx lacspace-leads hotels --city Kathmandu --target 150 \
|
|
113
|
+
--enrich --verify-emails --has-valid-email -f csv
|
|
114
|
+
|
|
115
|
+
# Phones in international format, ready for a CRM or bulk SMS
|
|
116
|
+
npx lacspace-leads pharmacies --city Biratnagar --target 200 --country NP --has-phone
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
### 6. Narrow it to the right prospects
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
# Businesses with NO website — the pitch list for web work
|
|
123
|
+
npx lacspace-leads "beauty salon" --city Pokhara --target 200 --no-website --has-phone -f csv
|
|
124
|
+
|
|
125
|
+
# Only well-reviewed places, best first
|
|
126
|
+
npx lacspace-leads restaurants --city Kathmandu --target 300 \
|
|
127
|
+
--min-rating 4 --min-reviews 50 --sort reviews --desc
|
|
128
|
+
|
|
129
|
+
# Just the columns an outreach list needs
|
|
130
|
+
npx lacspace-leads cafes --city Lalitpur --target 200 --preset outreach
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
### 7. Make it repeatable
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
# One master file that grows and de-duplicates every run
|
|
137
|
+
npx lacspace-leads restaurants --city Kathmandu --target 300 --append -o master.csv
|
|
138
|
+
|
|
139
|
+
# Export only the businesses that aren't already in it
|
|
140
|
+
npx lacspace-leads restaurants --city Kathmandu --target 300 \
|
|
141
|
+
--dedupe-across master.csv -o new-this-week.csv
|
|
142
|
+
|
|
143
|
+
# The whole campaign saved in one file, ready for a schedule
|
|
144
|
+
npx lacspace-leads --config campaign.json
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
### When something looks wrong
|
|
148
|
+
|
|
149
|
+
| What you see | What it means | What to do |
|
|
150
|
+
| --- | --- | --- |
|
|
151
|
+
| Far fewer leads than the target | The map ran out — that really is everything Google lists for that query | More areas or cities, related `--type`s, or a bigger `--step` |
|
|
152
|
+
| About 114 back from `--limit 500` | Google's per-search ceiling | Use `--target 500` instead |
|
|
153
|
+
| No leads at all | Usually a CAPTCHA or a page change | Drop `--headless` to watch it, raise `--delay`, try again shortly |
|
|
154
|
+
| Empty address / website columns | `--no-details` was on, so listings were never opened | Remove it (`--no-website` and `--split` turn details back on for you) |
|
|
155
|
+
| Everything grouped under "other" with `--split` | Same cause — no addresses to match against | Same fix |
|
|
156
|
+
|
|
30
157
|
## Install
|
|
31
158
|
|
|
32
159
|
Needs **Node 20+** and a Chromium-based browser. It uses your installed **Google Chrome** or **Microsoft Edge** automatically; if you have neither, run:
|
|
@@ -59,7 +186,12 @@ npx lacspace-leads [type] [options]
|
|
|
59
186
|
| `--enrich-out <file>` | Also write `{ name, website, domain }` NDJSON, ready to feed [`lacspace-enrich`](https://www.npmjs.com/package/lacspace-enrich) |
|
|
60
187
|
| `--resume` | Resume an interrupted **multi-search sweep** from its checkpoint file (`<out>.checkpoint.json`) |
|
|
61
188
|
| `--sheet <name>` | Excel sheet name (default `Leads`) |
|
|
62
|
-
| `-n, --limit <n>` | Max listings **per search** (default `60
|
|
189
|
+
| `-n, --limit <n>` | Max listings **per search** (default `60`; Google itself stops near 120) |
|
|
190
|
+
| `--target <n>` | How many leads you want **in total** — keeps searching until it has them. Use this for 300, 500, 1000 |
|
|
191
|
+
| `--step <dist>` | Spacing between map tiles in a target sweep (default `2.5km`) |
|
|
192
|
+
| `--tiles <n>` | Most map tiles to try (default `49`) |
|
|
193
|
+
| `--split <key>` | Write one file per `city`, `area` or `type` alongside the main file |
|
|
194
|
+
| `--no-website` | Only businesses with **no website** — the "you need a site" pitch list |
|
|
63
195
|
| `--total <n>` | Cap the merged result when sweeping several searches |
|
|
64
196
|
| `--no-details` | Skip opening each listing — names + Maps URLs only, much faster |
|
|
65
197
|
| `--sort <key>` | `rating` · `reviews` · `name` · `priceLevel` · `distance` (missing values last) |
|
|
@@ -111,6 +243,29 @@ npx lacspace-leads [type] [options]
|
|
|
111
243
|
|
|
112
244
|
Run with no arguments for an interactive walkthrough.
|
|
113
245
|
|
|
246
|
+
## Ask for a number and actually get it (`--target`)
|
|
247
|
+
|
|
248
|
+
One Google Maps search stops at roughly **120 results**, so `--limit 500` can only ever hand back about 114. `--target` is the honest version: say how many leads you want in total and the tool keeps going until it has them.
|
|
249
|
+
|
|
250
|
+
```bash
|
|
251
|
+
# 500 restaurants, however many searches that takes
|
|
252
|
+
npx lacspace-leads restaurants --city Kathmandu --target 500 -f xlsx
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
It expands coverage in this order, stopping the moment the target is met:
|
|
256
|
+
|
|
257
|
+
1. **Every place you named** — each city x area x type is its own search.
|
|
258
|
+
2. **Map tiles** — the same query re-centred on a grid of points around the city, walking outwards from the middle. No geocoding service and no API key: the centre comes from the map itself.
|
|
259
|
+
|
|
260
|
+
Every step is de-duplicated against everything collected so far, so the number you get is unique businesses, not rows. If the map runs dry before the target, it says so plainly instead of pretending:
|
|
261
|
+
|
|
262
|
+
```
|
|
263
|
+
! The map ran out of new results at 214 — that is everything Google lists here.
|
|
264
|
+
Widen it: more areas, more cities, a bigger --step, or related --types.
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
Tune the grid with `--step` (tighter spacing finds more in dense cities) and `--tiles` (how far out to go).
|
|
268
|
+
|
|
114
269
|
## Sweep a whole city
|
|
115
270
|
|
|
116
271
|
Comma-separate areas (and/or types) and `lacspace-leads` runs each search in turn, then **merges and de-duplicates** into a single list — sorted and filtered across the whole set:
|
|
@@ -123,6 +278,15 @@ npx lacspace-leads "coffee shop" \
|
|
|
123
278
|
|
|
124
279
|
# Two business types at once, capped at 100 unique leads total
|
|
125
280
|
npx lacspace-leads --type "gym,fitness studio" --city Pokhara --total 100 -f csv
|
|
281
|
+
|
|
282
|
+
# Several cities and several areas at once — 400 in total, one Excel file per city
|
|
283
|
+
npx lacspace-leads restaurants \
|
|
284
|
+
--cities "Kathmandu, Lalitpur, Bhaktapur" \
|
|
285
|
+
--areas "Baneshwor, Thamel, Patan" \
|
|
286
|
+
--target 400 --split city -f xlsx
|
|
287
|
+
|
|
288
|
+
# Everyone in town who has no website yet
|
|
289
|
+
npx lacspace-leads "beauty salon" --city Pokhara --target 200 --no-website --has-phone -f csv
|
|
126
290
|
```
|
|
127
291
|
|
|
128
292
|
## Search by radius
|
package/dist/cli.js
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
// src/cli.ts
|
|
4
4
|
import { writeFileSync as writeFileSync2, existsSync as existsSync2, readFileSync as readFileSync2 } from "fs";
|
|
5
|
-
import { resolve as resolve2 } from "path";
|
|
5
|
+
import { resolve as resolve2, extname as extname2 } from "path";
|
|
6
6
|
import { createInterface } from "readline/promises";
|
|
7
7
|
import { stdin, stdout, stderr, argv, exit } from "process";
|
|
8
8
|
|
|
@@ -308,6 +308,7 @@ function filterLeads(leads, filters = {}) {
|
|
|
308
308
|
if (filters.minReviews !== void 0 && !(typeof l.reviews === "number" && l.reviews >= filters.minReviews)) return false;
|
|
309
309
|
if (filters.hasPhone && !l.phone) return false;
|
|
310
310
|
if (filters.hasWebsite && !l.website) return false;
|
|
311
|
+
if (filters.noWebsite && l.website) return false;
|
|
311
312
|
if (filters.hasEmail && !l.email) return false;
|
|
312
313
|
if (filters.hasValidEmail && l.emailStatus !== "valid") return false;
|
|
313
314
|
if (filters.hasContact && !(l.phone || l.email || l.website)) return false;
|
|
@@ -950,11 +951,20 @@ async function scrapeLeads(opts) {
|
|
|
950
951
|
const page = await context.newPage();
|
|
951
952
|
const urlOpts = { hl: locale };
|
|
952
953
|
if (opts.region) urlOpts.gl = opts.region;
|
|
953
|
-
if (near) urlOpts.center = { lat: near.lat, lng: near.lng, zoom: zoomForRadius(opts.radiusM ?? 2e3, near.lat) };
|
|
954
|
+
if (near) urlOpts.center = { lat: near.lat, lng: near.lng, zoom: zoomForRadius(opts.radiusM ?? opts.zoomRadiusM ?? 2e3, near.lat) };
|
|
954
955
|
await page.goto(mapsSearchUrl(query, urlOpts), { waitUntil: "domcontentloaded", timeout: 45e3 });
|
|
955
956
|
await dismissConsent(page);
|
|
956
957
|
await loadResults(page, limit, delayMs, onProgress, signal);
|
|
957
958
|
if (signal?.aborted) throw new LeadsError("Search aborted.", "ABORTED");
|
|
959
|
+
if (opts.onCenter) {
|
|
960
|
+
const at = parseLatLng(page.url());
|
|
961
|
+
if (at.latitude !== void 0 && at.longitude !== void 0) {
|
|
962
|
+
try {
|
|
963
|
+
opts.onCenter({ lat: at.latitude, lng: at.longitude });
|
|
964
|
+
} catch {
|
|
965
|
+
}
|
|
966
|
+
}
|
|
967
|
+
}
|
|
958
968
|
const cards = await page.locator("a.hfpxzc").evaluateAll(
|
|
959
969
|
(els, lim) => els.slice(0, lim).map((a) => ({
|
|
960
970
|
href: a.href,
|
|
@@ -966,6 +976,22 @@ async function scrapeLeads(opts) {
|
|
|
966
976
|
onProgress?.("no listings found (Google may have shown a CAPTCHA or an empty result).");
|
|
967
977
|
return [];
|
|
968
978
|
}
|
|
979
|
+
if (opts.skipListing) {
|
|
980
|
+
const all = cards;
|
|
981
|
+
const before = all.length;
|
|
982
|
+
const fresh = all.filter((card) => {
|
|
983
|
+
try {
|
|
984
|
+
return !opts.skipListing({ name: card.name, href: card.href });
|
|
985
|
+
} catch {
|
|
986
|
+
return true;
|
|
987
|
+
}
|
|
988
|
+
});
|
|
989
|
+
if (fresh.length !== before) {
|
|
990
|
+
onProgress?.(`${before - fresh.length} of ${before} already collected \u2014 skipping them.`);
|
|
991
|
+
all.splice(0, before, ...fresh);
|
|
992
|
+
}
|
|
993
|
+
if (all.length === 0) return [];
|
|
994
|
+
}
|
|
969
995
|
let leads = [];
|
|
970
996
|
if (!wantDetails) {
|
|
971
997
|
leads = cards.map((c2) => {
|
|
@@ -1437,6 +1463,173 @@ function leadsToEnrichInput(leads) {
|
|
|
1437
1463
|
return out;
|
|
1438
1464
|
}
|
|
1439
1465
|
|
|
1466
|
+
// src/sweep.ts
|
|
1467
|
+
var MAX_PER_SEARCH = 120;
|
|
1468
|
+
var MIN_PER_STEP = 40;
|
|
1469
|
+
var M_PER_DEG_LAT = 111320;
|
|
1470
|
+
function gridPoints(center, stepM, rings) {
|
|
1471
|
+
const step = Math.max(100, stepM);
|
|
1472
|
+
const ringCount = Math.max(0, Math.trunc(rings));
|
|
1473
|
+
const dLat = step / M_PER_DEG_LAT;
|
|
1474
|
+
const cosLat = Math.cos(center.lat * Math.PI / 180);
|
|
1475
|
+
const dLng = step / (M_PER_DEG_LAT * (Math.abs(cosLat) < 0.01 ? 0.01 : cosLat));
|
|
1476
|
+
const out = [];
|
|
1477
|
+
for (let ring = 0; ring <= ringCount; ring++) {
|
|
1478
|
+
for (let x = -ring; x <= ring; x++) {
|
|
1479
|
+
for (let y = -ring; y <= ring; y++) {
|
|
1480
|
+
if (ring > 0 && Math.max(Math.abs(x), Math.abs(y)) !== ring) continue;
|
|
1481
|
+
out.push({
|
|
1482
|
+
lat: +(center.lat + y * dLat).toFixed(6),
|
|
1483
|
+
lng: +(center.lng + x * dLng).toFixed(6)
|
|
1484
|
+
});
|
|
1485
|
+
}
|
|
1486
|
+
}
|
|
1487
|
+
}
|
|
1488
|
+
return out;
|
|
1489
|
+
}
|
|
1490
|
+
function tilesForRings(rings) {
|
|
1491
|
+
const r = Math.max(0, Math.trunc(rings));
|
|
1492
|
+
return (2 * r + 1) ** 2;
|
|
1493
|
+
}
|
|
1494
|
+
function ringsForTiles(tiles) {
|
|
1495
|
+
const want = Math.max(1, Math.trunc(tiles));
|
|
1496
|
+
let rings = 0;
|
|
1497
|
+
while (tilesForRings(rings) < want) rings++;
|
|
1498
|
+
return rings;
|
|
1499
|
+
}
|
|
1500
|
+
function planNamedSteps(base) {
|
|
1501
|
+
return expandQueries(base).map((q) => ({ kind: "search", ...q }));
|
|
1502
|
+
}
|
|
1503
|
+
async function sweepLeads(opts) {
|
|
1504
|
+
const {
|
|
1505
|
+
target,
|
|
1506
|
+
stepM = 2500,
|
|
1507
|
+
maxTiles = 49,
|
|
1508
|
+
perSearch,
|
|
1509
|
+
onNotice,
|
|
1510
|
+
near,
|
|
1511
|
+
...shared
|
|
1512
|
+
} = opts;
|
|
1513
|
+
const want = Math.max(1, Math.trunc(target));
|
|
1514
|
+
const cap = Math.min(MAX_PER_SEARCH, Math.max(1, Math.trunc(perSearch ?? MAX_PER_SEARCH)));
|
|
1515
|
+
const dedupeBy = shared.dedupe ?? "smart";
|
|
1516
|
+
let collected = [];
|
|
1517
|
+
const stats = { searches: 0, tiles: 0, unique: 0, saturated: false };
|
|
1518
|
+
const seenNames = /* @__PURE__ */ new Set();
|
|
1519
|
+
const seenUrls = /* @__PURE__ */ new Set();
|
|
1520
|
+
const remember = (leads) => {
|
|
1521
|
+
for (const l of leads) {
|
|
1522
|
+
if (l.name) seenNames.add(l.name.trim().toLowerCase());
|
|
1523
|
+
if (l.mapsUrl) seenUrls.add(l.mapsUrl);
|
|
1524
|
+
}
|
|
1525
|
+
};
|
|
1526
|
+
const skipListing = (listing) => seenUrls.has(listing.href) || !!listing.name && seenNames.has(listing.name.trim().toLowerCase());
|
|
1527
|
+
const merge = (found) => {
|
|
1528
|
+
const before = collected.length;
|
|
1529
|
+
collected = dedupeLeads([...collected, ...found], dedupeBy);
|
|
1530
|
+
remember(found);
|
|
1531
|
+
stats.unique = collected.length;
|
|
1532
|
+
return collected.length - before;
|
|
1533
|
+
};
|
|
1534
|
+
const enough = () => collected.length >= want;
|
|
1535
|
+
const stepLimit = () => Math.min(cap, Math.max(MIN_PER_STEP, (want - collected.length) * 2));
|
|
1536
|
+
let center = near ? { lat: near.lat, lng: near.lng } : void 0;
|
|
1537
|
+
const captureCenter = (c2) => {
|
|
1538
|
+
center ??= c2;
|
|
1539
|
+
};
|
|
1540
|
+
for (const step of planNamedSteps(shared)) {
|
|
1541
|
+
if (enough() || shared.signal?.aborted) break;
|
|
1542
|
+
const searchOpts = {
|
|
1543
|
+
...shared,
|
|
1544
|
+
...step,
|
|
1545
|
+
limit: stepLimit(),
|
|
1546
|
+
onCenter: captureCenter
|
|
1547
|
+
};
|
|
1548
|
+
const found = await scrapeLeads(searchOpts);
|
|
1549
|
+
stats.searches++;
|
|
1550
|
+
const added = merge(found);
|
|
1551
|
+
onNotice?.(
|
|
1552
|
+
`search ${stats.searches}: ${[step.area, step.city].filter(Boolean).join(", ") || step.query || step.type} \u2192 +${added} new (${collected.length}/${want})`
|
|
1553
|
+
);
|
|
1554
|
+
}
|
|
1555
|
+
if (!enough() && !shared.signal?.aborted) {
|
|
1556
|
+
if (!center) {
|
|
1557
|
+
onNotice?.("no map centre available, so tiling is not possible \u2014 pass --near lat,lng to sweep a radius");
|
|
1558
|
+
stats.saturated = true;
|
|
1559
|
+
} else {
|
|
1560
|
+
const points = gridPoints(center, stepM, ringsForTiles(maxTiles)).slice(0, Math.max(1, maxTiles));
|
|
1561
|
+
onNotice?.(
|
|
1562
|
+
`named searches gave ${collected.length} of ${want} \u2014 tiling the map around ${center.lat.toFixed(4)},${center.lng.toFixed(4)} every ${Math.round(stepM)}m`
|
|
1563
|
+
);
|
|
1564
|
+
let dry = 0;
|
|
1565
|
+
for (const point of points) {
|
|
1566
|
+
if (enough() || shared.signal?.aborted) break;
|
|
1567
|
+
const tileOpts = {
|
|
1568
|
+
...shared,
|
|
1569
|
+
limit: stepLimit(),
|
|
1570
|
+
near: point,
|
|
1571
|
+
onCenter: captureCenter,
|
|
1572
|
+
skipListing
|
|
1573
|
+
};
|
|
1574
|
+
delete tileOpts.area;
|
|
1575
|
+
delete tileOpts.city;
|
|
1576
|
+
if (tileOpts.radiusM === void 0) tileOpts.zoomRadiusM = stepM;
|
|
1577
|
+
const found = await scrapeLeads(tileOpts);
|
|
1578
|
+
stats.tiles++;
|
|
1579
|
+
const added = merge(found);
|
|
1580
|
+
onNotice?.(`tile ${stats.tiles}/${points.length} \u2192 +${added} new (${collected.length}/${want})`);
|
|
1581
|
+
dry = added === 0 ? dry + 1 : 0;
|
|
1582
|
+
if (dry >= 4) {
|
|
1583
|
+
stats.saturated = true;
|
|
1584
|
+
break;
|
|
1585
|
+
}
|
|
1586
|
+
}
|
|
1587
|
+
if (!enough() && !stats.saturated) stats.saturated = stats.tiles >= points.length;
|
|
1588
|
+
}
|
|
1589
|
+
}
|
|
1590
|
+
const sorted = shared.sort ? sortLeads(collected, shared.sort, shared.sortDir) : collected;
|
|
1591
|
+
return { leads: sorted.slice(0, want), stats };
|
|
1592
|
+
}
|
|
1593
|
+
|
|
1594
|
+
// src/split.ts
|
|
1595
|
+
var SPLIT_KEYS = ["city", "area", "type"];
|
|
1596
|
+
function addressMentions(address, name) {
|
|
1597
|
+
const hay = (address ?? "").toLowerCase();
|
|
1598
|
+
const needle = name.trim().toLowerCase();
|
|
1599
|
+
if (!hay || !needle) return false;
|
|
1600
|
+
const at = hay.indexOf(needle);
|
|
1601
|
+
if (at === -1) return false;
|
|
1602
|
+
const before = at === 0 ? "" : hay[at - 1];
|
|
1603
|
+
const after = hay[at + needle.length] ?? "";
|
|
1604
|
+
const boundary = (ch) => ch === "" || !/[a-z0-9]/.test(ch);
|
|
1605
|
+
return boundary(before) && boundary(after);
|
|
1606
|
+
}
|
|
1607
|
+
function groupLeads(leads, key, requested = []) {
|
|
1608
|
+
const places = requested.map((p) => p.trim()).filter(Boolean);
|
|
1609
|
+
const groupOf = (lead) => {
|
|
1610
|
+
if (key === "type") return (lead.category ?? "").trim() || "other";
|
|
1611
|
+
const hit = places.find((p) => addressMentions(lead.address, p));
|
|
1612
|
+
return hit ?? "other";
|
|
1613
|
+
};
|
|
1614
|
+
const groups = /* @__PURE__ */ new Map();
|
|
1615
|
+
for (const lead of leads) {
|
|
1616
|
+
const g = groupOf(lead);
|
|
1617
|
+
const bucket = groups.get(g);
|
|
1618
|
+
if (bucket) bucket.push(lead);
|
|
1619
|
+
else groups.set(g, [lead]);
|
|
1620
|
+
}
|
|
1621
|
+
return new Map(
|
|
1622
|
+
[...groups].sort((a, b) => {
|
|
1623
|
+
if (a[0] === "other") return 1;
|
|
1624
|
+
if (b[0] === "other") return -1;
|
|
1625
|
+
return b[1].length - a[1].length;
|
|
1626
|
+
})
|
|
1627
|
+
);
|
|
1628
|
+
}
|
|
1629
|
+
function groupSlug(name) {
|
|
1630
|
+
return name.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "") || "other";
|
|
1631
|
+
}
|
|
1632
|
+
|
|
1440
1633
|
// src/cli.ts
|
|
1441
1634
|
var SOCIAL_FIELDS = ENRICHED_FIELDS.filter((f) => f !== "email");
|
|
1442
1635
|
var OUTPUT_FORMATS = ["json", "ndjson", "csv", "xlsx"];
|
|
@@ -1465,6 +1658,7 @@ function parseArgs(list) {
|
|
|
1465
1658
|
verifyEmails: false,
|
|
1466
1659
|
hasPhone: false,
|
|
1467
1660
|
hasWebsite: false,
|
|
1661
|
+
noWebsite: false,
|
|
1468
1662
|
hasEmail: false,
|
|
1469
1663
|
hasValidEmail: false,
|
|
1470
1664
|
hasContact: false,
|
|
@@ -1491,6 +1685,11 @@ function parseArgs(list) {
|
|
|
1491
1685
|
else if (arg === "-o" || arg === "--out") a.out = next();
|
|
1492
1686
|
else if (arg === "--append") a.append = true;
|
|
1493
1687
|
else if (arg === "-n" || arg === "--limit") a.limit = parseInt(next(), 10) || a.limit;
|
|
1688
|
+
else if (arg === "--target" || arg === "--want" || arg === "--goal") a.target = parseInt(next(), 10) || a.target;
|
|
1689
|
+
else if (arg === "--step" || arg === "--tile-step") a.step = next();
|
|
1690
|
+
else if (arg === "--tiles" || arg === "--max-tiles") a.tiles = parseInt(next(), 10) || a.tiles;
|
|
1691
|
+
else if (arg === "--split" || arg === "--split-by") a.split = next();
|
|
1692
|
+
else if (arg === "--no-website" || arg === "--without-website") a.noWebsite = true;
|
|
1494
1693
|
else if (arg === "--total") a.total = parseInt(next(), 10) || a.total;
|
|
1495
1694
|
else if (arg === "--headless") a.headless = true;
|
|
1496
1695
|
else if (arg === "--no-details") a.details = false;
|
|
@@ -1564,7 +1763,14 @@ ${c("bold", "Search options")}
|
|
|
1564
1763
|
--radius <dist> Keep only leads within this of --near, e.g. 2km, 500m, 1mi
|
|
1565
1764
|
--fields <list> Columns: ${ALL_FIELDS.join(",")}
|
|
1566
1765
|
--preset <name> Field bundle: ${Object.keys(FIELD_PRESETS).join(" | ")}
|
|
1567
|
-
-n, --limit <n> Max listings per
|
|
1766
|
+
-n, --limit <n> Max listings per SEARCH (default 60, Google caps ~120)
|
|
1767
|
+
--target <n> How many leads you want IN TOTAL \u2014 keeps searching
|
|
1768
|
+
(every area you named, then tiles the map) until it has
|
|
1769
|
+
them. Use this one for 500+.
|
|
1770
|
+
--step <dist> Spacing between map tiles (default 2.5km)
|
|
1771
|
+
--tiles <n> Most map tiles to try (default 49)
|
|
1772
|
+
--split <key> Write one file per city | area | type
|
|
1773
|
+
--no-website Only businesses with NO website (the pitch list)
|
|
1568
1774
|
--total <n> Cap the merged result (batch searches)
|
|
1569
1775
|
--no-details Names + Maps URLs only (fast, no per-listing open)
|
|
1570
1776
|
|
|
@@ -1810,12 +2016,18 @@ ${c("bold", c("magenta", "\u25C6 lacspace-leads"))} ${c("dim", "\u2014 Google Ma
|
|
|
1810
2016
|
`);
|
|
1811
2017
|
if (!args.query && !args.type && !args.yes) {
|
|
1812
2018
|
args.type = await prompt(`${c("green", "?")} Business type ${c("dim", "(e.g. restaurants)")}: `);
|
|
1813
|
-
args.city = args.city ?? await prompt(`${c("green", "?")} City ${c("dim", "(
|
|
1814
|
-
args.area = args.area ?? await prompt(`${c("green", "?")} Area ${c("dim", "(optional)")}: `);
|
|
2019
|
+
args.city = args.city ?? await prompt(`${c("green", "?")} City ${c("dim", "(one, or several: Kathmandu, Pokhara)")}: `);
|
|
2020
|
+
args.area = args.area ?? await prompt(`${c("green", "?")} Area ${c("dim", "(optional, several allowed: Baneshwor, Thamel)")}: `);
|
|
1815
2021
|
const fmt = await prompt(`${c("green", "?")} Format ${c("dim", "(json/csv/xlsx)")} ${c("dim", "[json]")}: `, "json");
|
|
1816
2022
|
if (fmt === "csv" || fmt === "xlsx" || fmt === "json") args.format = fmt;
|
|
1817
|
-
const lim = await prompt(`${c("green", "?")} How many ${c("dim", "[60]")}: `, "60");
|
|
1818
|
-
|
|
2023
|
+
const lim = await prompt(`${c("green", "?")} How many leads in total ${c("dim", "[60]")}: `, "60");
|
|
2024
|
+
const wanted2 = parseInt(lim, 10) || args.limit;
|
|
2025
|
+
if (wanted2 > MAX_PER_SEARCH) {
|
|
2026
|
+
args.target = wanted2;
|
|
2027
|
+
log(` ${c("dim", `${wanted2} is more than one search returns, so I'll sweep every area, then tile the map.`)}`);
|
|
2028
|
+
} else {
|
|
2029
|
+
args.limit = wanted2;
|
|
2030
|
+
}
|
|
1819
2031
|
const em = await prompt(`${c("green", "?")} Also find emails from websites? ${c("dim", "(slower) [y/N]")} `);
|
|
1820
2032
|
if (/^y/i.test(em)) args.emails = true;
|
|
1821
2033
|
}
|
|
@@ -1841,6 +2053,12 @@ ${c("bold", c("magenta", "\u25C6 lacspace-leads"))} ${c("dim", "\u2014 Google Ma
|
|
|
1841
2053
|
return;
|
|
1842
2054
|
}
|
|
1843
2055
|
const nearPoint = args.near ? parseLatLngPair(args.near) : void 0;
|
|
2056
|
+
const tileStepM = args.step ? parseDistance(args.step) : void 0;
|
|
2057
|
+
if (!args.details && (args.split || args.noWebsite)) {
|
|
2058
|
+
const needs = [args.split ? `--split ${args.split}` : "", args.noWebsite ? "--no-website" : ""].filter(Boolean).join(" and ");
|
|
2059
|
+
log(c("yellow", ` ! ${needs} needs each listing's details, but --no-details is on \u2014 turning details back on.`));
|
|
2060
|
+
args.details = true;
|
|
2061
|
+
}
|
|
1844
2062
|
if (args.near && !nearPoint) {
|
|
1845
2063
|
log(c("red", `
|
|
1846
2064
|
\u2717 --near must be "lat,lng", e.g. --near "27.7172,85.3240".`));
|
|
@@ -1884,6 +2102,7 @@ ${c("bold", c("magenta", "\u25C6 lacspace-leads"))} ${c("dim", "\u2014 Google Ma
|
|
|
1884
2102
|
if (args.minReviews !== void 0) filters.minReviews = args.minReviews;
|
|
1885
2103
|
if (args.hasPhone) filters.hasPhone = true;
|
|
1886
2104
|
if (args.hasWebsite) filters.hasWebsite = true;
|
|
2105
|
+
if (args.noWebsite) filters.noWebsite = true;
|
|
1887
2106
|
if (args.hasEmail) filters.hasEmail = true;
|
|
1888
2107
|
if (args.hasValidEmail) filters.hasValidEmail = true;
|
|
1889
2108
|
if (args.hasContact) filters.hasContact = true;
|
|
@@ -1987,8 +2206,22 @@ ${c("yellow", "!")} This opens a browser and searches Google Maps. Continue? ${c
|
|
|
1987
2206
|
}
|
|
1988
2207
|
}
|
|
1989
2208
|
let leads;
|
|
2209
|
+
let sweptShort = false;
|
|
1990
2210
|
try {
|
|
1991
|
-
if (
|
|
2211
|
+
if (args.target !== void 0) {
|
|
2212
|
+
const sweepOpts = {
|
|
2213
|
+
...opts,
|
|
2214
|
+
target: args.target,
|
|
2215
|
+
onNotice: (m) => log(` ${c("cyan", "\u25F7")} ${c("dim", m)}`)
|
|
2216
|
+
};
|
|
2217
|
+
if (tileStepM !== void 0) sweepOpts.stepM = tileStepM;
|
|
2218
|
+
if (args.tiles !== void 0) sweepOpts.maxTiles = args.tiles;
|
|
2219
|
+
if (args.limit !== 60) sweepOpts.perSearch = args.limit;
|
|
2220
|
+
const swept = await sweepLeads(sweepOpts);
|
|
2221
|
+
leads = swept.leads;
|
|
2222
|
+
sweptShort = swept.stats.saturated && leads.length < args.target;
|
|
2223
|
+
log(` ${c("cyan", "\u25F7")} ${c("dim", `${swept.stats.searches} search${swept.stats.searches === 1 ? "" : "es"} + ${swept.stats.tiles} map tile${swept.stats.tiles === 1 ? "" : "s"} \u2192 ${swept.stats.unique} unique`)}`);
|
|
2224
|
+
} else if (isBatch) {
|
|
1992
2225
|
const batchOpts = { ...opts };
|
|
1993
2226
|
if (args.total !== void 0) batchOpts.total = args.total;
|
|
1994
2227
|
if (checkpoint && cpFile) {
|
|
@@ -2013,6 +2246,16 @@ ${c("yellow", "!")} This opens a browser and searches Google Maps. Continue? ${c
|
|
|
2013
2246
|
} finally {
|
|
2014
2247
|
process.removeListener("SIGINT", onSig);
|
|
2015
2248
|
}
|
|
2249
|
+
if (sweptShort) {
|
|
2250
|
+
log(c("yellow", `
|
|
2251
|
+
! The map ran out of new results at ${leads.length} \u2014 that is everything Google lists here.`));
|
|
2252
|
+
log(c("dim", " Widen it: more areas (--areas a,b,c), more cities (--cities x,y), a bigger --step, or related --types.\n"));
|
|
2253
|
+
} else if (args.target === void 0 && args.limit > leads.length && leads.length >= MAX_PER_SEARCH - 20) {
|
|
2254
|
+
log(c("yellow", `
|
|
2255
|
+
! Google stops a single search at about ${MAX_PER_SEARCH} results, so --limit ${args.limit} could only return ${leads.length}.`));
|
|
2256
|
+
log(c("dim", ` To really get ${args.limit}, ask for a total: --target ${args.limit} \u2014 it sweeps every area you name, then tiles the map.
|
|
2257
|
+
`));
|
|
2258
|
+
}
|
|
2016
2259
|
if (leads.length === 0) {
|
|
2017
2260
|
log(c("yellow", "\n No leads collected. Try a broader area, looser filters, or a smaller --limit.\n"));
|
|
2018
2261
|
if (cpFile) clearCheckpoint(cpFile);
|
|
@@ -2072,6 +2315,24 @@ ${c("yellow", "!")} This opens a browser and searches Google Maps. Continue? ${c
|
|
|
2072
2315
|
printSummary();
|
|
2073
2316
|
return;
|
|
2074
2317
|
}
|
|
2318
|
+
if (args.split) {
|
|
2319
|
+
const key = args.split.trim().toLowerCase();
|
|
2320
|
+
if (!SPLIT_KEYS.includes(key)) {
|
|
2321
|
+
log(c("yellow", ` ! --split "${args.split}" is not one of ${SPLIT_KEYS.join(", ")} \u2014 writing a single file instead.`));
|
|
2322
|
+
} else {
|
|
2323
|
+
const requested = (key === "area" ? args.area : key === "city" ? args.city : "")?.split(",").map((x) => x.trim()).filter(Boolean) ?? [];
|
|
2324
|
+
const groups = groupLeads(leads, key, requested);
|
|
2325
|
+
const ext = extname2(out) || `.${args.format}`;
|
|
2326
|
+
const stem = out.slice(0, out.length - ext.length);
|
|
2327
|
+
log("");
|
|
2328
|
+
for (const [group, rows] of groups) {
|
|
2329
|
+
const file = `${stem}-${groupSlug(group)}${ext}`;
|
|
2330
|
+
const part = serialize(rows, args.format, fields, serOpts);
|
|
2331
|
+
writeFileSync2(file, part.binary ? Buffer.from(part.data) : part.data);
|
|
2332
|
+
log(` ${c("green", "\u2714")} ${c("bold", String(rows.length))} leads \u2192 ${c("cyan", file)}`);
|
|
2333
|
+
}
|
|
2334
|
+
}
|
|
2335
|
+
}
|
|
2075
2336
|
writeFileSync2(out, binary ? Buffer.from(data) : data);
|
|
2076
2337
|
if (cpFile) clearCheckpoint(cpFile);
|
|
2077
2338
|
const s = computeStats(leads);
|