lacspace-leads 1.2.1 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +89 -13
- package/dist/cli.js +135 -9
- package/dist/lib.cjs +104 -8
- package/dist/lib.d.cts +54 -7
- package/dist/lib.d.ts +54 -7
- package/dist/lib.js +99 -7
- package/package.json +4 -2
package/README.md
CHANGED
|
@@ -12,6 +12,7 @@ That opens a browser, searches Maps for *"restaurants in Baneshwor, Kathmandu"*,
|
|
|
12
12
|
|
|
13
13
|
- **Free & keyless** — uses a real browser (via [Playwright](https://playwright.dev)), not a paid Places API.
|
|
14
14
|
- **Sweep a whole city** — comma-separate areas and it runs each search, then **merges and de-duplicates** into one list: `--area "Thamel,Baneshwor,Patan"`.
|
|
15
|
+
- **Search by radius** — centre on a coordinate and keep only what's within range: `--near "27.72,85.32" --radius 2km`. Each lead gets a `distanceKm`, sorted nearest-first.
|
|
15
16
|
- **Accumulate a master list** — `--append` merges each run into your existing file and de-duplicates, so daily runs build one clean database.
|
|
16
17
|
- **Rich enrichment** — visit each website to pull an **email** and links for **Facebook, Instagram, WhatsApp, LinkedIn, X, YouTube, TikTok and Telegram** — a few sites in parallel.
|
|
17
18
|
- **Verified emails** — `--verify-emails` checks each address's domain has **MX records** (no message sent) and tags it `valid`/`no-mx`; `--has-valid-email` keeps only deliverable ones.
|
|
@@ -42,7 +43,9 @@ npx lacspace-leads [type] [options]
|
|
|
42
43
|
| `--city <text>` | City, e.g. `Kathmandu`. Comma-separate for several. |
|
|
43
44
|
| `--area <text>` | Area / neighbourhood, e.g. `Baneshwor`. Comma-separate to **sweep a whole city**. |
|
|
44
45
|
| `-q, --query <text>` | Raw query, used verbatim (overrides city/area/type) |
|
|
45
|
-
| `--
|
|
46
|
+
| `--near <lat,lng>` | Centre the search on a coordinate (radius search) |
|
|
47
|
+
| `--radius <dist>` | Keep only leads within this of `--near`, e.g. `2km`, `500m`, `1mi` |
|
|
48
|
+
| `--fields <list>` | Columns: `name,category,rating,reviews,priceLevel,address,phone,website,email,facebook,instagram,whatsapp,linkedin,twitter,youtube,tiktok,telegram,emailStatus,plusCode,latitude,longitude,distanceKm,hours,mapsUrl` |
|
|
46
49
|
| `--preset <name>` | Field bundle: `minimal` · `outreach` · `contact` · `geo` · `full` · `everything` |
|
|
47
50
|
| `-f, --format <fmt>` | `json` · `ndjson` · `csv` · `xlsx` (default `json`) |
|
|
48
51
|
| `-o, --out <file>` | Output file, or `-` for **stdout** (default: a slug + date) |
|
|
@@ -51,8 +54,8 @@ npx lacspace-leads [type] [options]
|
|
|
51
54
|
| `-n, --limit <n>` | Max listings **per search** (default `60`) |
|
|
52
55
|
| `--total <n>` | Cap the merged result when sweeping several searches |
|
|
53
56
|
| `--no-details` | Skip opening each listing — names + Maps URLs only, much faster |
|
|
54
|
-
| `--sort <key>` | `rating` · `reviews` · `name` · `priceLevel` (missing values last) |
|
|
55
|
-
| `--desc` / `--asc` | Sort direction (default: `desc` for
|
|
57
|
+
| `--sort <key>` | `rating` · `reviews` · `name` · `priceLevel` · `distance` (missing values last) |
|
|
58
|
+
| `--desc` / `--asc` | Sort direction (default: `desc` for quality keys, `asc` for name/distance) |
|
|
56
59
|
| `--delay <ms>` | Pause between listings (default `700`) |
|
|
57
60
|
| `--jitter` | Randomise the delay ±40% (more human) |
|
|
58
61
|
| `--retries <n>` | Retry a listing that fails to open (default `1`) |
|
|
@@ -108,6 +111,17 @@ npx lacspace-leads "coffee shop" \
|
|
|
108
111
|
npx lacspace-leads --type "gym,fitness studio" --city Pokhara --total 100 -f csv
|
|
109
112
|
```
|
|
110
113
|
|
|
114
|
+
## Search by radius
|
|
115
|
+
|
|
116
|
+
Centre the search on a coordinate and keep only what's within range — precise catchment-area targeting for delivery zones, field sales or store-radius research:
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
# Restaurants within 2 km of a point, nearest first, with a distance column
|
|
120
|
+
npx lacspace-leads restaurants --near "27.7172,85.3240" --radius 2km -f csv
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Every kept lead gains a `distanceKm` column and results are sorted **nearest-first** by default (override with `--sort`). `--radius` accepts `km`, `m` or `mi` (a bare number is metres); omit it to just centre the search without a hard cutoff. Under the hood it points Google Maps at the coordinate, then filters by real great-circle (haversine) distance.
|
|
124
|
+
|
|
111
125
|
## Field presets
|
|
112
126
|
|
|
113
127
|
Skip spelling out `--fields` with a ready-made bundle:
|
|
@@ -140,29 +154,90 @@ npx lacspace-leads cafes --city Kathmandu --area "Thamel,Baneshwor,Patan" \
|
|
|
140
154
|
|
|
141
155
|
`--verify-emails` does a DNS **MX lookup** on each email's domain (no message is sent) and adds an `emailStatus` column of `valid` · `no-mx` · `invalid-format`. `--append` reads the existing file back, merges, and de-duplicates with the **`smart`** key (website → phone → name) so even website-less businesses don't pile up on re-runs.
|
|
142
156
|
|
|
143
|
-
## Convert
|
|
157
|
+
## Convert your leads to any format
|
|
144
158
|
|
|
145
|
-
A general converter is built in — it works on any tabular file, not just leads:
|
|
159
|
+
Collected leads as JSON and now need them in Excel? A general converter is built in — it reads and writes **JSON · NDJSON · CSV · Excel** in any direction, and works on *any* tabular file, not just leads:
|
|
146
160
|
|
|
147
161
|
```bash
|
|
148
|
-
npx lacspace-leads convert leads.json
|
|
149
|
-
npx lacspace-leads convert
|
|
150
|
-
npx lacspace-leads convert
|
|
162
|
+
npx lacspace-leads convert leads.json -f xlsx # JSON → Excel
|
|
163
|
+
npx lacspace-leads convert leads.json -o leads.csv # JSON → CSV (format from the -o extension)
|
|
164
|
+
npx lacspace-leads convert data.csv -o data.json # CSV → JSON
|
|
165
|
+
npx lacspace-leads convert sheet.xlsx -f csv # Excel → CSV
|
|
166
|
+
npx lacspace-leads convert leads.ndjson -f xlsx # NDJSON → Excel
|
|
167
|
+
npx lacspace-leads convert big.csv -f ndjson # CSV → NDJSON (stream-friendly)
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
The output format comes from `-f` (or is inferred from the `-o` filename's extension); if you give neither, it defaults to JSON. Use `--sheet "My Leads"` to name the Excel tab.
|
|
171
|
+
|
|
172
|
+
**What converts to what** — every combination works, both ways:
|
|
173
|
+
|
|
174
|
+
| From \ To | JSON | NDJSON | CSV | Excel |
|
|
175
|
+
| --- | :---: | :---: | :---: | :---: |
|
|
176
|
+
| **JSON** | – | ✓ | ✓ | ✓ |
|
|
177
|
+
| **NDJSON** | ✓ | – | ✓ | ✓ |
|
|
178
|
+
| **CSV** | ✓ | ✓ | – | ✓ |
|
|
179
|
+
| **Excel** | ✓ | ✓ | ✓ | – |
|
|
180
|
+
|
|
181
|
+
**Sample.** `leads.json` in:
|
|
182
|
+
|
|
183
|
+
```json
|
|
184
|
+
[
|
|
185
|
+
{ "name": "Himalayan Java", "phone": "+9779801234567", "rating": 4.6, "website": "https://himalayanjava.com" },
|
|
186
|
+
{ "name": "Cafe Soma", "phone": "+9779807654321", "rating": 4.4 }
|
|
187
|
+
]
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
`npx lacspace-leads convert leads.json -o leads.csv` → `leads.csv` out (missing cells become blank; `+` is escaped so spreadsheets don't treat it as a formula):
|
|
191
|
+
|
|
192
|
+
```csv
|
|
193
|
+
name,phone,rating,website
|
|
194
|
+
Himalayan Java,'+9779801234567,4.6,https://himalayanjava.com
|
|
195
|
+
Cafe Soma,'+9779807654321,4.4,
|
|
151
196
|
```
|
|
152
197
|
|
|
153
|
-
|
|
198
|
+
**In code** — convert a file, or turn any `Lead[]` into a downloadable buffer:
|
|
199
|
+
|
|
200
|
+
```ts
|
|
201
|
+
import { convertFile, readRows, serializeRows, serialize } from "lacspace-leads";
|
|
154
202
|
|
|
155
|
-
|
|
203
|
+
// 1. Convert a file on disk (returns { out, format, count }).
|
|
204
|
+
await convertFile("leads.json", { format: "xlsx", sheetName: "Prospects" });
|
|
205
|
+
|
|
206
|
+
// 2. Read rows from any format, transform, write to another.
|
|
207
|
+
const rows = await readRows("leads.csv"); // → array of objects
|
|
208
|
+
const filtered = rows.filter((r) => Number(r.Rating) >= 4.5);
|
|
209
|
+
const { data, binary } = serializeRows(filtered, "xlsx"); // → bytes for xlsx
|
|
210
|
+
// (write `data` yourself, or in a server route send it as a download)
|
|
211
|
+
|
|
212
|
+
// 3. Serialize Lead[] straight from a search — pick the columns and order.
|
|
213
|
+
const { data: csv } = serialize(leads, "csv", ["name", "phone", "email"]);
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
## Recipes
|
|
156
217
|
|
|
157
218
|
```bash
|
|
158
219
|
# Excel of restaurants in a specific area
|
|
159
220
|
npx lacspace-leads restaurants --city Kathmandu --area Baneshwor -f xlsx
|
|
160
221
|
|
|
161
|
-
#
|
|
162
|
-
npx lacspace-leads
|
|
222
|
+
# Outreach list: name/phone/email/website/address, MX-verified, deliverable only
|
|
223
|
+
npx lacspace-leads "dental clinic" --city Pokhara --preset outreach \
|
|
224
|
+
--verify-emails --has-valid-email --country NP -f csv
|
|
225
|
+
|
|
226
|
+
# Cover a whole city, sorted best-reviewed first, into one Excel file
|
|
227
|
+
npx lacspace-leads cafes --city Kathmandu --area "Thamel,Baneshwor,Patan" \
|
|
228
|
+
--sort reviews --desc -f xlsx
|
|
229
|
+
|
|
230
|
+
# Everything within 1.5 km of a point, nearest first
|
|
231
|
+
npx lacspace-leads salons --near "27.7172,85.3240" --radius 1.5km -f csv
|
|
232
|
+
|
|
233
|
+
# Build a master list you top up daily (accumulate + dedupe)
|
|
234
|
+
npx lacspace-leads gyms --city Lalitpur -o gyms-master.xlsx --append
|
|
163
235
|
|
|
164
|
-
# Fast name + URL sweep, no per-listing opening
|
|
236
|
+
# Fast name + Maps-URL sweep, no per-listing opening
|
|
165
237
|
npx lacspace-leads gyms --city Lalitpur --no-details -n 100
|
|
238
|
+
|
|
239
|
+
# Pipe straight into jq (JSON on stdout, logs on stderr)
|
|
240
|
+
npx lacspace-leads bakeries --city Pokhara -f json -o - -y | jq '.[].phone'
|
|
166
241
|
```
|
|
167
242
|
|
|
168
243
|
## Library
|
|
@@ -213,6 +288,7 @@ const leads = await searchLeadsBatch(
|
|
|
213
288
|
| `toRows(leads, fields?)` | Header-keyed rows, for your own exporter. |
|
|
214
289
|
| `enrichContacts(website)` / `extractEmails` / `extractSocials` | Website enrichment, on tap. |
|
|
215
290
|
| `cleanWebsite` / `normalizePhone` / `sortLeads` | Pure data-cleaning helpers (unit-tested). |
|
|
291
|
+
| `haversineMeters` / `parseLatLngPair` / `parseDistance` | Pure geo helpers for radius search. |
|
|
216
292
|
| `filterLeads` / `dedupeLeads` | Pure post-processing over any `Lead[]`. |
|
|
217
293
|
| `expandQueries` / `resolvePreset` / `FIELD_PRESETS` | Batch expansion + field presets. |
|
|
218
294
|
| `composeQuery` / `mapsSearchUrl` / `normalizeFields` / `defaultFilename` | Query + helper utilities. |
|
package/dist/cli.js
CHANGED
|
@@ -32,9 +32,11 @@ var ALL_FIELDS = [
|
|
|
32
32
|
"plusCode",
|
|
33
33
|
"latitude",
|
|
34
34
|
"longitude",
|
|
35
|
+
"distanceKm",
|
|
35
36
|
"hours",
|
|
36
37
|
"mapsUrl"
|
|
37
38
|
];
|
|
39
|
+
var DERIVED_FIELDS = ["distanceKm"];
|
|
38
40
|
var ENRICHED_FIELDS = [
|
|
39
41
|
"email",
|
|
40
42
|
"facebook",
|
|
@@ -57,11 +59,13 @@ var FIELD_PRESETS = {
|
|
|
57
59
|
/** Map/geo columns for plotting. */
|
|
58
60
|
geo: ["name", "address", "latitude", "longitude", "plusCode", "mapsUrl"],
|
|
59
61
|
/** Everything Maps shows, no website enrichment. */
|
|
60
|
-
full: ALL_FIELDS.filter((f) => !ENRICHED_FIELDS.includes(f)),
|
|
62
|
+
full: ALL_FIELDS.filter((f) => !ENRICHED_FIELDS.includes(f) && !DERIVED_FIELDS.includes(f)),
|
|
61
63
|
/** Every field, including enriched ones. */
|
|
62
64
|
everything: [...ALL_FIELDS]
|
|
63
65
|
};
|
|
64
|
-
var DEFAULT_FIELDS = ALL_FIELDS.filter(
|
|
66
|
+
var DEFAULT_FIELDS = ALL_FIELDS.filter(
|
|
67
|
+
(f) => !ENRICHED_FIELDS.includes(f) && !DERIVED_FIELDS.includes(f)
|
|
68
|
+
);
|
|
65
69
|
|
|
66
70
|
// src/query.ts
|
|
67
71
|
function composeQuery(opts) {
|
|
@@ -75,7 +79,9 @@ function mapsSearchUrl(query, opts = {}) {
|
|
|
75
79
|
const hl = (opts.hl ?? "en").split("-")[0] || "en";
|
|
76
80
|
const params = new URLSearchParams({ hl });
|
|
77
81
|
if (opts.gl) params.set("gl", opts.gl.toLowerCase());
|
|
78
|
-
|
|
82
|
+
const base = `https://www.google.com/maps/search/${encodeURIComponent(query)}`;
|
|
83
|
+
const at = opts.center ? `/@${opts.center.lat},${opts.center.lng},${Math.round(opts.center.zoom ?? 14)}z` : "";
|
|
84
|
+
return `${base}${at}?${params.toString()}`;
|
|
79
85
|
}
|
|
80
86
|
function expandQueries(opts) {
|
|
81
87
|
if (opts.query && opts.query.trim()) return [{ query: opts.query.trim() }];
|
|
@@ -375,12 +381,14 @@ function plausible(digits) {
|
|
|
375
381
|
return digits.length >= 8 && digits.length <= 15;
|
|
376
382
|
}
|
|
377
383
|
var priceRank = (p) => p ? p.replace(/[^$€£₹¥₩]/g, "").length : 0;
|
|
384
|
+
var ASC_BY_DEFAULT = /* @__PURE__ */ new Set(["name", "distance"]);
|
|
378
385
|
function sortLeads(leads, by, dir) {
|
|
379
386
|
if (!by) return [...leads];
|
|
380
|
-
const desc = dir ? dir === "desc" : by
|
|
387
|
+
const desc = dir ? dir === "desc" : !ASC_BY_DEFAULT.has(by);
|
|
381
388
|
const val = (l) => {
|
|
382
389
|
if (by === "name") return l.name?.toLowerCase();
|
|
383
390
|
if (by === "priceLevel") return l.priceLevel ? priceRank(l.priceLevel) : void 0;
|
|
391
|
+
if (by === "distance") return l.distanceKm;
|
|
384
392
|
return l[by];
|
|
385
393
|
};
|
|
386
394
|
return [...leads].map((lead, i) => ({ lead, i })).sort((a, b) => {
|
|
@@ -448,6 +456,43 @@ async function verifyEmails(leads, opts = {}) {
|
|
|
448
456
|
return leads;
|
|
449
457
|
}
|
|
450
458
|
|
|
459
|
+
// src/geo.ts
|
|
460
|
+
var R_EARTH_M = 63710088e-1;
|
|
461
|
+
var toRad = (deg) => deg * Math.PI / 180;
|
|
462
|
+
function haversineMeters(a, b) {
|
|
463
|
+
const dLat = toRad(b.lat - a.lat);
|
|
464
|
+
const dLng = toRad(b.lng - a.lng);
|
|
465
|
+
const lat1 = toRad(a.lat);
|
|
466
|
+
const lat2 = toRad(b.lat);
|
|
467
|
+
const h = Math.sin(dLat / 2) ** 2 + Math.cos(lat1) * Math.cos(lat2) * Math.sin(dLng / 2) ** 2;
|
|
468
|
+
return 2 * R_EARTH_M * Math.asin(Math.min(1, Math.sqrt(h)));
|
|
469
|
+
}
|
|
470
|
+
function parseLatLngPair(input) {
|
|
471
|
+
if (!input) return void 0;
|
|
472
|
+
const m = input.replace(/^@/, "").match(/(-?\d+(?:\.\d+)?)\s*[, ]\s*(-?\d+(?:\.\d+)?)/);
|
|
473
|
+
if (!m) return void 0;
|
|
474
|
+
const lat = parseFloat(m[1]);
|
|
475
|
+
const lng = parseFloat(m[2]);
|
|
476
|
+
if (!Number.isFinite(lat) || !Number.isFinite(lng)) return void 0;
|
|
477
|
+
if (lat < -90 || lat > 90 || lng < -180 || lng > 180) return void 0;
|
|
478
|
+
return { lat, lng };
|
|
479
|
+
}
|
|
480
|
+
function parseDistance(input) {
|
|
481
|
+
if (!input) return void 0;
|
|
482
|
+
const m = input.trim().toLowerCase().match(/^(\d+(?:\.\d+)?)\s*(km|m|mi|mile|miles|meters?|metres?)?$/);
|
|
483
|
+
if (!m) return void 0;
|
|
484
|
+
const value = parseFloat(m[1]);
|
|
485
|
+
if (!Number.isFinite(value) || value <= 0) return void 0;
|
|
486
|
+
const unit = m[2] ?? "m";
|
|
487
|
+
const metres = unit === "km" ? value * 1e3 : unit.startsWith("mi") ? value * 1609.344 : value;
|
|
488
|
+
return metres;
|
|
489
|
+
}
|
|
490
|
+
function zoomForRadius(radiusM, lat = 0) {
|
|
491
|
+
const mpp = 2 * Math.max(50, radiusM) / 600;
|
|
492
|
+
const zoom = Math.log2(156543.03392 * Math.cos(toRad(lat)) / mpp);
|
|
493
|
+
return Math.max(3, Math.min(19, Math.round(zoom)));
|
|
494
|
+
}
|
|
495
|
+
|
|
451
496
|
// src/export.ts
|
|
452
497
|
import { stringify as csvStringify } from "@lacspace/csv";
|
|
453
498
|
import { jsonToXlsx } from "@lacspace/xlsx";
|
|
@@ -473,6 +518,7 @@ var HEADERS = {
|
|
|
473
518
|
plusCode: "Plus Code",
|
|
474
519
|
latitude: "Latitude",
|
|
475
520
|
longitude: "Longitude",
|
|
521
|
+
distanceKm: "Distance (km)",
|
|
476
522
|
hours: "Hours",
|
|
477
523
|
mapsUrl: "Maps URL"
|
|
478
524
|
};
|
|
@@ -533,7 +579,7 @@ var FIELD_BY_KEY = (() => {
|
|
|
533
579
|
}
|
|
534
580
|
return m;
|
|
535
581
|
})();
|
|
536
|
-
var NUMERIC_FIELDS = /* @__PURE__ */ new Set(["rating", "reviews", "latitude", "longitude"]);
|
|
582
|
+
var NUMERIC_FIELDS = /* @__PURE__ */ new Set(["rating", "reviews", "latitude", "longitude", "distanceKm"]);
|
|
537
583
|
function rowsToLeads(rows) {
|
|
538
584
|
return rows.map((row) => {
|
|
539
585
|
const lead = {};
|
|
@@ -744,8 +790,13 @@ async function scrapeLeads(opts) {
|
|
|
744
790
|
const pauseMs = () => opts.jitter ? Math.round(delayMs * (0.6 + Math.random() * 0.8)) : delayMs;
|
|
745
791
|
const wantVerify = Boolean(opts.verifyEmails) || Boolean(opts.filters?.hasValidEmail) || fields.has("emailStatus");
|
|
746
792
|
const wantEnrich = Boolean(opts.enrich) || ENRICHED_FIELDS.some((f) => fields.has(f)) || Boolean(opts.filters?.hasEmail) || wantVerify;
|
|
793
|
+
const near = opts.near;
|
|
747
794
|
const collect = new Set(fields);
|
|
748
795
|
if (wantEnrich) collect.add("website");
|
|
796
|
+
if (near) {
|
|
797
|
+
collect.add("latitude");
|
|
798
|
+
collect.add("longitude");
|
|
799
|
+
}
|
|
749
800
|
const detailOnly = ["name", "mapsUrl"];
|
|
750
801
|
const wantDetails = (opts.details ?? true) && ([...collect].some((f) => !detailOnly.includes(f)) || wantEnrich);
|
|
751
802
|
onProgress?.(`searching Google Maps for "${query}"\u2026`);
|
|
@@ -759,6 +810,7 @@ async function scrapeLeads(opts) {
|
|
|
759
810
|
const page = await context.newPage();
|
|
760
811
|
const urlOpts = { hl: locale };
|
|
761
812
|
if (opts.region) urlOpts.gl = opts.region;
|
|
813
|
+
if (near) urlOpts.center = { lat: near.lat, lng: near.lng, zoom: zoomForRadius(opts.radiusM ?? 2e3, near.lat) };
|
|
762
814
|
await page.goto(mapsSearchUrl(query, urlOpts), { waitUntil: "domcontentloaded", timeout: 45e3 });
|
|
763
815
|
await dismissConsent(page);
|
|
764
816
|
await loadResults(page, limit, delayMs, onProgress, signal);
|
|
@@ -836,6 +888,19 @@ async function scrapeLeads(opts) {
|
|
|
836
888
|
if (lead.website) lead.website = cleanWebsite(lead.website);
|
|
837
889
|
}
|
|
838
890
|
}
|
|
891
|
+
if (near) {
|
|
892
|
+
for (const lead of leads) {
|
|
893
|
+
if (typeof lead.latitude === "number" && typeof lead.longitude === "number") {
|
|
894
|
+
const m = haversineMeters(near, { lat: lead.latitude, lng: lead.longitude });
|
|
895
|
+
lead.distanceKm = Math.round(m / 1e3 * 100) / 100;
|
|
896
|
+
}
|
|
897
|
+
}
|
|
898
|
+
if (opts.radiusM !== void 0) {
|
|
899
|
+
const r = opts.radiusM;
|
|
900
|
+
leads = leads.filter((l) => l.distanceKm !== void 0 && l.distanceKm * 1e3 <= r);
|
|
901
|
+
onProgress?.(`within ${Math.round(r)} m: ${leads.length} listing${leads.length === 1 ? "" : "s"}.`);
|
|
902
|
+
}
|
|
903
|
+
}
|
|
839
904
|
leads = dedupeLeads(leads, opts.dedupe ?? "website");
|
|
840
905
|
if (wantEnrich) {
|
|
841
906
|
const socialFields = ENRICHED_FIELDS.filter((f) => f !== "email");
|
|
@@ -876,6 +941,16 @@ async function scrapeLeads(opts) {
|
|
|
876
941
|
if (opts.filters) leads = filterLeads(leads, opts.filters);
|
|
877
942
|
if (opts.sort) leads = sortLeads(leads, opts.sort, opts.sortDir);
|
|
878
943
|
if (wantVerify && !fields.has("emailStatus")) for (const l of leads) delete l.emailStatus;
|
|
944
|
+
if (near) {
|
|
945
|
+
const keepLat = fields.has("latitude");
|
|
946
|
+
const keepLng = fields.has("longitude");
|
|
947
|
+
const keepDist = fields.has("distanceKm");
|
|
948
|
+
for (const l of leads) {
|
|
949
|
+
if (!keepLat) delete l.latitude;
|
|
950
|
+
if (!keepLng) delete l.longitude;
|
|
951
|
+
if (!keepDist) delete l.distanceKm;
|
|
952
|
+
}
|
|
953
|
+
}
|
|
879
954
|
onProgress?.(`collected ${leads.length} lead${leads.length === 1 ? "" : "s"}.`);
|
|
880
955
|
return leads;
|
|
881
956
|
} finally {
|
|
@@ -946,9 +1021,10 @@ import { jsonToXlsx as jsonToXlsx2, xlsxToJson } from "@lacspace/xlsx";
|
|
|
946
1021
|
function detectFormat(file) {
|
|
947
1022
|
const ext = extname(file).toLowerCase().replace(/^\./, "");
|
|
948
1023
|
if (ext === "json") return "json";
|
|
1024
|
+
if (ext === "ndjson" || ext === "jsonl") return "ndjson";
|
|
949
1025
|
if (ext === "csv" || ext === "tsv") return "csv";
|
|
950
1026
|
if (ext === "xlsx" || ext === "xls") return "xlsx";
|
|
951
|
-
throw new Error(`Cannot infer a format from ".${ext}" \u2014 pass an explicit --format (json|csv|xlsx).`);
|
|
1027
|
+
throw new Error(`Cannot infer a format from ".${ext}" \u2014 pass an explicit --format (json|ndjson|csv|xlsx).`);
|
|
952
1028
|
}
|
|
953
1029
|
async function readRows(file, format) {
|
|
954
1030
|
const fmt = format ?? detectFormat(file);
|
|
@@ -960,6 +1036,15 @@ async function readRows(file, format) {
|
|
|
960
1036
|
if (fmt === "csv") {
|
|
961
1037
|
return csvParse(text, { header: true });
|
|
962
1038
|
}
|
|
1039
|
+
if (fmt === "ndjson") {
|
|
1040
|
+
return text.split(/\r?\n/).map((l) => l.trim()).filter(Boolean).map((l, i) => {
|
|
1041
|
+
try {
|
|
1042
|
+
return JSON.parse(l);
|
|
1043
|
+
} catch {
|
|
1044
|
+
throw new Error(`Invalid NDJSON on line ${i + 1}.`);
|
|
1045
|
+
}
|
|
1046
|
+
});
|
|
1047
|
+
}
|
|
963
1048
|
const parsed = JSON.parse(text);
|
|
964
1049
|
if (Array.isArray(parsed)) return parsed;
|
|
965
1050
|
if (parsed && typeof parsed === "object") return [parsed];
|
|
@@ -982,6 +1067,9 @@ function serializeRows(rows, format, opts = {}) {
|
|
|
982
1067
|
if (format === "json") {
|
|
983
1068
|
return { data: JSON.stringify(rows, null, 2), binary: false };
|
|
984
1069
|
}
|
|
1070
|
+
if (format === "ndjson") {
|
|
1071
|
+
return { data: rows.map((r) => JSON.stringify(r)).join("\n") + (rows.length ? "\n" : ""), binary: false };
|
|
1072
|
+
}
|
|
985
1073
|
const cols = columnsOf(rows);
|
|
986
1074
|
const flat = rows.map((row) => {
|
|
987
1075
|
const out = {};
|
|
@@ -1054,6 +1142,8 @@ function parseArgs(list) {
|
|
|
1054
1142
|
else if (arg === "--area" || arg === "--areas") a.area = next();
|
|
1055
1143
|
else if (arg === "-t" || arg === "--type" || arg === "--types") a.type = next();
|
|
1056
1144
|
else if (arg === "-q" || arg === "--query") a.query = next();
|
|
1145
|
+
else if (arg === "--near") a.near = next();
|
|
1146
|
+
else if (arg === "--radius") a.radius = next();
|
|
1057
1147
|
else if (arg === "--fields") a.fields = next();
|
|
1058
1148
|
else if (arg === "--preset") a.preset = next();
|
|
1059
1149
|
else if (arg === "-f" || arg === "--format") a.format = next();
|
|
@@ -1109,7 +1199,7 @@ ${c("bold", c("magenta", "\u25C6 lacspace-leads"))} ${c("dim", "\u2014 free loca
|
|
|
1109
1199
|
|
|
1110
1200
|
${c("bold", "Usage")}
|
|
1111
1201
|
npx lacspace-leads [type] [options]
|
|
1112
|
-
npx lacspace-leads convert <file> [-f json|csv|xlsx] [-o out]
|
|
1202
|
+
npx lacspace-leads convert <file> [-f json|ndjson|csv|xlsx] [-o out]
|
|
1113
1203
|
|
|
1114
1204
|
${c("bold", "Search options")}
|
|
1115
1205
|
-t, --type <text> Business type/keyword. Comma-separate for several,
|
|
@@ -1118,6 +1208,8 @@ ${c("bold", "Search options")}
|
|
|
1118
1208
|
--area <text> Area/neighbourhood. Comma-separate to sweep a whole
|
|
1119
1209
|
city, e.g. --area "Baneshwor,Thamel,Patan"
|
|
1120
1210
|
-q, --query <text> Raw query verbatim (overrides city/area/type)
|
|
1211
|
+
--near <lat,lng> Centre the search on a coordinate (radius search)
|
|
1212
|
+
--radius <dist> Keep only leads within this of --near, e.g. 2km, 500m, 1mi
|
|
1121
1213
|
--fields <list> Columns: ${ALL_FIELDS.join(",")}
|
|
1122
1214
|
--preset <name> Field bundle: ${Object.keys(FIELD_PRESETS).join(" | ")}
|
|
1123
1215
|
-n, --limit <n> Max listings per search (default 60)
|
|
@@ -1146,8 +1238,8 @@ ${c("bold", "Filters & order")}
|
|
|
1146
1238
|
--has-valid-email Keep only leads with an MX-verified email (implies --verify-emails)
|
|
1147
1239
|
--dedupe <key> website | phone | name | smart | none (default website;
|
|
1148
1240
|
--append uses smart: website\u2192phone\u2192name)
|
|
1149
|
-
--sort <key> rating | reviews | name | priceLevel
|
|
1150
|
-
--desc / --asc Sort direction
|
|
1241
|
+
--sort <key> rating | reviews | name | priceLevel | distance
|
|
1242
|
+
--desc / --asc Sort direction (distance defaults nearest-first)
|
|
1151
1243
|
|
|
1152
1244
|
${c("bold", "Output")}
|
|
1153
1245
|
-f, --format <fmt> json | ndjson | csv | xlsx (default json)
|
|
@@ -1175,6 +1267,7 @@ ${c("bold", "Examples")}
|
|
|
1175
1267
|
npx lacspace-leads salons --city Pokhara --preset outreach --country NP -f csv -o -
|
|
1176
1268
|
npx lacspace-leads dentists --city Pokhara --verify-emails --has-valid-email -f csv
|
|
1177
1269
|
npx lacspace-leads cafes --city Kathmandu -o master.csv --append # accumulate daily
|
|
1270
|
+
npx lacspace-leads restaurants --near "27.7172,85.3240" --radius 2km -f csv
|
|
1178
1271
|
npx lacspace-leads convert leads.json -f xlsx
|
|
1179
1272
|
|
|
1180
1273
|
${c("dim", "Please scrape responsibly: keep volumes small, respect Google's Terms of")}
|
|
@@ -1197,6 +1290,13 @@ async function runConvert(rest) {
|
|
|
1197
1290
|
exit(1);
|
|
1198
1291
|
return;
|
|
1199
1292
|
}
|
|
1293
|
+
if ((rest.includes("-f") || rest.includes("--format")) && !OUTPUT_FORMATS.includes(a.format)) {
|
|
1294
|
+
log(c("red", `
|
|
1295
|
+
\u2717 Unknown format "${a.format}". Use: ${OUTPUT_FORMATS.join(", ")}.
|
|
1296
|
+
`));
|
|
1297
|
+
exit(1);
|
|
1298
|
+
return;
|
|
1299
|
+
}
|
|
1200
1300
|
log(`
|
|
1201
1301
|
${c("bold", c("magenta", "\u25C6 lacspace-leads convert"))}
|
|
1202
1302
|
`);
|
|
@@ -1261,6 +1361,26 @@ ${c("bold", c("magenta", "\u25C6 lacspace-leads"))} ${c("dim", "\u2014 Google Ma
|
|
|
1261
1361
|
exit(1);
|
|
1262
1362
|
return;
|
|
1263
1363
|
}
|
|
1364
|
+
const nearPoint = args.near ? parseLatLngPair(args.near) : void 0;
|
|
1365
|
+
if (args.near && !nearPoint) {
|
|
1366
|
+
log(c("red", `
|
|
1367
|
+
\u2717 --near must be "lat,lng", e.g. --near "27.7172,85.3240".`));
|
|
1368
|
+
exit(1);
|
|
1369
|
+
return;
|
|
1370
|
+
}
|
|
1371
|
+
const radiusM = args.radius ? parseDistance(args.radius) : void 0;
|
|
1372
|
+
if (args.radius && radiusM === void 0) {
|
|
1373
|
+
log(c("red", `
|
|
1374
|
+
\u2717 --radius must be a distance like 2km, 500m or 1mi.`));
|
|
1375
|
+
exit(1);
|
|
1376
|
+
return;
|
|
1377
|
+
}
|
|
1378
|
+
if (args.radius && !nearPoint) {
|
|
1379
|
+
log(c("red", `
|
|
1380
|
+
\u2717 --radius needs a centre \u2014 add --near "lat,lng".`));
|
|
1381
|
+
exit(1);
|
|
1382
|
+
return;
|
|
1383
|
+
}
|
|
1264
1384
|
const base = args.fields ? normalizeFields(args.fields) : resolvePreset(args.preset) ?? [...DEFAULT_FIELDS];
|
|
1265
1385
|
const wanted = new Set(base);
|
|
1266
1386
|
if (args.emails) wanted.add("email");
|
|
@@ -1269,6 +1389,7 @@ ${c("bold", c("magenta", "\u25C6 lacspace-leads"))} ${c("dim", "\u2014 Google Ma
|
|
|
1269
1389
|
wanted.add("email");
|
|
1270
1390
|
wanted.add("emailStatus");
|
|
1271
1391
|
}
|
|
1392
|
+
if (nearPoint) wanted.add("distanceKm");
|
|
1272
1393
|
const fields = ALL_FIELDS.filter((f) => wanted.has(f));
|
|
1273
1394
|
const toStdout = args.out === "-";
|
|
1274
1395
|
const out = toStdout ? "-" : resolve2(args.out ?? defaultFilename(query, args.format));
|
|
@@ -1301,7 +1422,9 @@ ${c("bold", c("magenta", "\u25C6 lacspace-leads"))} ${c("dim", "\u2014 Google Ma
|
|
|
1301
1422
|
if (args.emails || args.socials) log(` ${c("dim", "enrich")} ${[args.emails && "emails", args.socials && "socials"].filter(Boolean).join(" + ")} ${c("dim", `(${args.concurrency ?? 3}\xD7 parallel, visits each website)`)}`);
|
|
1302
1423
|
if (args.verifyEmails) log(` ${c("dim", "verify")} email domains (MX lookup)`);
|
|
1303
1424
|
if (hasFilters) log(` ${c("dim", "filters")} ${Object.entries(filters).map(([k, v]) => `${k}=${v}`).join(", ")}`);
|
|
1425
|
+
if (nearPoint) log(` ${c("dim", "near")} ${nearPoint.lat},${nearPoint.lng}${radiusM !== void 0 ? c("dim", ` (within ${args.radius})`) : c("dim", " (centred, no radius filter)")}`);
|
|
1304
1426
|
if (args.sort) log(` ${c("dim", "sort")} ${args.sort} ${args.desc === false ? "asc" : "desc"}`);
|
|
1427
|
+
else if (nearPoint) log(` ${c("dim", "sort")} distance (nearest first)`);
|
|
1305
1428
|
if (args.country) log(` ${c("dim", "phones")} E.164 for ${args.country}`);
|
|
1306
1429
|
if (args.proxy) log(` ${c("dim", "proxy")} ${args.proxy.replace(/\/\/[^@]+@/, "//***@")}`);
|
|
1307
1430
|
log(` ${c("dim", "limit")} ${args.limit}${isBatch ? "/search" : ""}${args.total ? ` (cap ${args.total})` : ""} ${c("dim", "format")} ${args.format} ${c("dim", "\u2192")} ${toStdout ? "stdout" : out}${args.append ? c("dim", " (append)") : ""}`);
|
|
@@ -1338,7 +1461,10 @@ ${c("yellow", "!")} This opens a browser and searches Google Maps. Continue? ${c
|
|
|
1338
1461
|
};
|
|
1339
1462
|
if (hasFilters) opts.filters = filters;
|
|
1340
1463
|
if (args.dedupe) opts.dedupe = args.dedupe;
|
|
1464
|
+
if (nearPoint) opts.near = { lat: nearPoint.lat, lng: nearPoint.lng };
|
|
1465
|
+
if (radiusM !== void 0) opts.radiusM = radiusM;
|
|
1341
1466
|
if (args.sort) opts.sort = args.sort;
|
|
1467
|
+
else if (nearPoint) opts.sort = "distance";
|
|
1342
1468
|
if (args.desc !== void 0) opts.sortDir = args.desc ? "desc" : "asc";
|
|
1343
1469
|
if (args.country) opts.country = args.country;
|
|
1344
1470
|
if (args.locale) opts.locale = args.locale;
|
package/dist/lib.cjs
CHANGED
|
@@ -42,10 +42,13 @@ __export(lib_exports, {
|
|
|
42
42
|
extractEmails: () => extractEmails,
|
|
43
43
|
extractSocials: () => extractSocials,
|
|
44
44
|
filterLeads: () => filterLeads,
|
|
45
|
+
haversineMeters: () => haversineMeters,
|
|
45
46
|
mapsSearchUrl: () => mapsSearchUrl,
|
|
46
47
|
normalizeFields: () => normalizeFields,
|
|
47
48
|
normalizePhone: () => normalizePhone,
|
|
49
|
+
parseDistance: () => parseDistance,
|
|
48
50
|
parseLatLng: () => parseLatLng,
|
|
51
|
+
parseLatLngPair: () => parseLatLngPair,
|
|
49
52
|
parseRating: () => parseRating,
|
|
50
53
|
parseReviewCount: () => parseReviewCount,
|
|
51
54
|
readRows: () => readRows,
|
|
@@ -61,7 +64,8 @@ __export(lib_exports, {
|
|
|
61
64
|
sortLeads: () => sortLeads,
|
|
62
65
|
toRows: () => toRows,
|
|
63
66
|
verifyEmail: () => verifyEmail,
|
|
64
|
-
verifyEmails: () => verifyEmails
|
|
67
|
+
verifyEmails: () => verifyEmails,
|
|
68
|
+
zoomForRadius: () => zoomForRadius
|
|
65
69
|
});
|
|
66
70
|
module.exports = __toCommonJS(lib_exports);
|
|
67
71
|
|
|
@@ -91,9 +95,11 @@ var ALL_FIELDS = [
|
|
|
91
95
|
"plusCode",
|
|
92
96
|
"latitude",
|
|
93
97
|
"longitude",
|
|
98
|
+
"distanceKm",
|
|
94
99
|
"hours",
|
|
95
100
|
"mapsUrl"
|
|
96
101
|
];
|
|
102
|
+
var DERIVED_FIELDS = ["distanceKm"];
|
|
97
103
|
var ENRICHED_FIELDS = [
|
|
98
104
|
"email",
|
|
99
105
|
"facebook",
|
|
@@ -116,11 +122,13 @@ var FIELD_PRESETS = {
|
|
|
116
122
|
/** Map/geo columns for plotting. */
|
|
117
123
|
geo: ["name", "address", "latitude", "longitude", "plusCode", "mapsUrl"],
|
|
118
124
|
/** Everything Maps shows, no website enrichment. */
|
|
119
|
-
full: ALL_FIELDS.filter((f) => !ENRICHED_FIELDS.includes(f)),
|
|
125
|
+
full: ALL_FIELDS.filter((f) => !ENRICHED_FIELDS.includes(f) && !DERIVED_FIELDS.includes(f)),
|
|
120
126
|
/** Every field, including enriched ones. */
|
|
121
127
|
everything: [...ALL_FIELDS]
|
|
122
128
|
};
|
|
123
|
-
var DEFAULT_FIELDS = ALL_FIELDS.filter(
|
|
129
|
+
var DEFAULT_FIELDS = ALL_FIELDS.filter(
|
|
130
|
+
(f) => !ENRICHED_FIELDS.includes(f) && !DERIVED_FIELDS.includes(f)
|
|
131
|
+
);
|
|
124
132
|
|
|
125
133
|
// src/query.ts
|
|
126
134
|
function composeQuery(opts) {
|
|
@@ -134,7 +142,9 @@ function mapsSearchUrl(query, opts = {}) {
|
|
|
134
142
|
const hl = (opts.hl ?? "en").split("-")[0] || "en";
|
|
135
143
|
const params = new URLSearchParams({ hl });
|
|
136
144
|
if (opts.gl) params.set("gl", opts.gl.toLowerCase());
|
|
137
|
-
|
|
145
|
+
const base = `https://www.google.com/maps/search/${encodeURIComponent(query)}`;
|
|
146
|
+
const at = opts.center ? `/@${opts.center.lat},${opts.center.lng},${Math.round(opts.center.zoom ?? 14)}z` : "";
|
|
147
|
+
return `${base}${at}?${params.toString()}`;
|
|
138
148
|
}
|
|
139
149
|
function expandQueries(opts) {
|
|
140
150
|
if (opts.query && opts.query.trim()) return [{ query: opts.query.trim() }];
|
|
@@ -434,12 +444,14 @@ function plausible(digits) {
|
|
|
434
444
|
return digits.length >= 8 && digits.length <= 15;
|
|
435
445
|
}
|
|
436
446
|
var priceRank = (p) => p ? p.replace(/[^$€£₹¥₩]/g, "").length : 0;
|
|
447
|
+
var ASC_BY_DEFAULT = /* @__PURE__ */ new Set(["name", "distance"]);
|
|
437
448
|
function sortLeads(leads, by, dir) {
|
|
438
449
|
if (!by) return [...leads];
|
|
439
|
-
const desc = dir ? dir === "desc" : by
|
|
450
|
+
const desc = dir ? dir === "desc" : !ASC_BY_DEFAULT.has(by);
|
|
440
451
|
const val = (l) => {
|
|
441
452
|
if (by === "name") return l.name?.toLowerCase();
|
|
442
453
|
if (by === "priceLevel") return l.priceLevel ? priceRank(l.priceLevel) : void 0;
|
|
454
|
+
if (by === "distance") return l.distanceKm;
|
|
443
455
|
return l[by];
|
|
444
456
|
};
|
|
445
457
|
return [...leads].map((lead, i) => ({ lead, i })).sort((a, b) => {
|
|
@@ -507,6 +519,43 @@ async function verifyEmails(leads, opts = {}) {
|
|
|
507
519
|
return leads;
|
|
508
520
|
}
|
|
509
521
|
|
|
522
|
+
// src/geo.ts
|
|
523
|
+
var R_EARTH_M = 63710088e-1;
|
|
524
|
+
var toRad = (deg) => deg * Math.PI / 180;
|
|
525
|
+
function haversineMeters(a, b) {
|
|
526
|
+
const dLat = toRad(b.lat - a.lat);
|
|
527
|
+
const dLng = toRad(b.lng - a.lng);
|
|
528
|
+
const lat1 = toRad(a.lat);
|
|
529
|
+
const lat2 = toRad(b.lat);
|
|
530
|
+
const h = Math.sin(dLat / 2) ** 2 + Math.cos(lat1) * Math.cos(lat2) * Math.sin(dLng / 2) ** 2;
|
|
531
|
+
return 2 * R_EARTH_M * Math.asin(Math.min(1, Math.sqrt(h)));
|
|
532
|
+
}
|
|
533
|
+
function parseLatLngPair(input) {
|
|
534
|
+
if (!input) return void 0;
|
|
535
|
+
const m = input.replace(/^@/, "").match(/(-?\d+(?:\.\d+)?)\s*[, ]\s*(-?\d+(?:\.\d+)?)/);
|
|
536
|
+
if (!m) return void 0;
|
|
537
|
+
const lat = parseFloat(m[1]);
|
|
538
|
+
const lng = parseFloat(m[2]);
|
|
539
|
+
if (!Number.isFinite(lat) || !Number.isFinite(lng)) return void 0;
|
|
540
|
+
if (lat < -90 || lat > 90 || lng < -180 || lng > 180) return void 0;
|
|
541
|
+
return { lat, lng };
|
|
542
|
+
}
|
|
543
|
+
function parseDistance(input) {
|
|
544
|
+
if (!input) return void 0;
|
|
545
|
+
const m = input.trim().toLowerCase().match(/^(\d+(?:\.\d+)?)\s*(km|m|mi|mile|miles|meters?|metres?)?$/);
|
|
546
|
+
if (!m) return void 0;
|
|
547
|
+
const value = parseFloat(m[1]);
|
|
548
|
+
if (!Number.isFinite(value) || value <= 0) return void 0;
|
|
549
|
+
const unit = m[2] ?? "m";
|
|
550
|
+
const metres = unit === "km" ? value * 1e3 : unit.startsWith("mi") ? value * 1609.344 : value;
|
|
551
|
+
return metres;
|
|
552
|
+
}
|
|
553
|
+
function zoomForRadius(radiusM, lat = 0) {
|
|
554
|
+
const mpp = 2 * Math.max(50, radiusM) / 600;
|
|
555
|
+
const zoom = Math.log2(156543.03392 * Math.cos(toRad(lat)) / mpp);
|
|
556
|
+
return Math.max(3, Math.min(19, Math.round(zoom)));
|
|
557
|
+
}
|
|
558
|
+
|
|
510
559
|
// src/export.ts
|
|
511
560
|
var import_csv = require("@lacspace/csv");
|
|
512
561
|
var import_xlsx = require("@lacspace/xlsx");
|
|
@@ -532,6 +581,7 @@ var HEADERS = {
|
|
|
532
581
|
plusCode: "Plus Code",
|
|
533
582
|
latitude: "Latitude",
|
|
534
583
|
longitude: "Longitude",
|
|
584
|
+
distanceKm: "Distance (km)",
|
|
535
585
|
hours: "Hours",
|
|
536
586
|
mapsUrl: "Maps URL"
|
|
537
587
|
};
|
|
@@ -592,7 +642,7 @@ var FIELD_BY_KEY = (() => {
|
|
|
592
642
|
}
|
|
593
643
|
return m;
|
|
594
644
|
})();
|
|
595
|
-
var NUMERIC_FIELDS = /* @__PURE__ */ new Set(["rating", "reviews", "latitude", "longitude"]);
|
|
645
|
+
var NUMERIC_FIELDS = /* @__PURE__ */ new Set(["rating", "reviews", "latitude", "longitude", "distanceKm"]);
|
|
596
646
|
function rowsToLeads(rows) {
|
|
597
647
|
return rows.map((row) => {
|
|
598
648
|
const lead = {};
|
|
@@ -803,8 +853,13 @@ async function scrapeLeads(opts) {
|
|
|
803
853
|
const pauseMs = () => opts.jitter ? Math.round(delayMs * (0.6 + Math.random() * 0.8)) : delayMs;
|
|
804
854
|
const wantVerify = Boolean(opts.verifyEmails) || Boolean(opts.filters?.hasValidEmail) || fields.has("emailStatus");
|
|
805
855
|
const wantEnrich = Boolean(opts.enrich) || ENRICHED_FIELDS.some((f) => fields.has(f)) || Boolean(opts.filters?.hasEmail) || wantVerify;
|
|
856
|
+
const near = opts.near;
|
|
806
857
|
const collect = new Set(fields);
|
|
807
858
|
if (wantEnrich) collect.add("website");
|
|
859
|
+
if (near) {
|
|
860
|
+
collect.add("latitude");
|
|
861
|
+
collect.add("longitude");
|
|
862
|
+
}
|
|
808
863
|
const detailOnly = ["name", "mapsUrl"];
|
|
809
864
|
const wantDetails = (opts.details ?? true) && ([...collect].some((f) => !detailOnly.includes(f)) || wantEnrich);
|
|
810
865
|
onProgress?.(`searching Google Maps for "${query}"\u2026`);
|
|
@@ -818,6 +873,7 @@ async function scrapeLeads(opts) {
|
|
|
818
873
|
const page = await context.newPage();
|
|
819
874
|
const urlOpts = { hl: locale };
|
|
820
875
|
if (opts.region) urlOpts.gl = opts.region;
|
|
876
|
+
if (near) urlOpts.center = { lat: near.lat, lng: near.lng, zoom: zoomForRadius(opts.radiusM ?? 2e3, near.lat) };
|
|
821
877
|
await page.goto(mapsSearchUrl(query, urlOpts), { waitUntil: "domcontentloaded", timeout: 45e3 });
|
|
822
878
|
await dismissConsent(page);
|
|
823
879
|
await loadResults(page, limit, delayMs, onProgress, signal);
|
|
@@ -895,6 +951,19 @@ async function scrapeLeads(opts) {
|
|
|
895
951
|
if (lead.website) lead.website = cleanWebsite(lead.website);
|
|
896
952
|
}
|
|
897
953
|
}
|
|
954
|
+
if (near) {
|
|
955
|
+
for (const lead of leads) {
|
|
956
|
+
if (typeof lead.latitude === "number" && typeof lead.longitude === "number") {
|
|
957
|
+
const m = haversineMeters(near, { lat: lead.latitude, lng: lead.longitude });
|
|
958
|
+
lead.distanceKm = Math.round(m / 1e3 * 100) / 100;
|
|
959
|
+
}
|
|
960
|
+
}
|
|
961
|
+
if (opts.radiusM !== void 0) {
|
|
962
|
+
const r = opts.radiusM;
|
|
963
|
+
leads = leads.filter((l) => l.distanceKm !== void 0 && l.distanceKm * 1e3 <= r);
|
|
964
|
+
onProgress?.(`within ${Math.round(r)} m: ${leads.length} listing${leads.length === 1 ? "" : "s"}.`);
|
|
965
|
+
}
|
|
966
|
+
}
|
|
898
967
|
leads = dedupeLeads(leads, opts.dedupe ?? "website");
|
|
899
968
|
if (wantEnrich) {
|
|
900
969
|
const socialFields = ENRICHED_FIELDS.filter((f) => f !== "email");
|
|
@@ -935,6 +1004,16 @@ async function scrapeLeads(opts) {
|
|
|
935
1004
|
if (opts.filters) leads = filterLeads(leads, opts.filters);
|
|
936
1005
|
if (opts.sort) leads = sortLeads(leads, opts.sort, opts.sortDir);
|
|
937
1006
|
if (wantVerify && !fields.has("emailStatus")) for (const l of leads) delete l.emailStatus;
|
|
1007
|
+
if (near) {
|
|
1008
|
+
const keepLat = fields.has("latitude");
|
|
1009
|
+
const keepLng = fields.has("longitude");
|
|
1010
|
+
const keepDist = fields.has("distanceKm");
|
|
1011
|
+
for (const l of leads) {
|
|
1012
|
+
if (!keepLat) delete l.latitude;
|
|
1013
|
+
if (!keepLng) delete l.longitude;
|
|
1014
|
+
if (!keepDist) delete l.distanceKm;
|
|
1015
|
+
}
|
|
1016
|
+
}
|
|
938
1017
|
onProgress?.(`collected ${leads.length} lead${leads.length === 1 ? "" : "s"}.`);
|
|
939
1018
|
return leads;
|
|
940
1019
|
} finally {
|
|
@@ -1019,9 +1098,10 @@ var import_xlsx2 = require("@lacspace/xlsx");
|
|
|
1019
1098
|
function detectFormat(file) {
|
|
1020
1099
|
const ext = (0, import_node_path.extname)(file).toLowerCase().replace(/^\./, "");
|
|
1021
1100
|
if (ext === "json") return "json";
|
|
1101
|
+
if (ext === "ndjson" || ext === "jsonl") return "ndjson";
|
|
1022
1102
|
if (ext === "csv" || ext === "tsv") return "csv";
|
|
1023
1103
|
if (ext === "xlsx" || ext === "xls") return "xlsx";
|
|
1024
|
-
throw new Error(`Cannot infer a format from ".${ext}" \u2014 pass an explicit --format (json|csv|xlsx).`);
|
|
1104
|
+
throw new Error(`Cannot infer a format from ".${ext}" \u2014 pass an explicit --format (json|ndjson|csv|xlsx).`);
|
|
1025
1105
|
}
|
|
1026
1106
|
async function readRows(file, format) {
|
|
1027
1107
|
const fmt = format ?? detectFormat(file);
|
|
@@ -1033,6 +1113,15 @@ async function readRows(file, format) {
|
|
|
1033
1113
|
if (fmt === "csv") {
|
|
1034
1114
|
return (0, import_csv2.parse)(text, { header: true });
|
|
1035
1115
|
}
|
|
1116
|
+
if (fmt === "ndjson") {
|
|
1117
|
+
return text.split(/\r?\n/).map((l) => l.trim()).filter(Boolean).map((l, i) => {
|
|
1118
|
+
try {
|
|
1119
|
+
return JSON.parse(l);
|
|
1120
|
+
} catch {
|
|
1121
|
+
throw new Error(`Invalid NDJSON on line ${i + 1}.`);
|
|
1122
|
+
}
|
|
1123
|
+
});
|
|
1124
|
+
}
|
|
1036
1125
|
const parsed = JSON.parse(text);
|
|
1037
1126
|
if (Array.isArray(parsed)) return parsed;
|
|
1038
1127
|
if (parsed && typeof parsed === "object") return [parsed];
|
|
@@ -1055,6 +1144,9 @@ function serializeRows(rows, format, opts = {}) {
|
|
|
1055
1144
|
if (format === "json") {
|
|
1056
1145
|
return { data: JSON.stringify(rows, null, 2), binary: false };
|
|
1057
1146
|
}
|
|
1147
|
+
if (format === "ndjson") {
|
|
1148
|
+
return { data: rows.map((r) => JSON.stringify(r)).join("\n") + (rows.length ? "\n" : ""), binary: false };
|
|
1149
|
+
}
|
|
1058
1150
|
const cols = columnsOf(rows);
|
|
1059
1151
|
const flat = rows.map((row) => {
|
|
1060
1152
|
const out = {};
|
|
@@ -1108,10 +1200,13 @@ async function convertFile(input, opts = {}) {
|
|
|
1108
1200
|
extractEmails,
|
|
1109
1201
|
extractSocials,
|
|
1110
1202
|
filterLeads,
|
|
1203
|
+
haversineMeters,
|
|
1111
1204
|
mapsSearchUrl,
|
|
1112
1205
|
normalizeFields,
|
|
1113
1206
|
normalizePhone,
|
|
1207
|
+
parseDistance,
|
|
1114
1208
|
parseLatLng,
|
|
1209
|
+
parseLatLngPair,
|
|
1115
1210
|
parseRating,
|
|
1116
1211
|
parseReviewCount,
|
|
1117
1212
|
readRows,
|
|
@@ -1127,5 +1222,6 @@ async function convertFile(input, opts = {}) {
|
|
|
1127
1222
|
sortLeads,
|
|
1128
1223
|
toRows,
|
|
1129
1224
|
verifyEmail,
|
|
1130
|
-
verifyEmails
|
|
1225
|
+
verifyEmails,
|
|
1226
|
+
zoomForRadius
|
|
1131
1227
|
});
|
package/dist/lib.d.cts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
/** The fields a {@link Lead} can carry — request any subset via `fields`. */
|
|
2
|
-
type LeadField = "name" | "category" | "rating" | "reviews" | "priceLevel" | "address" | "phone" | "website" | "email" | "facebook" | "instagram" | "whatsapp" | "linkedin" | "twitter" | "youtube" | "tiktok" | "telegram" | "emailStatus" | "plusCode" | "latitude" | "longitude" | "hours" | "mapsUrl";
|
|
2
|
+
type LeadField = "name" | "category" | "rating" | "reviews" | "priceLevel" | "address" | "phone" | "website" | "email" | "facebook" | "instagram" | "whatsapp" | "linkedin" | "twitter" | "youtube" | "tiktok" | "telegram" | "emailStatus" | "plusCode" | "latitude" | "longitude" | "distanceKm" | "hours" | "mapsUrl";
|
|
3
3
|
/** Deliverability verdict for a discovered email (see `verifyEmails`). */
|
|
4
4
|
type EmailStatus = "valid" | "no-mx" | "invalid-format" | "unknown";
|
|
5
5
|
/** Every field, in a sensible column order for exports. */
|
|
@@ -53,6 +53,8 @@ interface Lead {
|
|
|
53
53
|
latitude?: number;
|
|
54
54
|
/** Longitude, parsed from the listing's Maps URL. */
|
|
55
55
|
longitude?: number;
|
|
56
|
+
/** Distance in km from the `near` point (set only in radius search). */
|
|
57
|
+
distanceKm?: number;
|
|
56
58
|
/** Opening-hours summary, when shown. */
|
|
57
59
|
hours?: string;
|
|
58
60
|
/** Canonical Google Maps URL for the listing. */
|
|
@@ -76,7 +78,7 @@ interface LeadFilters {
|
|
|
76
78
|
/** Output formats the tool can write. */
|
|
77
79
|
type OutputFormat = "json" | "ndjson" | "csv" | "xlsx";
|
|
78
80
|
/** How to sort collected leads before export. */
|
|
79
|
-
type SortKey$1 = "rating" | "reviews" | "name" | "priceLevel";
|
|
81
|
+
type SortKey$1 = "rating" | "reviews" | "name" | "priceLevel" | "distance";
|
|
80
82
|
/** Options for a lead search. */
|
|
81
83
|
interface SearchOptions {
|
|
82
84
|
/** City, e.g. "Kathmandu". */
|
|
@@ -87,6 +89,16 @@ interface SearchOptions {
|
|
|
87
89
|
type: string;
|
|
88
90
|
/** A ready-made query, used verbatim instead of composing city/area/type. */
|
|
89
91
|
query?: string;
|
|
92
|
+
/**
|
|
93
|
+
* Centre the search on a coordinate. Results are biased to this point, and —
|
|
94
|
+
* when `radiusM` is set — filtered to within it. Each lead gets `distanceKm`.
|
|
95
|
+
*/
|
|
96
|
+
near?: {
|
|
97
|
+
lat: number;
|
|
98
|
+
lng: number;
|
|
99
|
+
};
|
|
100
|
+
/** Keep only leads within this many metres of `near`. Requires `near`. */
|
|
101
|
+
radiusM?: number;
|
|
90
102
|
/** Max number of leads to collect. Default 60. */
|
|
91
103
|
limit?: number;
|
|
92
104
|
/** Fields to collect. Default: all of {@link ALL_FIELDS}. */
|
|
@@ -290,11 +302,17 @@ declare function composeQuery(opts: {
|
|
|
290
302
|
}): string;
|
|
291
303
|
/**
|
|
292
304
|
* The Google Maps search URL for a query. `hl` (interface language) keeps the
|
|
293
|
-
* scraped aria-labels predictable; `gl` biases results to a region
|
|
305
|
+
* scraped aria-labels predictable; `gl` biases results to a region; `center`
|
|
306
|
+
* (`{lat,lng,zoom}`) points the map at a location for a radius search.
|
|
294
307
|
*/
|
|
295
308
|
declare function mapsSearchUrl(query: string, opts?: {
|
|
296
309
|
hl?: string;
|
|
297
310
|
gl?: string;
|
|
311
|
+
center?: {
|
|
312
|
+
lat: number;
|
|
313
|
+
lng: number;
|
|
314
|
+
zoom?: number;
|
|
315
|
+
};
|
|
298
316
|
}): string;
|
|
299
317
|
/**
|
|
300
318
|
* Expand a possibly-multi search request into individual `{type, city, area}`
|
|
@@ -359,10 +377,11 @@ declare function callingCode(country?: string): string | undefined;
|
|
|
359
377
|
*/
|
|
360
378
|
declare function normalizePhone(raw?: string, country?: string): string | undefined;
|
|
361
379
|
/** Keys a lead list can be sorted by. */
|
|
362
|
-
type SortKey = "rating" | "reviews" | "name" | "priceLevel";
|
|
380
|
+
type SortKey = "rating" | "reviews" | "name" | "priceLevel" | "distance";
|
|
363
381
|
/**
|
|
364
382
|
* Sort leads by a key, missing values always last. Stable, pure, new array.
|
|
365
|
-
* `dir` defaults to descending for numeric keys
|
|
383
|
+
* `dir` defaults to descending for numeric quality keys, ascending for `name`
|
|
384
|
+
* and `distance` (nearest first).
|
|
366
385
|
*/
|
|
367
386
|
declare function sortLeads(leads: Lead[], by?: SortKey, dir?: "asc" | "desc"): Lead[];
|
|
368
387
|
|
|
@@ -399,11 +418,39 @@ declare function enrichContacts(website: string, opts?: {
|
|
|
399
418
|
contactPages?: boolean;
|
|
400
419
|
}): Promise<Contacts>;
|
|
401
420
|
|
|
421
|
+
/**
|
|
422
|
+
* Geo helpers for radius search — parse a coordinate/radius, measure distance
|
|
423
|
+
* (haversine), and pick a Maps zoom that frames a radius. All pure, unit-tested.
|
|
424
|
+
*/
|
|
425
|
+
/** A latitude/longitude point. */
|
|
426
|
+
interface LatLng {
|
|
427
|
+
lat: number;
|
|
428
|
+
lng: number;
|
|
429
|
+
}
|
|
430
|
+
/** Great-circle distance between two points, in metres. Pure. */
|
|
431
|
+
declare function haversineMeters(a: LatLng, b: LatLng): number;
|
|
432
|
+
/**
|
|
433
|
+
* Parse a `"lat,lng"` pair (also accepts a bare `@lat,lng` or whitespace).
|
|
434
|
+
* Returns `undefined` if it isn't two in-range numbers. Pure.
|
|
435
|
+
*/
|
|
436
|
+
declare function parseLatLngPair(input: string | undefined): LatLng | undefined;
|
|
437
|
+
/**
|
|
438
|
+
* Parse a distance like `"2km"`, `"500m"`, `"1.5mi"` (or a bare number = metres)
|
|
439
|
+
* into metres. Returns `undefined` for junk or non-positive values. Pure.
|
|
440
|
+
*/
|
|
441
|
+
declare function parseDistance(input: string | undefined): number | undefined;
|
|
442
|
+
/**
|
|
443
|
+
* A Google Maps zoom level (integer) that roughly frames a circle of `radiusM`
|
|
444
|
+
* at a given latitude in the map viewport. Clamped to 3–19. Pure/best-effort —
|
|
445
|
+
* Maps re-fits results anyway; this just centres the search sensibly.
|
|
446
|
+
*/
|
|
447
|
+
declare function zoomForRadius(radiusM: number, lat?: number): number;
|
|
448
|
+
|
|
402
449
|
/** An arbitrary tabular row. */
|
|
403
450
|
type DataRow = Record<string, unknown>;
|
|
404
451
|
/** Infer the format from a filename extension. Throws on an unknown extension. */
|
|
405
452
|
declare function detectFormat(file: string): OutputFormat;
|
|
406
|
-
/** Read a JSON / CSV / Excel file into an array of rows. */
|
|
453
|
+
/** Read a JSON / NDJSON / CSV / Excel file into an array of rows. */
|
|
407
454
|
declare function readRows(file: string, format?: OutputFormat): Promise<DataRow[]>;
|
|
408
455
|
/** The union of keys across all rows, in first-seen order — the column set. */
|
|
409
456
|
declare function columnsOf(rows: DataRow[]): string[];
|
|
@@ -425,4 +472,4 @@ declare function convertFile(input: string, opts?: {
|
|
|
425
472
|
count: number;
|
|
426
473
|
}>;
|
|
427
474
|
|
|
428
|
-
export { ALL_FIELDS, type BatchQuery, CALLING_CODES, type Contacts, DEFAULT_FIELDS, type DataRow, type DetailedResult, ENRICHED_FIELDS, type EmailStatus, FIELD_PRESETS, type Lead, type LeadField, type LeadFilters, type LeadStats, LeadsError, type OutputFormat, type SearchOptions, type SortKey$1 as SortKey, callingCode, cleanWebsite, columnsOf, composeQuery, computeStats, convertFile, dedupeLeads, defaultFilename, detectFormat, emailDomain, emailFormatValid, enrichContacts, expandQueries, extractEmails, extractSocials, filterLeads, mapsSearchUrl, normalizeFields, normalizePhone, parseLatLng, parseRating, parseReviewCount, readRows, resolvePreset, rowsToLeads, scrapeLeads, scrapeLeads as searchLeads, searchLeadsBatch, searchLeadsDetailed, searchLeadsMulti, serialize, serializeRows, sortLeads, toRows, verifyEmail, verifyEmails };
|
|
475
|
+
export { ALL_FIELDS, type BatchQuery, CALLING_CODES, type Contacts, DEFAULT_FIELDS, type DataRow, type DetailedResult, ENRICHED_FIELDS, type EmailStatus, FIELD_PRESETS, type LatLng, type Lead, type LeadField, type LeadFilters, type LeadStats, LeadsError, type OutputFormat, type SearchOptions, type SortKey$1 as SortKey, callingCode, cleanWebsite, columnsOf, composeQuery, computeStats, convertFile, dedupeLeads, defaultFilename, detectFormat, emailDomain, emailFormatValid, enrichContacts, expandQueries, extractEmails, extractSocials, filterLeads, haversineMeters, mapsSearchUrl, normalizeFields, normalizePhone, parseDistance, parseLatLng, parseLatLngPair, parseRating, parseReviewCount, readRows, resolvePreset, rowsToLeads, scrapeLeads, scrapeLeads as searchLeads, searchLeadsBatch, searchLeadsDetailed, searchLeadsMulti, serialize, serializeRows, sortLeads, toRows, verifyEmail, verifyEmails, zoomForRadius };
|
package/dist/lib.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
/** The fields a {@link Lead} can carry — request any subset via `fields`. */
|
|
2
|
-
type LeadField = "name" | "category" | "rating" | "reviews" | "priceLevel" | "address" | "phone" | "website" | "email" | "facebook" | "instagram" | "whatsapp" | "linkedin" | "twitter" | "youtube" | "tiktok" | "telegram" | "emailStatus" | "plusCode" | "latitude" | "longitude" | "hours" | "mapsUrl";
|
|
2
|
+
type LeadField = "name" | "category" | "rating" | "reviews" | "priceLevel" | "address" | "phone" | "website" | "email" | "facebook" | "instagram" | "whatsapp" | "linkedin" | "twitter" | "youtube" | "tiktok" | "telegram" | "emailStatus" | "plusCode" | "latitude" | "longitude" | "distanceKm" | "hours" | "mapsUrl";
|
|
3
3
|
/** Deliverability verdict for a discovered email (see `verifyEmails`). */
|
|
4
4
|
type EmailStatus = "valid" | "no-mx" | "invalid-format" | "unknown";
|
|
5
5
|
/** Every field, in a sensible column order for exports. */
|
|
@@ -53,6 +53,8 @@ interface Lead {
|
|
|
53
53
|
latitude?: number;
|
|
54
54
|
/** Longitude, parsed from the listing's Maps URL. */
|
|
55
55
|
longitude?: number;
|
|
56
|
+
/** Distance in km from the `near` point (set only in radius search). */
|
|
57
|
+
distanceKm?: number;
|
|
56
58
|
/** Opening-hours summary, when shown. */
|
|
57
59
|
hours?: string;
|
|
58
60
|
/** Canonical Google Maps URL for the listing. */
|
|
@@ -76,7 +78,7 @@ interface LeadFilters {
|
|
|
76
78
|
/** Output formats the tool can write. */
|
|
77
79
|
type OutputFormat = "json" | "ndjson" | "csv" | "xlsx";
|
|
78
80
|
/** How to sort collected leads before export. */
|
|
79
|
-
type SortKey$1 = "rating" | "reviews" | "name" | "priceLevel";
|
|
81
|
+
type SortKey$1 = "rating" | "reviews" | "name" | "priceLevel" | "distance";
|
|
80
82
|
/** Options for a lead search. */
|
|
81
83
|
interface SearchOptions {
|
|
82
84
|
/** City, e.g. "Kathmandu". */
|
|
@@ -87,6 +89,16 @@ interface SearchOptions {
|
|
|
87
89
|
type: string;
|
|
88
90
|
/** A ready-made query, used verbatim instead of composing city/area/type. */
|
|
89
91
|
query?: string;
|
|
92
|
+
/**
|
|
93
|
+
* Centre the search on a coordinate. Results are biased to this point, and —
|
|
94
|
+
* when `radiusM` is set — filtered to within it. Each lead gets `distanceKm`.
|
|
95
|
+
*/
|
|
96
|
+
near?: {
|
|
97
|
+
lat: number;
|
|
98
|
+
lng: number;
|
|
99
|
+
};
|
|
100
|
+
/** Keep only leads within this many metres of `near`. Requires `near`. */
|
|
101
|
+
radiusM?: number;
|
|
90
102
|
/** Max number of leads to collect. Default 60. */
|
|
91
103
|
limit?: number;
|
|
92
104
|
/** Fields to collect. Default: all of {@link ALL_FIELDS}. */
|
|
@@ -290,11 +302,17 @@ declare function composeQuery(opts: {
|
|
|
290
302
|
}): string;
|
|
291
303
|
/**
|
|
292
304
|
* The Google Maps search URL for a query. `hl` (interface language) keeps the
|
|
293
|
-
* scraped aria-labels predictable; `gl` biases results to a region
|
|
305
|
+
* scraped aria-labels predictable; `gl` biases results to a region; `center`
|
|
306
|
+
* (`{lat,lng,zoom}`) points the map at a location for a radius search.
|
|
294
307
|
*/
|
|
295
308
|
declare function mapsSearchUrl(query: string, opts?: {
|
|
296
309
|
hl?: string;
|
|
297
310
|
gl?: string;
|
|
311
|
+
center?: {
|
|
312
|
+
lat: number;
|
|
313
|
+
lng: number;
|
|
314
|
+
zoom?: number;
|
|
315
|
+
};
|
|
298
316
|
}): string;
|
|
299
317
|
/**
|
|
300
318
|
* Expand a possibly-multi search request into individual `{type, city, area}`
|
|
@@ -359,10 +377,11 @@ declare function callingCode(country?: string): string | undefined;
|
|
|
359
377
|
*/
|
|
360
378
|
declare function normalizePhone(raw?: string, country?: string): string | undefined;
|
|
361
379
|
/** Keys a lead list can be sorted by. */
|
|
362
|
-
type SortKey = "rating" | "reviews" | "name" | "priceLevel";
|
|
380
|
+
type SortKey = "rating" | "reviews" | "name" | "priceLevel" | "distance";
|
|
363
381
|
/**
|
|
364
382
|
* Sort leads by a key, missing values always last. Stable, pure, new array.
|
|
365
|
-
* `dir` defaults to descending for numeric keys
|
|
383
|
+
* `dir` defaults to descending for numeric quality keys, ascending for `name`
|
|
384
|
+
* and `distance` (nearest first).
|
|
366
385
|
*/
|
|
367
386
|
declare function sortLeads(leads: Lead[], by?: SortKey, dir?: "asc" | "desc"): Lead[];
|
|
368
387
|
|
|
@@ -399,11 +418,39 @@ declare function enrichContacts(website: string, opts?: {
|
|
|
399
418
|
contactPages?: boolean;
|
|
400
419
|
}): Promise<Contacts>;
|
|
401
420
|
|
|
421
|
+
/**
|
|
422
|
+
* Geo helpers for radius search — parse a coordinate/radius, measure distance
|
|
423
|
+
* (haversine), and pick a Maps zoom that frames a radius. All pure, unit-tested.
|
|
424
|
+
*/
|
|
425
|
+
/** A latitude/longitude point. */
|
|
426
|
+
interface LatLng {
|
|
427
|
+
lat: number;
|
|
428
|
+
lng: number;
|
|
429
|
+
}
|
|
430
|
+
/** Great-circle distance between two points, in metres. Pure. */
|
|
431
|
+
declare function haversineMeters(a: LatLng, b: LatLng): number;
|
|
432
|
+
/**
|
|
433
|
+
* Parse a `"lat,lng"` pair (also accepts a bare `@lat,lng` or whitespace).
|
|
434
|
+
* Returns `undefined` if it isn't two in-range numbers. Pure.
|
|
435
|
+
*/
|
|
436
|
+
declare function parseLatLngPair(input: string | undefined): LatLng | undefined;
|
|
437
|
+
/**
|
|
438
|
+
* Parse a distance like `"2km"`, `"500m"`, `"1.5mi"` (or a bare number = metres)
|
|
439
|
+
* into metres. Returns `undefined` for junk or non-positive values. Pure.
|
|
440
|
+
*/
|
|
441
|
+
declare function parseDistance(input: string | undefined): number | undefined;
|
|
442
|
+
/**
|
|
443
|
+
* A Google Maps zoom level (integer) that roughly frames a circle of `radiusM`
|
|
444
|
+
* at a given latitude in the map viewport. Clamped to 3–19. Pure/best-effort —
|
|
445
|
+
* Maps re-fits results anyway; this just centres the search sensibly.
|
|
446
|
+
*/
|
|
447
|
+
declare function zoomForRadius(radiusM: number, lat?: number): number;
|
|
448
|
+
|
|
402
449
|
/** An arbitrary tabular row. */
|
|
403
450
|
type DataRow = Record<string, unknown>;
|
|
404
451
|
/** Infer the format from a filename extension. Throws on an unknown extension. */
|
|
405
452
|
declare function detectFormat(file: string): OutputFormat;
|
|
406
|
-
/** Read a JSON / CSV / Excel file into an array of rows. */
|
|
453
|
+
/** Read a JSON / NDJSON / CSV / Excel file into an array of rows. */
|
|
407
454
|
declare function readRows(file: string, format?: OutputFormat): Promise<DataRow[]>;
|
|
408
455
|
/** The union of keys across all rows, in first-seen order — the column set. */
|
|
409
456
|
declare function columnsOf(rows: DataRow[]): string[];
|
|
@@ -425,4 +472,4 @@ declare function convertFile(input: string, opts?: {
|
|
|
425
472
|
count: number;
|
|
426
473
|
}>;
|
|
427
474
|
|
|
428
|
-
export { ALL_FIELDS, type BatchQuery, CALLING_CODES, type Contacts, DEFAULT_FIELDS, type DataRow, type DetailedResult, ENRICHED_FIELDS, type EmailStatus, FIELD_PRESETS, type Lead, type LeadField, type LeadFilters, type LeadStats, LeadsError, type OutputFormat, type SearchOptions, type SortKey$1 as SortKey, callingCode, cleanWebsite, columnsOf, composeQuery, computeStats, convertFile, dedupeLeads, defaultFilename, detectFormat, emailDomain, emailFormatValid, enrichContacts, expandQueries, extractEmails, extractSocials, filterLeads, mapsSearchUrl, normalizeFields, normalizePhone, parseLatLng, parseRating, parseReviewCount, readRows, resolvePreset, rowsToLeads, scrapeLeads, scrapeLeads as searchLeads, searchLeadsBatch, searchLeadsDetailed, searchLeadsMulti, serialize, serializeRows, sortLeads, toRows, verifyEmail, verifyEmails };
|
|
475
|
+
export { ALL_FIELDS, type BatchQuery, CALLING_CODES, type Contacts, DEFAULT_FIELDS, type DataRow, type DetailedResult, ENRICHED_FIELDS, type EmailStatus, FIELD_PRESETS, type LatLng, type Lead, type LeadField, type LeadFilters, type LeadStats, LeadsError, type OutputFormat, type SearchOptions, type SortKey$1 as SortKey, callingCode, cleanWebsite, columnsOf, composeQuery, computeStats, convertFile, dedupeLeads, defaultFilename, detectFormat, emailDomain, emailFormatValid, enrichContacts, expandQueries, extractEmails, extractSocials, filterLeads, haversineMeters, mapsSearchUrl, normalizeFields, normalizePhone, parseDistance, parseLatLng, parseLatLngPair, parseRating, parseReviewCount, readRows, resolvePreset, rowsToLeads, scrapeLeads, scrapeLeads as searchLeads, searchLeadsBatch, searchLeadsDetailed, searchLeadsMulti, serialize, serializeRows, sortLeads, toRows, verifyEmail, verifyEmails, zoomForRadius };
|
package/dist/lib.js
CHANGED
|
@@ -24,9 +24,11 @@ var ALL_FIELDS = [
|
|
|
24
24
|
"plusCode",
|
|
25
25
|
"latitude",
|
|
26
26
|
"longitude",
|
|
27
|
+
"distanceKm",
|
|
27
28
|
"hours",
|
|
28
29
|
"mapsUrl"
|
|
29
30
|
];
|
|
31
|
+
var DERIVED_FIELDS = ["distanceKm"];
|
|
30
32
|
var ENRICHED_FIELDS = [
|
|
31
33
|
"email",
|
|
32
34
|
"facebook",
|
|
@@ -49,11 +51,13 @@ var FIELD_PRESETS = {
|
|
|
49
51
|
/** Map/geo columns for plotting. */
|
|
50
52
|
geo: ["name", "address", "latitude", "longitude", "plusCode", "mapsUrl"],
|
|
51
53
|
/** Everything Maps shows, no website enrichment. */
|
|
52
|
-
full: ALL_FIELDS.filter((f) => !ENRICHED_FIELDS.includes(f)),
|
|
54
|
+
full: ALL_FIELDS.filter((f) => !ENRICHED_FIELDS.includes(f) && !DERIVED_FIELDS.includes(f)),
|
|
53
55
|
/** Every field, including enriched ones. */
|
|
54
56
|
everything: [...ALL_FIELDS]
|
|
55
57
|
};
|
|
56
|
-
var DEFAULT_FIELDS = ALL_FIELDS.filter(
|
|
58
|
+
var DEFAULT_FIELDS = ALL_FIELDS.filter(
|
|
59
|
+
(f) => !ENRICHED_FIELDS.includes(f) && !DERIVED_FIELDS.includes(f)
|
|
60
|
+
);
|
|
57
61
|
|
|
58
62
|
// src/query.ts
|
|
59
63
|
function composeQuery(opts) {
|
|
@@ -67,7 +71,9 @@ function mapsSearchUrl(query, opts = {}) {
|
|
|
67
71
|
const hl = (opts.hl ?? "en").split("-")[0] || "en";
|
|
68
72
|
const params = new URLSearchParams({ hl });
|
|
69
73
|
if (opts.gl) params.set("gl", opts.gl.toLowerCase());
|
|
70
|
-
|
|
74
|
+
const base = `https://www.google.com/maps/search/${encodeURIComponent(query)}`;
|
|
75
|
+
const at = opts.center ? `/@${opts.center.lat},${opts.center.lng},${Math.round(opts.center.zoom ?? 14)}z` : "";
|
|
76
|
+
return `${base}${at}?${params.toString()}`;
|
|
71
77
|
}
|
|
72
78
|
function expandQueries(opts) {
|
|
73
79
|
if (opts.query && opts.query.trim()) return [{ query: opts.query.trim() }];
|
|
@@ -367,12 +373,14 @@ function plausible(digits) {
|
|
|
367
373
|
return digits.length >= 8 && digits.length <= 15;
|
|
368
374
|
}
|
|
369
375
|
var priceRank = (p) => p ? p.replace(/[^$€£₹¥₩]/g, "").length : 0;
|
|
376
|
+
var ASC_BY_DEFAULT = /* @__PURE__ */ new Set(["name", "distance"]);
|
|
370
377
|
function sortLeads(leads, by, dir) {
|
|
371
378
|
if (!by) return [...leads];
|
|
372
|
-
const desc = dir ? dir === "desc" : by
|
|
379
|
+
const desc = dir ? dir === "desc" : !ASC_BY_DEFAULT.has(by);
|
|
373
380
|
const val = (l) => {
|
|
374
381
|
if (by === "name") return l.name?.toLowerCase();
|
|
375
382
|
if (by === "priceLevel") return l.priceLevel ? priceRank(l.priceLevel) : void 0;
|
|
383
|
+
if (by === "distance") return l.distanceKm;
|
|
376
384
|
return l[by];
|
|
377
385
|
};
|
|
378
386
|
return [...leads].map((lead, i) => ({ lead, i })).sort((a, b) => {
|
|
@@ -440,6 +448,43 @@ async function verifyEmails(leads, opts = {}) {
|
|
|
440
448
|
return leads;
|
|
441
449
|
}
|
|
442
450
|
|
|
451
|
+
// src/geo.ts
|
|
452
|
+
var R_EARTH_M = 63710088e-1;
|
|
453
|
+
var toRad = (deg) => deg * Math.PI / 180;
|
|
454
|
+
function haversineMeters(a, b) {
|
|
455
|
+
const dLat = toRad(b.lat - a.lat);
|
|
456
|
+
const dLng = toRad(b.lng - a.lng);
|
|
457
|
+
const lat1 = toRad(a.lat);
|
|
458
|
+
const lat2 = toRad(b.lat);
|
|
459
|
+
const h = Math.sin(dLat / 2) ** 2 + Math.cos(lat1) * Math.cos(lat2) * Math.sin(dLng / 2) ** 2;
|
|
460
|
+
return 2 * R_EARTH_M * Math.asin(Math.min(1, Math.sqrt(h)));
|
|
461
|
+
}
|
|
462
|
+
function parseLatLngPair(input) {
|
|
463
|
+
if (!input) return void 0;
|
|
464
|
+
const m = input.replace(/^@/, "").match(/(-?\d+(?:\.\d+)?)\s*[, ]\s*(-?\d+(?:\.\d+)?)/);
|
|
465
|
+
if (!m) return void 0;
|
|
466
|
+
const lat = parseFloat(m[1]);
|
|
467
|
+
const lng = parseFloat(m[2]);
|
|
468
|
+
if (!Number.isFinite(lat) || !Number.isFinite(lng)) return void 0;
|
|
469
|
+
if (lat < -90 || lat > 90 || lng < -180 || lng > 180) return void 0;
|
|
470
|
+
return { lat, lng };
|
|
471
|
+
}
|
|
472
|
+
function parseDistance(input) {
|
|
473
|
+
if (!input) return void 0;
|
|
474
|
+
const m = input.trim().toLowerCase().match(/^(\d+(?:\.\d+)?)\s*(km|m|mi|mile|miles|meters?|metres?)?$/);
|
|
475
|
+
if (!m) return void 0;
|
|
476
|
+
const value = parseFloat(m[1]);
|
|
477
|
+
if (!Number.isFinite(value) || value <= 0) return void 0;
|
|
478
|
+
const unit = m[2] ?? "m";
|
|
479
|
+
const metres = unit === "km" ? value * 1e3 : unit.startsWith("mi") ? value * 1609.344 : value;
|
|
480
|
+
return metres;
|
|
481
|
+
}
|
|
482
|
+
function zoomForRadius(radiusM, lat = 0) {
|
|
483
|
+
const mpp = 2 * Math.max(50, radiusM) / 600;
|
|
484
|
+
const zoom = Math.log2(156543.03392 * Math.cos(toRad(lat)) / mpp);
|
|
485
|
+
return Math.max(3, Math.min(19, Math.round(zoom)));
|
|
486
|
+
}
|
|
487
|
+
|
|
443
488
|
// src/export.ts
|
|
444
489
|
import { stringify as csvStringify } from "@lacspace/csv";
|
|
445
490
|
import { jsonToXlsx } from "@lacspace/xlsx";
|
|
@@ -465,6 +510,7 @@ var HEADERS = {
|
|
|
465
510
|
plusCode: "Plus Code",
|
|
466
511
|
latitude: "Latitude",
|
|
467
512
|
longitude: "Longitude",
|
|
513
|
+
distanceKm: "Distance (km)",
|
|
468
514
|
hours: "Hours",
|
|
469
515
|
mapsUrl: "Maps URL"
|
|
470
516
|
};
|
|
@@ -525,7 +571,7 @@ var FIELD_BY_KEY = (() => {
|
|
|
525
571
|
}
|
|
526
572
|
return m;
|
|
527
573
|
})();
|
|
528
|
-
var NUMERIC_FIELDS = /* @__PURE__ */ new Set(["rating", "reviews", "latitude", "longitude"]);
|
|
574
|
+
var NUMERIC_FIELDS = /* @__PURE__ */ new Set(["rating", "reviews", "latitude", "longitude", "distanceKm"]);
|
|
529
575
|
function rowsToLeads(rows) {
|
|
530
576
|
return rows.map((row) => {
|
|
531
577
|
const lead = {};
|
|
@@ -736,8 +782,13 @@ async function scrapeLeads(opts) {
|
|
|
736
782
|
const pauseMs = () => opts.jitter ? Math.round(delayMs * (0.6 + Math.random() * 0.8)) : delayMs;
|
|
737
783
|
const wantVerify = Boolean(opts.verifyEmails) || Boolean(opts.filters?.hasValidEmail) || fields.has("emailStatus");
|
|
738
784
|
const wantEnrich = Boolean(opts.enrich) || ENRICHED_FIELDS.some((f) => fields.has(f)) || Boolean(opts.filters?.hasEmail) || wantVerify;
|
|
785
|
+
const near = opts.near;
|
|
739
786
|
const collect = new Set(fields);
|
|
740
787
|
if (wantEnrich) collect.add("website");
|
|
788
|
+
if (near) {
|
|
789
|
+
collect.add("latitude");
|
|
790
|
+
collect.add("longitude");
|
|
791
|
+
}
|
|
741
792
|
const detailOnly = ["name", "mapsUrl"];
|
|
742
793
|
const wantDetails = (opts.details ?? true) && ([...collect].some((f) => !detailOnly.includes(f)) || wantEnrich);
|
|
743
794
|
onProgress?.(`searching Google Maps for "${query}"\u2026`);
|
|
@@ -751,6 +802,7 @@ async function scrapeLeads(opts) {
|
|
|
751
802
|
const page = await context.newPage();
|
|
752
803
|
const urlOpts = { hl: locale };
|
|
753
804
|
if (opts.region) urlOpts.gl = opts.region;
|
|
805
|
+
if (near) urlOpts.center = { lat: near.lat, lng: near.lng, zoom: zoomForRadius(opts.radiusM ?? 2e3, near.lat) };
|
|
754
806
|
await page.goto(mapsSearchUrl(query, urlOpts), { waitUntil: "domcontentloaded", timeout: 45e3 });
|
|
755
807
|
await dismissConsent(page);
|
|
756
808
|
await loadResults(page, limit, delayMs, onProgress, signal);
|
|
@@ -828,6 +880,19 @@ async function scrapeLeads(opts) {
|
|
|
828
880
|
if (lead.website) lead.website = cleanWebsite(lead.website);
|
|
829
881
|
}
|
|
830
882
|
}
|
|
883
|
+
if (near) {
|
|
884
|
+
for (const lead of leads) {
|
|
885
|
+
if (typeof lead.latitude === "number" && typeof lead.longitude === "number") {
|
|
886
|
+
const m = haversineMeters(near, { lat: lead.latitude, lng: lead.longitude });
|
|
887
|
+
lead.distanceKm = Math.round(m / 1e3 * 100) / 100;
|
|
888
|
+
}
|
|
889
|
+
}
|
|
890
|
+
if (opts.radiusM !== void 0) {
|
|
891
|
+
const r = opts.radiusM;
|
|
892
|
+
leads = leads.filter((l) => l.distanceKm !== void 0 && l.distanceKm * 1e3 <= r);
|
|
893
|
+
onProgress?.(`within ${Math.round(r)} m: ${leads.length} listing${leads.length === 1 ? "" : "s"}.`);
|
|
894
|
+
}
|
|
895
|
+
}
|
|
831
896
|
leads = dedupeLeads(leads, opts.dedupe ?? "website");
|
|
832
897
|
if (wantEnrich) {
|
|
833
898
|
const socialFields = ENRICHED_FIELDS.filter((f) => f !== "email");
|
|
@@ -868,6 +933,16 @@ async function scrapeLeads(opts) {
|
|
|
868
933
|
if (opts.filters) leads = filterLeads(leads, opts.filters);
|
|
869
934
|
if (opts.sort) leads = sortLeads(leads, opts.sort, opts.sortDir);
|
|
870
935
|
if (wantVerify && !fields.has("emailStatus")) for (const l of leads) delete l.emailStatus;
|
|
936
|
+
if (near) {
|
|
937
|
+
const keepLat = fields.has("latitude");
|
|
938
|
+
const keepLng = fields.has("longitude");
|
|
939
|
+
const keepDist = fields.has("distanceKm");
|
|
940
|
+
for (const l of leads) {
|
|
941
|
+
if (!keepLat) delete l.latitude;
|
|
942
|
+
if (!keepLng) delete l.longitude;
|
|
943
|
+
if (!keepDist) delete l.distanceKm;
|
|
944
|
+
}
|
|
945
|
+
}
|
|
871
946
|
onProgress?.(`collected ${leads.length} lead${leads.length === 1 ? "" : "s"}.`);
|
|
872
947
|
return leads;
|
|
873
948
|
} finally {
|
|
@@ -952,9 +1027,10 @@ import { jsonToXlsx as jsonToXlsx2, xlsxToJson } from "@lacspace/xlsx";
|
|
|
952
1027
|
function detectFormat(file) {
|
|
953
1028
|
const ext = extname(file).toLowerCase().replace(/^\./, "");
|
|
954
1029
|
if (ext === "json") return "json";
|
|
1030
|
+
if (ext === "ndjson" || ext === "jsonl") return "ndjson";
|
|
955
1031
|
if (ext === "csv" || ext === "tsv") return "csv";
|
|
956
1032
|
if (ext === "xlsx" || ext === "xls") return "xlsx";
|
|
957
|
-
throw new Error(`Cannot infer a format from ".${ext}" \u2014 pass an explicit --format (json|csv|xlsx).`);
|
|
1033
|
+
throw new Error(`Cannot infer a format from ".${ext}" \u2014 pass an explicit --format (json|ndjson|csv|xlsx).`);
|
|
958
1034
|
}
|
|
959
1035
|
async function readRows(file, format) {
|
|
960
1036
|
const fmt = format ?? detectFormat(file);
|
|
@@ -966,6 +1042,15 @@ async function readRows(file, format) {
|
|
|
966
1042
|
if (fmt === "csv") {
|
|
967
1043
|
return csvParse(text, { header: true });
|
|
968
1044
|
}
|
|
1045
|
+
if (fmt === "ndjson") {
|
|
1046
|
+
return text.split(/\r?\n/).map((l) => l.trim()).filter(Boolean).map((l, i) => {
|
|
1047
|
+
try {
|
|
1048
|
+
return JSON.parse(l);
|
|
1049
|
+
} catch {
|
|
1050
|
+
throw new Error(`Invalid NDJSON on line ${i + 1}.`);
|
|
1051
|
+
}
|
|
1052
|
+
});
|
|
1053
|
+
}
|
|
969
1054
|
const parsed = JSON.parse(text);
|
|
970
1055
|
if (Array.isArray(parsed)) return parsed;
|
|
971
1056
|
if (parsed && typeof parsed === "object") return [parsed];
|
|
@@ -988,6 +1073,9 @@ function serializeRows(rows, format, opts = {}) {
|
|
|
988
1073
|
if (format === "json") {
|
|
989
1074
|
return { data: JSON.stringify(rows, null, 2), binary: false };
|
|
990
1075
|
}
|
|
1076
|
+
if (format === "ndjson") {
|
|
1077
|
+
return { data: rows.map((r) => JSON.stringify(r)).join("\n") + (rows.length ? "\n" : ""), binary: false };
|
|
1078
|
+
}
|
|
991
1079
|
const cols = columnsOf(rows);
|
|
992
1080
|
const flat = rows.map((row) => {
|
|
993
1081
|
const out = {};
|
|
@@ -1040,10 +1128,13 @@ export {
|
|
|
1040
1128
|
extractEmails,
|
|
1041
1129
|
extractSocials,
|
|
1042
1130
|
filterLeads,
|
|
1131
|
+
haversineMeters,
|
|
1043
1132
|
mapsSearchUrl,
|
|
1044
1133
|
normalizeFields,
|
|
1045
1134
|
normalizePhone,
|
|
1135
|
+
parseDistance,
|
|
1046
1136
|
parseLatLng,
|
|
1137
|
+
parseLatLngPair,
|
|
1047
1138
|
parseRating,
|
|
1048
1139
|
parseReviewCount,
|
|
1049
1140
|
readRows,
|
|
@@ -1059,5 +1150,6 @@ export {
|
|
|
1059
1150
|
sortLeads,
|
|
1060
1151
|
toRows,
|
|
1061
1152
|
verifyEmail,
|
|
1062
|
-
verifyEmails
|
|
1153
|
+
verifyEmails,
|
|
1154
|
+
zoomForRadius
|
|
1063
1155
|
};
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "lacspace-leads",
|
|
3
|
-
"version": "1.
|
|
4
|
-
"description": "Free, open-source local-business lead finder — name a city,
|
|
3
|
+
"version": "1.4.0",
|
|
4
|
+
"description": "Free, open-source local-business lead finder — name a city/area/business type, or search by coordinates + radius, and it drives a real browser over Google Maps to collect names, phones, websites, ratings, addresses, emails and social links, then exports to JSON, NDJSON, CSV or Excel. Sweep multiple areas at once, verify emails by MX, normalise phones to E.164, accumulate into a master file, and enrich in parallel through an optional proxy. No API keys.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"lacspace-leads": "dist/cli.js"
|
|
@@ -33,6 +33,8 @@
|
|
|
33
33
|
"google-maps",
|
|
34
34
|
"scraper",
|
|
35
35
|
"local-business",
|
|
36
|
+
"radius-search",
|
|
37
|
+
"geosearch",
|
|
36
38
|
"prospecting",
|
|
37
39
|
"b2b",
|
|
38
40
|
"email-finder",
|