lacspace-leads 1.7.2 → 1.9.0

package/README.md

@@ -10,6 +10,7 @@ That opens a browser, searches Maps for *"restaurants in Baneshwor, Kathmandu"*,
 
 ## Why it's different
 
+- **Live file, row by row** — every lead is written to your file the moment it's found: CSV/NDJSON rows are appended, JSON stays a valid array, Excel is rewritten each time. Open it mid-run and the rows are there; stop with Ctrl-C and nothing collected is lost. `--resume` carries on from the file.
 - **Free & keyless** — uses a real browser (via [Playwright](https://playwright.dev)), not a paid Places API.
 - **Ask for a number, get that number** — one Google search stops at ~120 results; `--target 500` keeps searching (every area you name, then tiles of the map) until it has 500 unique businesses.
 - **Sweep a whole city** — comma-separate areas and it runs each search, then **merges and de-duplicates** into one list: `--area "Thamel,Baneshwor,Patan"`.
@@ -151,6 +152,8 @@ npx lacspace-leads --config campaign.json
 | Far fewer leads than the target | The map ran out — that really is everything Google lists for that query | More areas or cities, related `--type`s, or a bigger `--step` |
 | About 114 back from `--limit 500` | Google's per-search ceiling | Use `--target 500` instead |
 | No leads at all | Usually a CAPTCHA or a page change | Drop `--headless` to watch it, raise `--delay`, try again shortly |
+| Empty "Reviews" column | Google currently hides review counts from signed-out visitors (the rating still shows) | Nothing to fix on our side; sort by `rating` instead |
+| Excel file "can't be written" warning | The file is open in Excel, which locks it | Rows are kept and written as soon as you close it |
 | Empty address / website columns | `--no-details` was on, so listings were never opened | Remove it (`--no-website` and `--split` turn details back on for you) |
 | Everything grouped under "other" with `--split` | Same cause — no addresses to match against | Same fix |
 
@@ -205,6 +208,8 @@ npx lacspace-leads [type] [options]
 | `--region <cc>` | Region bias for results, e.g. `np`, `us` |
 | `--headless` | Run the browser without a visible window |
 | `-y, --yes` | Skip prompts and the browser-open confirmation |
+| `--push <url>` | Also stream finished leads to an https endpoint (CRM, Lacspace Mail) — the file is still written. See [Push leads](#push-leads-to-your-crm--lacspace-mail) |
+| `--push-token <t>` | Bearer token for `--push` (prefer the `LACSPACE_LEADS_PUSH_TOKEN` env var) |
 
 **Enrichment** (visits each business website — slower, opt-in):
 
@@ -361,6 +366,87 @@ npx lacspace-leads --config campaign.json
 
 Every search is run, merged and de-duplicated; the shared options, filters and output settings apply across the whole campaign. Programmatically: `runConfig(config)`.
 
+## Live output: watch the file fill up
+
+Ask for 100 leads and the file gets row 1 as soon as the first business is read, row 2 a few seconds later, and so on. Each lead is finished before it is written: website cleaned, email and socials found (in the background, while the browser opens the next listing), email verified, phone normalised, de-duplicated and checked against every filter. Each row is a final row, and a `--target 100` with filters stops at 100 *matching* leads.
+
+```
+  ✚ 37/100 Himalayan Java Coffee — 01-4422519 · himalayanjava.com  (14.2/min, ~5 min left)
+```
+
+| Format | While it runs |
+| --- | --- |
+| `csv` | header first, then one appended line per lead |
+| `ndjson` | one appended line per lead (also streams live to stdout with `-o -`) |
+| `json` | always a valid JSON array: the closing `]` is moved after each lead |
+| `xlsx` | the workbook is rewritten safely (temp file + rename) after each lead |
+
+When the run ends the file is rewritten once more in its final order (`--sort`, `--append` merges). Press **Ctrl-C** and it stops cleanly, saving everything collected. Then run the same command with `--resume`: it keeps the rows in the file, skips those listings and fills up to the `--target`. `--no-live` turns this off and writes only at the end.
+
+## Push leads to your CRM / Lacspace Mail
+
+Add `--push <url>` and every finished lead is also POSTed to that endpoint while the run goes on — for example straight into the **Leads board of [Lacspace Mail](https://mail.lacspace.com)**:
+
+```bash
+export LACSPACE_LEADS_PUSH_TOKEN="…your token…"
+npx lacspace-leads restaurants --city Kathmandu --target 300 --emails --verify-emails --push https://api.lacspace.com/api/webmail/leads/import
+```
+
+**The file is always written too.** `--push` is in addition to the normal output, never instead of it, so if the endpoint is down, slow or rejects the token you still have every lead on disk (and `--resume` works as usual).
+
+**Token.** Prefer the `LACSPACE_LEADS_PUSH_TOKEN` environment variable over `--push-token`, because a token on the command line can be seen in your shell history and in the process list (`ps`). Precedence is `--push-token`, then the config file, then the env var. The token is only ever sent in the `Authorization: Bearer …` header: it is never printed (echoes show `***`) and never written to the output or checkpoint files.
+
+**URL.** Only `https://` is accepted, except `http://localhost`, `http://127.0.0.1` and `http://[::1]` for local development. Anything else stops with a clear error before the browser opens.
+
+**In a campaign file** (`--config`), add a `push` block (CLI flags still win):
+
+```jsonc
+{
+  "searches": [{ "type": "cafes", "city": "Kathmandu" }],
+  "push": { "url": "https://api.lacspace.com/api/webmail/leads/import" }  // token: env var, or "token": "…"
+}
+```
+
+**How it sends.** Finished leads (after enrichment, verification and filters, the same rows the file gets) are batched and flushed **every 10 leads or every 3 seconds**, whichever comes first, in the background, with one request in flight at a time, so pushing never slows the scrape. If the endpoint is slow, waiting leads merge into the next request; at most 2,000 wait in memory, and any beyond that are skipped for the push (with one warning) but still saved to the file. Each request is a `POST` with `Content-Type: application/json`, `Authorization: Bearer <token>` (when a token is set), `User-Agent: lacspace-leads/<version>` and a 15-second timeout:
+
+```jsonc
+{
+  "leads": [ { "name": "Himalayan Java", "phone": "+97714422519", "website": "https://himalayanjava.com", … } ],
+  "search": { "type": "restaurants", "city": "Kathmandu", "area": null, "target": 300 },
+  "run": "6f1c…-…",   // one random UUID per run, the same on every request
+  "seq": 1,           // batch number, from 1
+  "done": false
+}
+```
+
+When the run ends (including a Ctrl-C), the rest is flushed and one final request is sent, bounded to a few seconds so Ctrl-C still exits quickly:
+
+```jsonc
+{ "leads": [], "search": { … }, "run": "6f1c…", "seq": 31, "done": true,
+  "stats": { "total": 300, "withPhone": 281, "withWebsite": 190, "withEmail": 122, "withValidEmail": 117, "withSocial": 0, "avgRating": 4.3 } }
+```
+
+**Failures never cost you the run:**
+
+| Endpoint answers | What happens |
+| --- | --- |
+| network error / timeout / `5xx` | retried 3 times, after 1s, 3s and 9s; then the batch is counted as failed and pushing carries on |
+| `429` | waits for `Retry-After` (seconds or an HTTP date, capped at 60s), still at most 3 retries |
+| `401` / `403` | one error, `push rejected: token invalid or expired, leads are still being saved to <file>`; pushing stops for the rest of the run, scraping and the file carry on |
+| any other `4xx` | one warning per status code, that batch is dropped, pushing carries on |
+
+The run ends with one line such as `push: sent 300 leads in 30 batches, failed 0, not rejected`.
+
+Programmatically, `createPusher({ url, token })` gives you `push(lead)`, `finish(stats)` and `stats()`. Wire it to `onResult`:
+
+```ts
+import { searchLeads, createPusher, computeStats, formatPushSummary } from "lacspace-leads";
+
+const pusher = createPusher({ url: "https://crm.example.com/leads", token: process.env.CRM_TOKEN, search: { type: "cafes", city: "Kathmandu" } });
+const leads = await searchLeads({ type: "cafes", city: "Kathmandu", onResult: (l) => pusher.push(l) });
+console.log(formatPushSummary(await pusher.finish(computeStats(leads))));
+```
+
 ## Resume a long sweep (`--resume`)
 
 Sweeping a whole city one neighbourhood at a time can take a while, and Google may throw a CAPTCHA halfway through. Add `--resume` and every completed sub-search is checkpointed to `<out>.checkpoint.json` — the leads gathered so far plus which searches are done. Re-run the exact same command and it **skips the areas already collected** and picks up where it stopped; when the sweep finishes cleanly the checkpoint is deleted.
@@ -372,7 +458,7 @@ npx lacspace-leads cafes --city Kathmandu \
 # crashes after Patan? just run it again — Thamel/Baneshwor/Patan are skipped.
 ```
 
-`--resume` applies to multi-search sweeps and `--config` campaigns (it's a no-op for a single search). The checkpoint key normalises case/whitespace, so the same area sweep always lines up.
+For multi-search sweeps and `--config` campaigns, `--resume` uses the checkpoint. For a single search or a `--target` run (since 1.8.0), it continues from the rows already in the output file. The checkpoint key normalises case/whitespace, so the same area sweep always lines up.
 
 ## Run summary (`--summary`)
 
@@ -565,7 +651,9 @@ const leads = await searchLeadsBatch(
 | `enrichContacts(website)` / `extractEmails` / `extractSocials` | Website enrichment, on tap. |
 | `cleanWebsite` / `normalizePhone` / `sortLeads` | Pure data-cleaning helpers (unit-tested). |
 | `haversineMeters` / `parseLatLngPair` / `parseDistance` | Pure geo helpers for radius search. |
-| `filterLeads` / `dedupeLeads` / `subtractLeads` | Pure post-processing — filter, in-list dedupe, and cross-file dedupe. |
+| `filterLeads` / `dedupeLeads` / `subtractLeads` / `dedupeKey` | Pure post-processing — filter, in-list dedupe, and cross-file dedupe. |
+| `createLiveWriter(options)` | Write leads to csv / ndjson / json / xlsx (or stdout) one by one as they arrive. |
+| `createPusher(options)` / `validatePushUrl` / `resolvePush` / `parseRetryAfter` / `formatPushSummary` | Stream finished leads to an HTTP endpoint in batches, with retries (1.9.0). |
 | `summarize(leads)` / `formatSummary(s)` | Rating bands, contact coverage %, top categories — the `--summary` engine. |
 | `parsePriceLevel` / `priceLevelValue` / `parseBusinessStatus` / `parseClaimed` / `parseOpenNow` / `parseCategoryTags` | Pure field parsers (unit-tested against HTML/aria snippets). |
 | `pipeToEnrich(handler)` / `leadsToEnrichInput` / `leadDomains` / `leadDomain` | Bridge collected leads to `lacspace-enrich` (no hard dep). |
@@ -573,7 +661,21 @@ const leads = await searchLeadsBatch(
 | `expandQueries` / `resolvePreset` / `FIELD_PRESETS` | Batch expansion + field presets. |
 | `composeQuery` / `mapsSearchUrl` / `normalizeFields` / `defaultFilename` | Query + helper utilities. |
 
-The `onProgress` and `onLead` callbacks stream progress and each lead as it's found — handy for live UIs or crash-safe writing. Everything is fully typed (`Lead`, `LeadField`, `SearchOptions`, `LeadStats`, `EmailStatus`, `SortKey`, `OutputFormat`, `Contacts` …) and ships dual **ESM + CJS**.
+Write leads as they're found (1.8.0):
+
+```ts
+import { searchLeads, createLiveWriter, DEFAULT_FIELDS } from "lacspace-leads";
+
+const writer = createLiveWriter({ file: "cafes.csv", format: "csv", fields: [...DEFAULT_FIELDS] });
+const leads = await searchLeads({
+  type: "cafes", city: "Kathmandu", limit: 100, headless: true,
+  onResult: (lead) => writer.add(lead),          // a finished row, the moment it's ready
+  remaining: () => 100 - writer.added,           // stop opening listings once there are 100
+});
+writer.finish(leads);
+```
+
+`onResult` gets each lead once it is final (enriched, verified, normalised, de-duplicated, filtered). `onLead` still gets the raw listing straight off the page, and `onProgress` streams status messages. `searchLeadsBatch` and target sweeps forward `onResult` across every search and tile, each business once. Everything is fully typed (`Lead`, `LeadField`, `SearchOptions`, `LeadStats`, `EmailStatus`, `SortKey`, `OutputFormat`, `Contacts` …) and ships dual **ESM + CJS**.
 
 ## Please use it responsibly