whitepages-scraper 0.1.0__tar.gz

whitepages_scraper-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Buğrahan Saka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

whitepages_scraper-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md LICENSE
+recursive-include tests *.py *.html
+recursive-include examples *.py

whitepages_scraper-0.1.0/PKG-INFO

@@ -0,0 +1,115 @@
+Metadata-Version: 2.4
+Name: whitepages-scraper
+Version: 0.1.0
+Summary: Scrape Whitepages.com name searches and person pages (name, age range, address, phones, relatives, associates) with Python.
+Author: Buğrahan Saka
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/bugrahansss/whitepages-scraper
+Project-URL: Source, https://github.com/bugrahansss/whitepages-scraper
+Project-URL: Issues, https://github.com/bugrahansss/whitepages-scraper/issues
+Project-URL: Guide, https://scrape.do/blog/white-pages-scraping/
+Keywords: whitepages,web scraping,people search,scraper,json-ld,cloudflare,scrapedo
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.25
+Requires-Dist: beautifulsoup4>=4.9
+Dynamic: license-file
+
+# Whitepages Scraper
+
+[![PyPI](https://img.shields.io/pypi/v/whitepages-scraper)](https://pypi.org/project/whitepages-scraper/)
+[![Python](https://img.shields.io/pypi/pyversions/whitepages-scraper)](https://pypi.org/project/whitepages-scraper/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+Scrape [Whitepages.com](https://www.whitepages.com) people search into structured data. Search a name (optionally in a state or city) and get every person on the result pages with their **current address, phone numbers, age range, neighborhood, previous locations and relatives**, then open person pages for **associates** and the rest. JSON or CSV.
+
+```bash
+pip install whitepages-scraper
+export SCRAPEDO_TOKEN=your_token
+whitepages-scraper --name "John Smith" --city Seattle --state WA --pages 2 -f csv -o people.csv
+```
+
+## What is in the way
+
+```python
+import requests
+
+requests.get("https://www.whitepages.com/name/John-Smith/Seattle-WA")
+# outside the US: the connection is dropped; from a datacenter IP: Cloudflare challenge
+```
+
+1. **US only.** Visitors outside the United States, datacenter IPs and most VPNs are dropped without an error page.
+2. **Cloudflare.** Every page sits behind Cloudflare's bot checks, so a plain HTTP client gets a challenge instead of HTML.
+
+This package fetches every page through [Scrape.do](https://scrape.do/?utm_source=github&utm_medium=readme&utm_campaign=whitepages-scraper) with `super=true&geoCode=us` (a US residential IP): about 3 seconds and 10 credits per page.
+
+## Where the data is
+
+Whitepages renders on the server and publishes the people on each page as schema.org JSON-LD:
+
+* **Search pages** carry an `ItemList` with up to **25 `Person` items**: full name, the current street address, phone numbers, `Age Range`, `Neighborhood` and `Previous Location` properties, relatives (`relatedTo`) and the person page URL. The visible cards mask phones ("(206) \*\*\* \*\*\*\*"); the JSON-LD has the full number. In a live test, one Seattle search page had 25 people, 25 addresses and 15 phone numbers in one request.
+* **Person pages** carry one `Person` with phones under `contactPoint`, relatives and associates (`knows`). The address is often only in the HTML (`.address-line1`, `.address-line2`) and the age range in the text ("is in their 60s"); the parser reads both.
+* **Pagination** is `?page=2`, `?page=3`, 25 people each.
+
+Ages are ranges ("40s", "80+"), never exact. Free pages show the current phone at most; phone history, emails and background reports are behind Whitepages' paid tier and are not scraped.
+
+## Python
+
+```python
+from whitepages_scraper import WhitepagesScraper
+
+wp = WhitepagesScraper(token="YOUR_SCRAPEDO_TOKEN")  # or set SCRAPEDO_TOKEN
+
+people = wp.search("John Smith", city="Seattle", state="WA", pages=2)
+for p in people[:3]:
+    print(p.name, p.age_range, p.address, p.city, p.phone, p.relatives)
+
+person = wp.get_profile(people[0].url)
+print(person.associates, person.phones)
+print(wp.credits_used)
+```
+
+## Command line
+
+```bash
+# Name search: 25 people per page, no extra requests
+whitepages-scraper --name "Mary Johnson" --state TX --pages 2 -f csv -o mary.csv
+
+# Open every person page found (one more request each)
+whitepages-scraper --name "John Smith" --city Seattle --state WA --profiles --limit 10
+
+# Person page URLs
+whitepages-scraper "https://www.whitepages.com/name/<Name>/<City-ST>/<id>"
+```
+
+## Fields
+
+`name`, `first_name`, `middle_name`, `last_name`, `age_range`, `address`, `city`, `state`, `zip_code`, `neighborhood`, `phones`, `aliases`, `relatives`, `associates`, `previous_locations`, `url`.
+
+## Use it responsibly
+
+This is personal data about real people. Whitepages is not a consumer reporting agency: don't use this data for decisions about employment, housing, credit or insurance (the FCRA regulates those in the US). Honor opt-out requests, follow the privacy laws that apply to you (CCPA, GDPR) and Whitepages' terms, and keep request rates modest.
+
+## Also available
+
+- **No-code:** [Whitepages Scraper on Apify](https://apify.com/scrapedo/whitepages-scraper)
+- **Browser demo:** [Hugging Face Space](https://huggingface.co/spaces/bugrahans/whitepages-scraper)
+- **Full guide:** [Whitepages.com Scraping: Bypass Cloudflare and Extract People Data](https://scrape.do/blog/white-pages-scraping/?utm_source=github&utm_medium=readme&utm_campaign=whitepages-scraper)
+
+## Tests
+
+```bash
+python -m unittest discover -s tests
+```
+
+The fixtures are fictional people (Jane Doe, 555 numbers); the tests never hit the network.
+
+## License
+
+MIT

whitepages_scraper-0.1.0/README.md

@@ -0,0 +1,92 @@
+# Whitepages Scraper
+
+[![PyPI](https://img.shields.io/pypi/v/whitepages-scraper)](https://pypi.org/project/whitepages-scraper/)
+[![Python](https://img.shields.io/pypi/pyversions/whitepages-scraper)](https://pypi.org/project/whitepages-scraper/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+Scrape [Whitepages.com](https://www.whitepages.com) people search into structured data. Search a name (optionally in a state or city) and get every person on the result pages with their **current address, phone numbers, age range, neighborhood, previous locations and relatives**, then open person pages for **associates** and the rest. JSON or CSV.
+
+```bash
+pip install whitepages-scraper
+export SCRAPEDO_TOKEN=your_token
+whitepages-scraper --name "John Smith" --city Seattle --state WA --pages 2 -f csv -o people.csv
+```
+
+## What is in the way
+
+```python
+import requests
+
+requests.get("https://www.whitepages.com/name/John-Smith/Seattle-WA")
+# outside the US: the connection is dropped; from a datacenter IP: Cloudflare challenge
+```
+
+1. **US only.** Visitors outside the United States, datacenter IPs and most VPNs are dropped without an error page.
+2. **Cloudflare.** Every page sits behind Cloudflare's bot checks, so a plain HTTP client gets a challenge instead of HTML.
+
+This package fetches every page through [Scrape.do](https://scrape.do/?utm_source=github&utm_medium=readme&utm_campaign=whitepages-scraper) with `super=true&geoCode=us` (a US residential IP): about 3 seconds and 10 credits per page.
+
+## Where the data is
+
+Whitepages renders on the server and publishes the people on each page as schema.org JSON-LD:
+
+* **Search pages** carry an `ItemList` with up to **25 `Person` items**: full name, the current street address, phone numbers, `Age Range`, `Neighborhood` and `Previous Location` properties, relatives (`relatedTo`) and the person page URL. The visible cards mask phones ("(206) \*\*\* \*\*\*\*"); the JSON-LD has the full number. In a live test, one Seattle search page had 25 people, 25 addresses and 15 phone numbers in one request.
+* **Person pages** carry one `Person` with phones under `contactPoint`, relatives and associates (`knows`). The address is often only in the HTML (`.address-line1`, `.address-line2`) and the age range in the text ("is in their 60s"); the parser reads both.
+* **Pagination** is `?page=2`, `?page=3`, 25 people each.
+
+Ages are ranges ("40s", "80+"), never exact. Free pages show the current phone at most; phone history, emails and background reports are behind Whitepages' paid tier and are not scraped.
+
+## Python
+
+```python
+from whitepages_scraper import WhitepagesScraper
+
+wp = WhitepagesScraper(token="YOUR_SCRAPEDO_TOKEN")  # or set SCRAPEDO_TOKEN
+
+people = wp.search("John Smith", city="Seattle", state="WA", pages=2)
+for p in people[:3]:
+    print(p.name, p.age_range, p.address, p.city, p.phone, p.relatives)
+
+person = wp.get_profile(people[0].url)
+print(person.associates, person.phones)
+print(wp.credits_used)
+```
+
+## Command line
+
+```bash
+# Name search: 25 people per page, no extra requests
+whitepages-scraper --name "Mary Johnson" --state TX --pages 2 -f csv -o mary.csv
+
+# Open every person page found (one more request each)
+whitepages-scraper --name "John Smith" --city Seattle --state WA --profiles --limit 10
+
+# Person page URLs
+whitepages-scraper "https://www.whitepages.com/name/<Name>/<City-ST>/<id>"
+```
+
+## Fields
+
+`name`, `first_name`, `middle_name`, `last_name`, `age_range`, `address`, `city`, `state`, `zip_code`, `neighborhood`, `phones`, `aliases`, `relatives`, `associates`, `previous_locations`, `url`.
+
+## Use it responsibly
+
+This is personal data about real people. Whitepages is not a consumer reporting agency: don't use this data for decisions about employment, housing, credit or insurance (the FCRA regulates those in the US). Honor opt-out requests, follow the privacy laws that apply to you (CCPA, GDPR) and Whitepages' terms, and keep request rates modest.
+
+## Also available
+
+- **No-code:** [Whitepages Scraper on Apify](https://apify.com/scrapedo/whitepages-scraper)
+- **Browser demo:** [Hugging Face Space](https://huggingface.co/spaces/bugrahans/whitepages-scraper)
+- **Full guide:** [Whitepages.com Scraping: Bypass Cloudflare and Extract People Data](https://scrape.do/blog/white-pages-scraping/?utm_source=github&utm_medium=readme&utm_campaign=whitepages-scraper)
+
+## Tests
+
+```bash
+python -m unittest discover -s tests
+```
+
+The fixtures are fictional people (Jane Doe, 555 numbers); the tests never hit the network.
+
+## License
+
+MIT

whitepages_scraper-0.1.0/examples/search_to_csv.py

@@ -0,0 +1,25 @@
+"""Search one name in one city, then open every person page and save a CSV.
+
+    export SCRAPEDO_TOKEN=...
+    python examples/search_to_csv.py
+"""
+
+import csv
+
+from whitepages_scraper import CSV_FIELDS, WhitepagesScraper
+
+wp = WhitepagesScraper()  # reads SCRAPEDO_TOKEN
+
+people = wp.search("John Smith", city="Seattle", state="WA", pages=2)
+print(f"{len(people)} people on the search pages")
+
+# The search pages already carry address, phone and age range.
+# Open the person pages only when you need associates or the HTML-only address.
+detailed = [wp.get_profile(p.url) for p in people[:5] if p.url]
+
+with open("people.csv", "w", newline="", encoding="utf-8") as handle:
+    writer = csv.DictWriter(handle, fieldnames=CSV_FIELDS)
+    writer.writeheader()
+    writer.writerows(p.to_row() for p in people)
+
+print(f"saved people.csv, {wp.credits_used} credits used")

whitepages_scraper-0.1.0/pyproject.toml

@@ -0,0 +1,34 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "whitepages-scraper"
+version = "0.1.0"
+description = "Scrape Whitepages.com name searches and person pages (name, age range, address, phones, relatives, associates) with Python."
+readme = "README.md"
+requires-python = ">=3.8"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Buğrahan Saka" }]
+keywords = ["whitepages", "web scraping", "people search", "scraper", "json-ld", "cloudflare", "scrapedo"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+]
+dependencies = ["requests>=2.25", "beautifulsoup4>=4.9"]
+
+[project.urls]
+Homepage = "https://github.com/bugrahansss/whitepages-scraper"
+Source = "https://github.com/bugrahansss/whitepages-scraper"
+Issues = "https://github.com/bugrahansss/whitepages-scraper/issues"
+Guide = "https://scrape.do/blog/white-pages-scraping/"
+
+[project.scripts]
+whitepages-scraper = "whitepages_scraper.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

whitepages_scraper-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

whitepages_scraper-0.1.0/src/whitepages_scraper/__init__.py

@@ -0,0 +1,18 @@
+"""Scrape Whitepages.com name searches and person pages with Python."""
+
+from .client import ScrapeError, WhitepagesScraper
+from .parser import CSV_FIELDS, Person, format_phone, is_profile_url, name_search_url, parse_profile, parse_search_results
+
+__all__ = [
+    "CSV_FIELDS",
+    "Person",
+    "ScrapeError",
+    "WhitepagesScraper",
+    "format_phone",
+    "is_profile_url",
+    "name_search_url",
+    "parse_profile",
+    "parse_search_results",
+]
+
+__version__ = "0.1.0"

whitepages_scraper-0.1.0/src/whitepages_scraper/cli.py

@@ -0,0 +1,100 @@
+"""Command line entry point.
+
+    whitepages-scraper --name "John Smith" --city Seattle --state WA --pages 2 -f csv -o people.csv
+    whitepages-scraper https://www.whitepages.com/name/<Name>/<City-ST>/<id>
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import sys
+from typing import List, Optional
+
+from . import __version__
+from .client import TOKEN_ENV, ScrapeError, WhitepagesScraper
+from .parser import CSV_FIELDS, Person
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(prog="whitepages-scraper",
+                                description="Scrape Whitepages.com name searches and person pages into JSON or CSV.")
+    p.add_argument("urls", nargs="*", help="Person page URLs to scrape")
+    p.add_argument("-i", "--input", help="Text file with one person page URL per line")
+    p.add_argument("-n", "--name", action="append", default=[], help='Search a name, e.g. "John Smith" (repeatable)')
+    p.add_argument("--city", default="", help="City for --name searches (needs --state)")
+    p.add_argument("-s", "--state", default="", help="Two-letter state code for --name searches, e.g. WA")
+    p.add_argument("--pages", type=int, default=1, help="Result pages per name, 25 people each (default: 1)")
+    p.add_argument("--profiles", action="store_true", help="Also open every person page found by --name (1 request each)")
+    p.add_argument("--limit", type=int, default=0, help="Stop after this many people (default: no limit)")
+    p.add_argument("-t", "--token", help=f"Scrape.do API token (default: ${TOKEN_ENV})")
+    p.add_argument("-f", "--format", choices=("json", "csv"), default="json", help="Output format (default: json)")
+    p.add_argument("-o", "--output", help="Write to this file instead of stdout")
+    p.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+    return p
+
+
+def _write(people: List[Person], fmt: str, stream) -> None:
+    if fmt == "csv":
+        writer = csv.DictWriter(stream, fieldnames=CSV_FIELDS)
+        writer.writeheader()
+        for person in people:
+            writer.writerow(person.to_row())
+    else:
+        json.dump([p.to_dict() for p in people], stream, indent=2, ensure_ascii=False)
+        stream.write("\n")
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    args = build_parser().parse_args(argv)
+    urls = list(args.urls)
+    if args.input:
+        with open(args.input, encoding="utf-8") as handle:
+            urls.extend(l.strip() for l in handle if l.strip() and not l.startswith("#"))
+    if not urls and not args.name:
+        print("Nothing to do. Pass person page URLs, --input FILE or --name NAME.", file=sys.stderr)
+        return 2
+    try:
+        wp = WhitepagesScraper(token=args.token)
+    except ValueError as exc:
+        print(exc, file=sys.stderr)
+        return 2
+
+    people: List[Person] = []
+    failures = 0
+    for name in args.name:
+        try:
+            found = wp.search(name, args.city, args.state, args.pages)
+        except (ScrapeError, ValueError) as exc:
+            failures += 1
+            print(f"fail  search {name!r}: {exc}", file=sys.stderr)
+            continue
+        print(f"found {len(found)} people for {name!r}", file=sys.stderr)
+        if args.profiles:
+            urls.extend(p.url for p in found if p.url and p.url not in urls)
+        else:
+            people.extend(found)
+    if args.limit > 0:
+        urls = urls[: max(0, args.limit - len(people))]
+    for url in urls:
+        try:
+            people.append(wp.get_profile(url))
+            print(f"ok    {url}", file=sys.stderr)
+        except (ScrapeError, ValueError) as exc:
+            failures += 1
+            print(f"fail  {url}: {exc}", file=sys.stderr)
+    if args.limit > 0:
+        people = people[: args.limit]
+
+    if args.output:
+        with open(args.output, "w", encoding="utf-8", newline="" if args.format == "csv" else None) as handle:
+            _write(people, args.format, handle)
+        print(f"Saved {len(people)} people to {args.output} ({wp.credits_used} credits)", file=sys.stderr)
+    else:
+        _write(people, args.format, sys.stdout)
+    return 1 if failures and not people else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

whitepages_scraper-0.1.0/src/whitepages_scraper/client.py

@@ -0,0 +1,110 @@
+"""Fetch Whitepages pages through the Scrape.do API.
+
+Whitepages only answers visitors from the United States (other IPs, datacenter
+ranges and most VPNs are dropped) and puts Cloudflare in front of every page,
+so a plain HTTP client never gets the HTML. Sending the request through
+Scrape.do with ``super=true`` and ``geoCode=us`` routes it through a US
+residential IP, which clears both. Each page is 10 credits.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import time
+from typing import Iterator, List, Optional
+
+import requests
+
+from .parser import Person, is_profile_url, name_search_url, parse_profile, parse_search_results
+
+API_URL = "https://api.scrape.do/"
+TOKEN_ENV = "SCRAPEDO_TOKEN"
+BLOCK_RE = re.compile(r"just a moment\.\.\.|cf-chl-|attention required|cf-turnstile", re.I)
+
+
+class ScrapeError(RuntimeError):
+    """Raised when a page cannot be fetched."""
+
+    def __init__(self, message: str, status_code: Optional[int] = None):
+        super().__init__(message)
+        self.status_code = status_code
+
+
+class WhitepagesScraper:
+    """Search Whitepages by name and read person pages.
+
+    >>> wp = WhitepagesScraper(token="YOUR_SCRAPEDO_TOKEN")
+    >>> people = wp.search("John Smith", city="Seattle", state="WA")
+    >>> people[0].name, people[0].age_range, people[0].phone
+    >>> person = wp.get_profile(people[0].url)
+    """
+
+    def __init__(self, token: Optional[str] = None, timeout: float = 90.0, retries: int = 2,
+                 session: Optional[requests.Session] = None):
+        self.token = token or os.environ.get(TOKEN_ENV, "")
+        if not self.token:
+            raise ValueError(f"A Scrape.do API token is required. Pass token=... or set {TOKEN_ENV}.")
+        self.timeout = timeout
+        self.retries = max(0, retries)
+        self.session = session or requests.Session()
+        self.requests_made = 0
+
+    def fetch(self, url: str) -> str:
+        """Raw HTML of ``url`` from a US residential IP. Cloudflare pages are retried."""
+        params = {"token": self.token, "url": url, "super": "true", "geoCode": "us"}
+        last_error: Optional[ScrapeError] = None
+        for attempt in range(self.retries + 1):
+            try:
+                response = self.session.get(API_URL, params=params, timeout=self.timeout)
+            except requests.RequestException as exc:
+                last_error = ScrapeError(f"Request failed: {exc}")
+            else:
+                if response.status_code == 200:
+                    self.requests_made += 1
+                    if not BLOCK_RE.search(response.text[:5000]):
+                        return response.text
+                    last_error = ScrapeError(f"Cloudflare challenge page for {url}", 200)
+                else:
+                    last_error = ScrapeError(f"Request for {url} failed with status {response.status_code}",
+                                             status_code=response.status_code)
+                    if 400 <= response.status_code < 500 and response.status_code != 429:
+                        break
+            if attempt < self.retries:
+                time.sleep(2 * (attempt + 1))
+        assert last_error is not None
+        raise last_error
+
+    def search(self, name: str, city: str = "", state: str = "", pages: int = 1) -> List[Person]:
+        """People for a name search, 25 per page. Stops when a page brings nobody new."""
+        people: List[Person] = []
+        seen = set()
+        for page in range(1, max(1, pages) + 1):
+            found = parse_search_results(self.fetch(name_search_url(name, city, state, page)))
+            fresh = [p for p in found if (p.url or p.name + p.address) not in seen]
+            if not fresh:
+                break
+            for person in fresh:
+                seen.add(person.url or person.name + person.address)
+                people.append(person)
+            if len(found) < 25:
+                break
+        return people
+
+    def get_profile(self, url: str) -> Person:
+        """One person page (``/name/<Name>/<City-ST>/<id>``)."""
+        if not is_profile_url(url):
+            raise ValueError(f"Not a Whitepages person page: {url}")
+        if url.startswith("/"):
+            url = "https://www.whitepages.com" + url
+        return parse_profile(self.fetch(url), url=url)
+
+    def search_profiles(self, name: str, city: str = "", state: str = "", pages: int = 1, limit: int = 0) -> Iterator[Person]:
+        """Search, then open each person page (one more request per person)."""
+        for person in self.search(name, city, state, pages)[: limit or None]:
+            if person.url:
+                yield self.get_profile(person.url)
+
+    @property
+    def credits_used(self) -> int:
+        return self.requests_made * 10

whitepages_scraper-0.1.0/src/whitepages_scraper/parser.py

@@ -0,0 +1,274 @@
+"""Parse Whitepages.com name search pages and person pages.
+
+Whitepages renders its pages on the server and publishes the people on them as
+schema.org JSON-LD for search engines:
+
+* A **name search page** (``/name/John-Smith/Seattle-WA``) carries one ``ItemList``
+  block with up to 25 ``Person`` items: full name, current street address, phone
+  numbers, an age range, the neighborhood, previous locations, relatives and the
+  link to the person page. The visible cards mask the phone ("(206) *** ****");
+  the JSON-LD block does not.
+* A **person page** (``/name/John-Smith/Seattle-WA/PAbc123XyZ9``) carries one
+  ``Person`` block with phones under ``contactPoint``, relatives (``relatedTo``) and
+  associates (``knows``). Its current address is often only in the HTML
+  (``.address-line1`` and ``.address-line2``), and the age range is in the text
+  ("John Smith is in their 60s").
+
+Ages are ranges ("60s", "80+"), never exact, and free pages show at most the
+current phone; phone history and emails are behind Whitepages' paid tier.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import asdict, dataclass, field
+from typing import Any, Dict, Iterable, List, Optional
+from urllib.parse import quote
+
+from bs4 import BeautifulSoup
+
+BASE_URL = "https://www.whitepages.com"
+PROFILE_RE = re.compile(r"^(?:https?://(?:www\.)?whitepages\.com)?/name/[^/?#]+/[^/?#]+/[A-Za-z0-9]{8,14}/?(?:[?#].*)?$")
+AGE_TEXT_RE = re.compile(r"\bin (?:their|his|her) (\d{2}s|\d{2}\+)", re.I)
+AGE_RANGE_RE = re.compile(r"^\d{2}(?:s|\+)$")
+CITY_LINE_RE = re.compile(r"^\s*(.+?),\s*([A-Z]{2})\s*(\d{5}(?:-\d{4})?)?\s*$")
+
+
+@dataclass
+class Person:
+    """One person from a Whitepages search result or person page."""
+
+    name: str = ""
+    first_name: str = ""
+    middle_name: str = ""
+    last_name: str = ""
+    age_range: str = ""
+    address: str = ""
+    city: str = ""
+    state: str = ""
+    zip_code: str = ""
+    neighborhood: str = ""
+    phones: List[str] = field(default_factory=list)
+    aliases: List[str] = field(default_factory=list)
+    relatives: List[str] = field(default_factory=list)
+    associates: List[str] = field(default_factory=list)
+    previous_locations: List[str] = field(default_factory=list)
+    url: str = ""
+
+    @property
+    def phone(self) -> str:
+        return self.phones[0] if self.phones else ""
+
+    @property
+    def profile_id(self) -> str:
+        match = re.search(r"/([A-Za-z0-9]{8,14})/?$", self.url.split("?")[0])
+        return match.group(1) if match else ""
+
+    def to_dict(self) -> Dict[str, Any]:
+        return asdict(self)
+
+    def to_row(self, separator: str = " | ") -> Dict[str, Any]:
+        row = self.to_dict()
+        for key in ("phones", "aliases", "relatives", "associates", "previous_locations"):
+            row[key] = separator.join(row[key])
+        return row
+
+
+CSV_FIELDS = [
+    "name", "first_name", "middle_name", "last_name", "age_range", "address", "city", "state", "zip_code",
+    "neighborhood", "phones", "aliases", "relatives", "associates", "previous_locations", "url",
+]
+
+
+# --------------------------------------------------------------------------- helpers
+
+
+def _squash(value: Any) -> str:
+    return " ".join(str(value).split()) if value is not None else ""
+
+
+def _as_list(value: Any) -> List[Any]:
+    if value is None:
+        return []
+    return value if isinstance(value, list) else [value]
+
+
+def _unique(values: Iterable[str]) -> List[str]:
+    seen, out = set(), []
+    for value in values:
+        value = _squash(value)
+        if value and value not in seen:
+            seen.add(value)
+            out.append(value)
+    return out
+
+
+def _names(values: Any) -> List[str]:
+    return _unique(v.get("name", "") if isinstance(v, dict) else str(v) for v in _as_list(values))
+
+
+def format_phone(value: str) -> str:
+    """Normalize ``+1-206-555-0142`` and similar to ``(206) 555-0142``; masked numbers return ""."""
+    value = _squash(value)
+    if not value or "*" in value:
+        return ""
+    digits = re.sub(r"\D", "", value)
+    if len(digits) == 11 and digits.startswith("1"):
+        digits = digits[1:]
+    if len(digits) != 10:
+        return value
+    return f"({digits[:3]}) {digits[3:6]}-{digits[6:]}"
+
+
+def absolute_url(url: str) -> str:
+    url = _squash(url)
+    if url.startswith("/"):
+        url = BASE_URL + url
+    return url.split("#")[0]
+
+
+def _slug(text: str) -> str:
+    words = re.findall(r"[A-Za-z0-9'.]+", text or "")
+    return "-".join(w.replace("'", "").replace(".", "") for w in words if w.strip("'."))
+
+
+def name_search_url(name: str, city: str = "", state: str = "", page: int = 1) -> str:
+    """Build a name search URL: ``/name/John-Smith``, ``/name/John-Smith/WA`` or ``/name/John-Smith/Seattle-WA``.
+
+    ``page`` adds ``?page=N`` (25 people per page).
+    """
+    slug = _slug(name)
+    if not slug or not re.search(r"[A-Za-z]", slug):
+        raise ValueError(f"Not a name: {name!r}")
+    state = (state or "").strip().upper()
+    if state and not re.fullmatch(r"[A-Z]{2}", state):
+        raise ValueError(f"State must be a two-letter code like WA, got {state!r}")
+    if city and not state:
+        raise ValueError("A city needs a state, e.g. city='Seattle', state='WA'")
+    url = f"{BASE_URL}/name/{quote(slug)}"
+    if city:
+        url += f"/{quote(_slug(city))}-{state}"
+    elif state:
+        url += f"/{state}"
+    if page and int(page) > 1:
+        url += f"?page={int(page)}"
+    return url
+
+
+def iter_jsonld(soup: BeautifulSoup) -> Iterable[Dict[str, Any]]:
+    for script in soup.find_all("script", type="application/ld+json"):
+        try:
+            data = json.loads(script.string or "")
+        except (TypeError, ValueError):
+            continue
+        for item in _as_list(data):
+            if isinstance(item, dict):
+                yield item
+
+
+def _apply_address(person: Person, address: Any) -> None:
+    for addr in _as_list(address):
+        if not isinstance(addr, dict):
+            continue
+        person.address = _squash(addr.get("streetAddress"))
+        person.city = _squash(addr.get("addressLocality"))
+        person.state = _squash(addr.get("addressRegion"))
+        person.zip_code = _squash(addr.get("postalCode"))
+        if person.address or person.city:
+            return
+
+
+def _person_from_jsonld(item: Dict[str, Any]) -> Person:
+    person = Person(
+        name=_squash(item.get("name")),
+        first_name=_squash(item.get("givenName")),
+        middle_name=" ".join(_unique(_as_list(item.get("additionalName")))),
+        last_name=_squash(item.get("familyName")),
+        url=absolute_url(item.get("url") or item.get("@id") or ""),
+    )
+    _apply_address(person, item.get("address"))
+    phones = [format_phone(p) for p in _as_list(item.get("telephone"))]
+    for cp in _as_list(item.get("contactPoint")):
+        if isinstance(cp, dict):
+            phones.append(format_phone(cp.get("telephone", "")))
+    person.phones = _unique(p for p in phones if p)
+    person.aliases = [a for a in _unique(_as_list(item.get("alternateName"))) if a != person.name]
+    person.relatives = _names(item.get("relatedTo"))
+    person.associates = _names(item.get("knows"))
+    for prop in _as_list(item.get("additionalProperty")):
+        if not isinstance(prop, dict):
+            continue
+        key, value = _squash(prop.get("name")).lower(), _squash(prop.get("value"))
+        if key == "age range" and AGE_RANGE_RE.match(value):
+            person.age_range = value
+        elif key == "neighborhood":
+            person.neighborhood = value
+        elif key == "previous location" and value:
+            person.previous_locations.append(value)
+    person.previous_locations = _unique(person.previous_locations)
+    if not person.age_range:
+        match = AGE_TEXT_RE.search(_squash(item.get("description")))
+        if match:
+            person.age_range = match.group(1)
+    return person
+
+
+# --------------------------------------------------------------------------- pages
+
+
+def parse_search_results(html: str) -> List[Person]:
+    """People on a name search page, from its ``ItemList`` JSON-LD (up to 25)."""
+    soup = BeautifulSoup(html, "html.parser")
+    people: List[Person] = []
+    for block in iter_jsonld(soup):
+        if block.get("@type") != "ItemList":
+            continue
+        for element in _as_list(block.get("itemListElement")):
+            item = element.get("item") if isinstance(element, dict) else None
+            if isinstance(item, dict) and item.get("@type") == "Person":
+                people.append(_person_from_jsonld(item))
+    # Ages missing from the JSON-LD are often on the visible card
+    cards = {}
+    for card in soup.select(".serp-card"):
+        link = card.select_one("a.name-link-text, [data-qa-selector=organic-card-person-name]")
+        age = card.select_one(".person-age")
+        if link and link.get("href") and age:
+            cards[absolute_url(link["href"])] = _squash(age.get_text())
+    for person in people:
+        if not person.age_range and AGE_RANGE_RE.match(cards.get(person.url, "")):
+            person.age_range = cards[person.url]
+    return people
+
+
+def parse_profile(html: str, url: str = "") -> Person:
+    """A person page: JSON-LD ``Person`` plus the address and age from the HTML."""
+    soup = BeautifulSoup(html, "html.parser")
+    item = next((b for b in iter_jsonld(soup) if b.get("@type") == "Person"), None)
+    if item is None:
+        raise ValueError("No Person JSON-LD on this page; it is not a Whitepages person page")
+    person = _person_from_jsonld(item)
+    if url:
+        person.url = absolute_url(url)
+    if not person.address:
+        line1 = soup.select_one(".address-line1")
+        line2 = soup.select_one(".address-line2")
+        if line1:
+            person.address = _squash(line1.get_text(" "))
+        if line2:
+            match = CITY_LINE_RE.match(_squash(line2.get_text(" ")))
+            if match:
+                person.city, person.state, person.zip_code = match.group(1), match.group(2), match.group(3) or ""
+    if not person.age_range:
+        for el in soup.select("[data-qa-selector=person-age-mobile], .person-age"):
+            match = re.search(r"\d{2}(?:s|\+)", el.get_text(" "))
+            if match:
+                person.age_range = match.group(0)
+                break
+    html_phones = [format_phone(a.get_text()) for a in soup.select("[data-qa-selector=phone-number-link]")]
+    person.phones = _unique(person.phones + [p for p in html_phones if p])
+    return person
+
+
+def is_profile_url(url: str) -> bool:
+    return bool(PROFILE_RE.match(_squash(url)))

whitepages_scraper-0.1.0/src/whitepages_scraper.egg-info/PKG-INFO

@@ -0,0 +1,115 @@
+Metadata-Version: 2.4
+Name: whitepages-scraper
+Version: 0.1.0
+Summary: Scrape Whitepages.com name searches and person pages (name, age range, address, phones, relatives, associates) with Python.
+Author: Buğrahan Saka
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/bugrahansss/whitepages-scraper
+Project-URL: Source, https://github.com/bugrahansss/whitepages-scraper
+Project-URL: Issues, https://github.com/bugrahansss/whitepages-scraper/issues
+Project-URL: Guide, https://scrape.do/blog/white-pages-scraping/
+Keywords: whitepages,web scraping,people search,scraper,json-ld,cloudflare,scrapedo
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.25
+Requires-Dist: beautifulsoup4>=4.9
+Dynamic: license-file
+
+# Whitepages Scraper
+
+[![PyPI](https://img.shields.io/pypi/v/whitepages-scraper)](https://pypi.org/project/whitepages-scraper/)
+[![Python](https://img.shields.io/pypi/pyversions/whitepages-scraper)](https://pypi.org/project/whitepages-scraper/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+Scrape [Whitepages.com](https://www.whitepages.com) people search into structured data. Search a name (optionally in a state or city) and get every person on the result pages with their **current address, phone numbers, age range, neighborhood, previous locations and relatives**, then open person pages for **associates** and the rest. JSON or CSV.
+
+```bash
+pip install whitepages-scraper
+export SCRAPEDO_TOKEN=your_token
+whitepages-scraper --name "John Smith" --city Seattle --state WA --pages 2 -f csv -o people.csv
+```
+
+## What is in the way
+
+```python
+import requests
+
+requests.get("https://www.whitepages.com/name/John-Smith/Seattle-WA")
+# outside the US: the connection is dropped; from a datacenter IP: Cloudflare challenge
+```
+
+1. **US only.** Visitors outside the United States, datacenter IPs and most VPNs are dropped without an error page.
+2. **Cloudflare.** Every page sits behind Cloudflare's bot checks, so a plain HTTP client gets a challenge instead of HTML.
+
+This package fetches every page through [Scrape.do](https://scrape.do/?utm_source=github&utm_medium=readme&utm_campaign=whitepages-scraper) with `super=true&geoCode=us` (a US residential IP): about 3 seconds and 10 credits per page.
+
+## Where the data is
+
+Whitepages renders on the server and publishes the people on each page as schema.org JSON-LD:
+
+* **Search pages** carry an `ItemList` with up to **25 `Person` items**: full name, the current street address, phone numbers, `Age Range`, `Neighborhood` and `Previous Location` properties, relatives (`relatedTo`) and the person page URL. The visible cards mask phones ("(206) \*\*\* \*\*\*\*"); the JSON-LD has the full number. In a live test, one Seattle search page had 25 people, 25 addresses and 15 phone numbers in one request.
+* **Person pages** carry one `Person` with phones under `contactPoint`, relatives and associates (`knows`). The address is often only in the HTML (`.address-line1`, `.address-line2`) and the age range in the text ("is in their 60s"); the parser reads both.
+* **Pagination** is `?page=2`, `?page=3`, 25 people each.
+
+Ages are ranges ("40s", "80+"), never exact. Free pages show the current phone at most; phone history, emails and background reports are behind Whitepages' paid tier and are not scraped.
+
+## Python
+
+```python
+from whitepages_scraper import WhitepagesScraper
+
+wp = WhitepagesScraper(token="YOUR_SCRAPEDO_TOKEN")  # or set SCRAPEDO_TOKEN
+
+people = wp.search("John Smith", city="Seattle", state="WA", pages=2)
+for p in people[:3]:
+    print(p.name, p.age_range, p.address, p.city, p.phone, p.relatives)
+
+person = wp.get_profile(people[0].url)
+print(person.associates, person.phones)
+print(wp.credits_used)
+```
+
+## Command line
+
+```bash
+# Name search: 25 people per page, no extra requests
+whitepages-scraper --name "Mary Johnson" --state TX --pages 2 -f csv -o mary.csv
+
+# Open every person page found (one more request each)
+whitepages-scraper --name "John Smith" --city Seattle --state WA --profiles --limit 10
+
+# Person page URLs
+whitepages-scraper "https://www.whitepages.com/name/<Name>/<City-ST>/<id>"
+```
+
+## Fields
+
+`name`, `first_name`, `middle_name`, `last_name`, `age_range`, `address`, `city`, `state`, `zip_code`, `neighborhood`, `phones`, `aliases`, `relatives`, `associates`, `previous_locations`, `url`.
+
+## Use it responsibly
+
+This is personal data about real people. Whitepages is not a consumer reporting agency: don't use this data for decisions about employment, housing, credit or insurance (the FCRA regulates those in the US). Honor opt-out requests, follow the privacy laws that apply to you (CCPA, GDPR) and Whitepages' terms, and keep request rates modest.
+
+## Also available
+
+- **No-code:** [Whitepages Scraper on Apify](https://apify.com/scrapedo/whitepages-scraper)
+- **Browser demo:** [Hugging Face Space](https://huggingface.co/spaces/bugrahans/whitepages-scraper)
+- **Full guide:** [Whitepages.com Scraping: Bypass Cloudflare and Extract People Data](https://scrape.do/blog/white-pages-scraping/?utm_source=github&utm_medium=readme&utm_campaign=whitepages-scraper)
+
+## Tests
+
+```bash
+python -m unittest discover -s tests
+```
+
+The fixtures are fictional people (Jane Doe, 555 numbers); the tests never hit the network.
+
+## License
+
+MIT

whitepages_scraper-0.1.0/src/whitepages_scraper.egg-info/SOURCES.txt

@@ -0,0 +1,19 @@
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+examples/search_to_csv.py
+src/whitepages_scraper/__init__.py
+src/whitepages_scraper/cli.py
+src/whitepages_scraper/client.py
+src/whitepages_scraper/parser.py
+src/whitepages_scraper.egg-info/PKG-INFO
+src/whitepages_scraper.egg-info/SOURCES.txt
+src/whitepages_scraper.egg-info/dependency_links.txt
+src/whitepages_scraper.egg-info/entry_points.txt
+src/whitepages_scraper.egg-info/requires.txt
+src/whitepages_scraper.egg-info/top_level.txt
+tests/test_parser.py
+tests/fixtures/challenge.html
+tests/fixtures/profile_jane_doe.html
+tests/fixtures/search_jane_doe.html

whitepages_scraper-0.1.0/src/whitepages_scraper.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

whitepages_scraper-0.1.0/src/whitepages_scraper.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+whitepages-scraper = whitepages_scraper.cli:main

whitepages_scraper-0.1.0/src/whitepages_scraper.egg-info/requires.txt

@@ -0,0 +1,2 @@
+requests>=2.25
+beautifulsoup4>=4.9

whitepages_scraper-0.1.0/src/whitepages_scraper.egg-info/top_level.txt

@@ -0,0 +1 @@
+whitepages_scraper

whitepages_scraper-0.1.0/tests/fixtures/challenge.html

@@ -0,0 +1 @@
+<html><head><title>Just a moment...</title></head><body><div class="cf-turnstile"></div></body></html>

whitepages_scraper-0.1.0/tests/fixtures/profile_jane_doe.html

@@ -0,0 +1,3 @@
+<!doctype html><html><head><title>Jane Doe | 40s | 100 Example St Apt 1, Seattle, WA | Whitepages</title><script type="application/ld+json">{"@context": "https://schema.org", "@type": "Person", "description": "Jane Doe is in their 40s, currently living in Seattle, WA", "name": "Jane Doe", "givenName": "Jane", "familyName": "Doe", "@id": "https://www.whitepages.com/name/Jane-Doe/Seattle-WA/PExample001#main", "url": "https://www.whitepages.com/name/Jane-Doe/Seattle-WA/PExample001", "alternateName": ["Jane Q Doe"], "contactPoint": [{"@type": "ContactPoint", "telephone": "+1-206-555-0101", "contactType": "landline"}], "address": [], "relatedTo": [{"@type": "Person", "name": "John Doe", "url": "/name/John-Doe/Seattle-WA/PRel0000001"}], "knows": [{"@type": "Person", "name": "Alex Roe"}]}</script><script type="application/ld+json">{"@type":"WebPage"}</script></head>
+<body><div class="big-name">Jane Doe</div><div data-qa-selector="person-age-mobile">Age 40s</div><div class="address-line1">100 Example St Apt 1</div><div class="address-line2">Seattle, WA 98101</div>
+<a data-qa-selector="phone-number-link" href="tel:2065550101">(206) 555-0101</a><a data-qa-selector="phone-number-link">(206) 555-0199</a><span data-qa-selector="masked-phone">(206) ***-****</span></body></html>

whitepages_scraper-0.1.0/tests/fixtures/search_jane_doe.html

@@ -0,0 +1 @@
+<!doctype html><html><head><title>Jane Doe in Seattle, Washington (WA) | Whitepages</title><script type="application/ld+json">[{"@context": "https://schema.org", "@type": "ItemList", "name": "Jane Doe", "numberOfItems": 3, "itemListElement": [{"@type": "ListItem", "position": 1, "item": {"@type": "Person", "name": "Jane Doe", "givenName": "Jane", "familyName": "Doe", "description": "Jane Doe lives at 100 Example St Apt 1 Seattle WA 98101 in the Capitol Hill neighborhood.", "@id": "/name/Jane-Doe/Seattle-WA/PExample001#main", "url": "/name/Jane-Doe/Seattle-WA/PExample001", "address": [{"@type": "PostalAddress", "streetAddress": "100 Example St Apt 1", "addressLocality": "Seattle", "addressRegion": "WA", "addressCountry": {"@type": "Country", "name": "US"}, "postalCode": "98101"}], "additionalProperty": [{"@type": "PropertyValue", "name": "Neighborhood", "value": "Capitol Hill"}, {"@type": "PropertyValue", "name": "Age Range", "value": "40s"}, {"@type": "PropertyValue", "name": "Previous Location", "value": "Tacoma, WA"}], "relatedTo": [{"@type": "Person", "name": "John Doe", "url": "/name/x/y/PRel0000001"}, {"@type": "Person", "name": "Mary Doe", "url": "/name/x/y/PRel0000001"}], "knows": [], "telephone": ["(206) 555-0101"], "additionalName": "Q", "alternateName": ["Jane Q Doe", "Jane Doe"]}}, {"@type": "ListItem", "position": 2, "item": {"@type": "Person", "name": "Jane Doe", "givenName": "Jane", "familyName": "Doe", "description": "Jane Doe lives at 200 Sample Ave Seattle WA 98101 in the Fremont neighborhood.", "@id": "/name/Jane-Doe/Seattle-WA/PExample002#main", "url": "/name/Jane-Doe/Seattle-WA/PExample002", "address": [{"@type": "PostalAddress", "streetAddress": "200 Sample Ave", "addressLocality": "Seattle", "addressRegion": "WA", "addressCountry": {"@type": "Country", "name": "US"}, "postalCode": "98101"}], "additionalProperty": [{"@type": "PropertyValue", "name": "Neighborhood", "value": "Fremont"}], "relatedTo": [], "knows": []}}, {"@type": "ListItem", "position": 3, "item": {"@type": "Person", "name": "Jane Doe", "givenName": "Jane", "familyName": "Doe", "description": "Jane Doe lives at 300 Test Rd Seattle WA 98101 in the Capitol Hill neighborhood.", "@id": "/name/Jane-Doe/Seattle-WA/PExample003#main", "url": "/name/Jane-Doe/Seattle-WA/PExample003", "address": [{"@type": "PostalAddress", "streetAddress": "300 Test Rd", "addressLocality": "Seattle", "addressRegion": "WA", "addressCountry": {"@type": "Country", "name": "US"}, "postalCode": "98101"}], "additionalProperty": [{"@type": "PropertyValue", "name": "Neighborhood", "value": "Capitol Hill"}, {"@type": "PropertyValue", "name": "Age Range", "value": "80+"}], "relatedTo": [], "knows": [], "telephone": ["+1-206-555-0103"]}}]}]</script><script type="application/ld+json">{"@type":"FAQPage"}</script></head><body><div class="serp-card"><dl class="age-container"><dt class="age-title">AGE</dt><dd class="person-age">40s</dd></dl><a class="amp-event name-link-text" data-qa-selector="organic-card-person-name" href="/name/Jane-Doe/Seattle-WA/PExample001">Jane Doe</a><dd class="person-location-city">Seattle, WA</dd><div>Phone (206) *** ****</div></div><div class="serp-card"><dl class="age-container"><dt class="age-title">AGE</dt><dd class="person-age">50s</dd></dl><a class="amp-event name-link-text" data-qa-selector="organic-card-person-name" href="/name/Jane-Doe/Seattle-WA/PExample002">Jane Doe</a><dd class="person-location-city">Seattle, WA</dd><div>Phone (206) *** ****</div></div><div class="serp-card"><dl class="age-container"><dt class="age-title">AGE</dt><dd class="person-age">80+</dd></dl><a class="amp-event name-link-text" data-qa-selector="organic-card-person-name" href="/name/Jane-Doe/Seattle-WA/PExample003">Jane Doe</a><dd class="person-location-city">Seattle, WA</dd><div>Phone (206) *** ****</div></div></body></html>

whitepages_scraper-0.1.0/tests/test_parser.py

@@ -0,0 +1,164 @@
+import csv
+import io
+import os
+import sys
+import unittest
+from pathlib import Path
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
+
+from whitepages_scraper import (  # noqa: E402
+    CSV_FIELDS,
+    ScrapeError,
+    WhitepagesScraper,
+    format_phone,
+    is_profile_url,
+    name_search_url,
+    parse_profile,
+    parse_search_results,
+)
+from whitepages_scraper import cli  # noqa: E402
+
+FIX = Path(__file__).parent / "fixtures"
+SEARCH = (FIX / "search_jane_doe.html").read_text(encoding="utf-8")
+PROFILE = (FIX / "profile_jane_doe.html").read_text(encoding="utf-8")
+CHALLENGE = (FIX / "challenge.html").read_text(encoding="utf-8")
+PURL = "https://www.whitepages.com/name/Jane-Doe/Seattle-WA/PExample001"
+
+
+class R:
+    def __init__(self, status, text=""):
+        self.status_code, self.text = status, text
+
+
+class Session:
+    def __init__(self, responses):
+        self.responses, self.calls = list(responses), []
+
+    def get(self, url, params=None, timeout=None):
+        self.calls.append(dict(params))
+        return self.responses.pop(0)
+
+
+class ParserTests(unittest.TestCase):
+    def test_search_url(self):
+        self.assertEqual(name_search_url("John Smith"), "https://www.whitepages.com/name/John-Smith")
+        self.assertEqual(name_search_url("John Smith", state="wa"), "https://www.whitepages.com/name/John-Smith/WA")
+        self.assertEqual(name_search_url("John  Smith", "Seattle", "WA", 2), "https://www.whitepages.com/name/John-Smith/Seattle-WA?page=2")
+        self.assertEqual(name_search_url("Mary O'Neil", "San Jose", "CA"), "https://www.whitepages.com/name/Mary-ONeil/San-Jose-CA")
+
+    def test_search_url_errors(self):
+        for args in (("",), ("123",), ("John", "", "Texas"), ("John", "Seattle", "")):
+            with self.assertRaises(ValueError):
+                name_search_url(*args)
+
+    def test_format_phone(self):
+        self.assertEqual(format_phone("+1-206-555-0103"), "(206) 555-0103")
+        self.assertEqual(format_phone("(206) 555-0101"), "(206) 555-0101")
+        self.assertEqual(format_phone("(206) *** ****"), "")
+        self.assertEqual(format_phone(""), "")
+
+    def test_parse_search(self):
+        people = parse_search_results(SEARCH)
+        self.assertEqual(len(people), 3)
+        jane = people[0]
+        self.assertEqual((jane.name, jane.first_name, jane.middle_name, jane.last_name), ("Jane Doe", "Jane", "Q", "Doe"))
+        self.assertEqual((jane.address, jane.city, jane.state, jane.zip_code), ("100 Example St Apt 1", "Seattle", "WA", "98101"))
+        self.assertEqual(jane.phones, ["(206) 555-0101"])
+        self.assertEqual(jane.age_range, "40s")
+        self.assertEqual(jane.neighborhood, "Capitol Hill")
+        self.assertEqual(jane.previous_locations, ["Tacoma, WA"])
+        self.assertEqual(jane.relatives, ["John Doe", "Mary Doe"])
+        self.assertEqual(jane.aliases, ["Jane Q Doe"])
+        self.assertEqual(jane.url, PURL)
+        self.assertEqual(jane.profile_id, "PExample001")
+
+    def test_search_age_falls_back_to_card(self):
+        people = parse_search_results(SEARCH)
+        self.assertEqual(people[1].age_range, "50s")  # not in JSON-LD, read from the card
+        self.assertEqual(people[1].phones, [])
+        self.assertEqual(people[2].phones, ["(206) 555-0103"])
+        self.assertEqual(people[2].age_range, "80+")
+
+    def test_parse_search_empty(self):
+        self.assertEqual(parse_search_results("<html></html>"), [])
+
+    def test_parse_profile(self):
+        p = parse_profile(PROFILE)
+        self.assertEqual(p.name, "Jane Doe")
+        self.assertEqual(p.age_range, "40s")
+        self.assertEqual((p.address, p.city, p.state, p.zip_code), ("100 Example St Apt 1", "Seattle", "WA", "98101"))
+        self.assertEqual(p.phones, ["(206) 555-0101", "(206) 555-0199"])
+        self.assertEqual(p.relatives, ["John Doe"])
+        self.assertEqual(p.associates, ["Alex Roe"])
+        self.assertEqual(p.aliases, ["Jane Q Doe"])
+
+    def test_parse_profile_rejects_other_pages(self):
+        with self.assertRaises(ValueError):
+            parse_profile(SEARCH)
+
+    def test_profile_url(self):
+        self.assertTrue(is_profile_url(PURL))
+        self.assertTrue(is_profile_url("/name/Jane-Doe/Seattle-WA/PExample001"))
+        self.assertFalse(is_profile_url("https://www.whitepages.com/name/Jane-Doe/Seattle-WA"))
+        self.assertFalse(is_profile_url("https://example.com/name/a/b/PExample001"))
+
+    def test_rows(self):
+        row = parse_search_results(SEARCH)[0].to_row()
+        self.assertEqual(list(row), CSV_FIELDS)
+        self.assertEqual(row["relatives"], "John Doe | Mary Doe")
+
+
+class ClientTests(unittest.TestCase):
+    def test_needs_token(self):
+        with mock.patch.dict(os.environ, {}, clear=True):
+            with self.assertRaises(ValueError):
+                WhitepagesScraper()
+
+    def test_search_params_and_pagination(self):
+        s = Session([R(200, SEARCH)])
+        wp = WhitepagesScraper(token="T", session=s)
+        people = wp.search("Jane Doe", "Seattle", "WA", pages=3)
+        self.assertEqual(len(people), 3)
+        self.assertEqual(len(s.calls), 1)  # fewer than 25, so no page 2
+        self.assertEqual(s.calls[0]["url"], "https://www.whitepages.com/name/Jane-Doe/Seattle-WA")
+        self.assertEqual((s.calls[0]["super"], s.calls[0]["geoCode"]), ("true", "us"))
+        self.assertEqual(wp.credits_used, 10)
+
+    def test_challenge_retried(self):
+        s = Session([R(200, CHALLENGE), R(200, PROFILE)])
+        with mock.patch("whitepages_scraper.client.time.sleep"):
+            p = WhitepagesScraper(token="T", session=s, retries=1).get_profile(PURL)
+        self.assertEqual(p.name, "Jane Doe")
+        self.assertEqual(len(s.calls), 2)
+
+    def test_client_error_not_retried(self):
+        s = Session([R(401), R(200, PROFILE)])
+        with self.assertRaises(ScrapeError) as ctx:
+            WhitepagesScraper(token="T", session=s, retries=2).get_profile(PURL)
+        self.assertEqual(ctx.exception.status_code, 401)
+        self.assertEqual(len(s.calls), 1)
+
+    def test_rejects_non_profile(self):
+        with self.assertRaises(ValueError):
+            WhitepagesScraper(token="T", session=Session([])).get_profile("https://example.com/x")
+
+
+class CliTests(unittest.TestCase):
+    def test_search_csv(self):
+        out = io.StringIO()
+        with mock.patch("whitepages_scraper.client.requests.Session", return_value=Session([R(200, SEARCH)])), \
+                mock.patch.object(sys, "stdout", out), mock.patch.object(sys, "stderr", io.StringIO()):
+            code = cli.main(["-n", "Jane Doe", "--city", "Seattle", "-s", "WA", "-t", "T", "-f", "csv"])
+        self.assertEqual(code, 0)
+        rows = list(csv.DictReader(io.StringIO(out.getvalue())))
+        self.assertEqual([r["age_range"] for r in rows], ["40s", "50s", "80+"])
+
+    def test_nothing_to_do(self):
+        with mock.patch.object(sys, "stderr", io.StringIO()):
+            self.assertEqual(cli.main(["-t", "T"]), 2)
+
+
+if __name__ == "__main__":
+    unittest.main()