@pencil-agent/nano-pencil 2.0.1 → 2.0.2

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/zillow/scraping.md

@@ -1,433 +1,433 @@
-# Zillow — Scraping & Data Extraction
-
-Field-tested against `www.zillow.com` on 2026-04-18 using `http_get` (no browser).
-
-## Quick summary
-
-- **Search listing pages (`/homes/`, `/sold/`, `/rentals/`)** — `http_get` works with full Chrome headers. Returns ~973 KB HTML with all listing data embedded in `__NEXT_DATA__` JSON.
-- **Individual property detail pages (`/homedetails/`)** — `http_get` returns **HTTP 403** unconditionally. No header combination bypasses this.
-- **Internal API endpoints** (`/async-create-search-page-state`, `/graphql/`) — **403** for all server-side requests regardless of headers.
-- **Redfin** — `http_get` works; HTML contains both JSON-LD per listing and a stingray JSON API.
-
----
-
-## What works: search listing pages via `__NEXT_DATA__`
-
-Zillow search pages embed all listing data in `<script id="__NEXT_DATA__">`. This is standard Next.js SSR output — it is the same data Zillow's React app hydrates from.
-
-**Required headers** — The single-word User-Agent (`"Mozilla/5.0"`) used by `http_get` internally gets 403. You must pass a full Chrome UA plus Accept/Accept-Language headers:
-
-```python
-import re, json
-from helpers import http_get
-
-HEADERS = {
-    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
-    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
-    "Accept-Language": "en-US,en;q=0.9",
-}
-
-def extract_listings(html):
-    """Parse Zillow __NEXT_DATA__ and return list of listing dicts."""
-    m = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
-    if not m:
-        return []
-    d = json.loads(m.group(1))
-    sps = d['props']['pageProps']['searchPageState']
-    return sps['cat1']['searchResults']['listResults']
-
-html = http_get("https://www.zillow.com/homes/San-Francisco,-CA_rb/", headers=HEADERS)
-listings = extract_listings(html)
-print(len(listings))  # 41 — always 41 per page
-```
-
-### Fields available in each listing card
-
-The `listResults` array is the canonical source. Each entry includes:
-
-| Field | Source | Example |
-|---|---|---|
-| `zpid` | listing | `15081707` |
-| `address` | listing | `"212 Spruce St, San Francisco, CA 94118"` |
-| `addressStreet`, `addressCity`, `addressState`, `addressZipcode` | listing | split address components |
-| `price` | listing | `"$4,395,000"` (formatted string) |
-| `unformattedPrice` | listing | `4395000` (int, use for math) |
-| `beds` | listing | `4` |
-| `baths` | listing | `4` |
-| `area` | listing | `4133` (sqft) |
-| `latLong` | listing | `{'latitude': 37.78867, 'longitude': -122.45361}` |
-| `statusType` | listing | `"FOR_SALE"` / `"FOR_RENT"` / `"RECENTLY_SOLD"` |
-| `detailUrl` | listing | full `https://www.zillow.com/homedetails/...` URL |
-| `zestimate` | listing | `4857200` (Zillow AI estimate, int) |
-| `imgSrc` | listing | thumbnail URL |
-| `has3DModel` | listing | `True`/`False` |
-| `hasOpenHouse` | listing | `True`/`False` |
-| `openHouseStartDate`, `openHouseEndDate` | listing | ISO strings |
-| `isFeaturedListing` | listing | sponsored/featured flag |
-| `brokerName` | listing | `"Sotheby's International Realty"` |
-| `statusText` | listing | `"FOR SALE"` display string |
-| `hdpData.homeInfo.price` | nested | raw price int (matches `unformattedPrice`) |
-| `hdpData.homeInfo.zestimate` | nested | raw zestimate int |
-| `hdpData.homeInfo.rentZestimate` | nested | monthly rent estimate |
-| `hdpData.homeInfo.homeType` | nested | `"SINGLE_FAMILY"`, `"CONDO"`, `"TOWNHOUSE"` etc. |
-| `hdpData.homeInfo.daysOnZillow` | nested | int |
-| `hdpData.homeInfo.taxAssessedValue` | nested | int |
-| `hdpData.homeInfo.lotAreaValue` + `lotAreaUnit` | nested | e.g. `2957.724`, `"sqft"` |
-| `hdpData.homeInfo.priceForHDP` | nested | reliable sold price for recently-sold listings |
-
-```python
-# Full extraction snippet
-listing = listings[0]
-hi = listing.get('hdpData', {}).get('homeInfo', {})
-
-record = {
-    "zpid":         listing['zpid'],
-    "address":      listing['address'],
-    "price_raw":    listing.get('unformattedPrice') or hi.get('price'),
-    "beds":         listing.get('beds'),
-    "baths":        listing.get('baths'),
-    "sqft":         listing.get('area'),
-    "lat":          listing['latLong']['latitude'],
-    "lon":          listing['latLong']['longitude'],
-    "status":       listing['statusType'],
-    "zestimate":    listing.get('zestimate'),
-    "rent_zest":    hi.get('rentZestimate'),
-    "home_type":    hi.get('homeType'),
-    "days_listed":  hi.get('daysOnZillow'),
-    "tax_assessed": hi.get('taxAssessedValue'),
-    "url":          listing['detailUrl'],
-}
-```
-
-### Total result count and pagination
-
-```python
-d = json.loads(re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL).group(1))
-sps = d['props']['pageProps']['searchPageState']
-
-# Total listings in this search
-total = sps['categoryTotals']['cat1']['totalResultCount']
-print(total)  # 1037
-
-# Each page returns exactly 41 listings. Add /<N>_p/ for subsequent pages:
-# Page 2: https://www.zillow.com/homes/San-Francisco,-CA_rb/2_p/
-# Page 3: https://www.zillow.com/homes/San-Francisco,-CA_rb/3_p/
-
-max_pages = (total + 40) // 41
-```
-
-### Scrape all pages
-
-```python
-import re, json, time
-from helpers import http_get
-
-HEADERS = {
-    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
-    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
-    "Accept-Language": "en-US,en;q=0.9",
-}
-
-def get_listings(city_slug, page=1):
-    """city_slug: e.g. 'San-Francisco,-CA', 'Seattle,-WA', 'Austin,-TX'"""
-    if page == 1:
-        url = f"https://www.zillow.com/homes/{city_slug}_rb/"
-    else:
-        url = f"https://www.zillow.com/homes/{city_slug}_rb/{page}_p/"
-    html = http_get(url, headers=HEADERS)
-    m = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
-    d = json.loads(m.group(1))
-    sps = d['props']['pageProps']['searchPageState']
-    total = sps['categoryTotals']['cat1']['totalResultCount']
-    listings = sps['cat1']['searchResults']['listResults']
-    return listings, total
-
-all_listings = []
-listings, total = get_listings("San-Francisco,-CA")
-all_listings.extend(listings)
-
-max_pages = (total + 40) // 41
-for page in range(2, min(max_pages + 1, 6)):   # cap at 5 pages for demo
-    time.sleep(1.0)   # polite delay
-    page_listings, _ = get_listings("San-Francisco,-CA", page)
-    all_listings.extend(page_listings)
-
-print(f"Fetched {len(all_listings)} of {total} listings")
-```
-
----
-
-## URL patterns that work (all confirmed)
-
-| URL pattern | Status | Notes |
-|---|---|---|
-| `/homes/{city}_rb/` | **Works** | For-sale listings |
-| `/homes/{city}_rb/{N}_p/` | **Works** | Pagination |
-| `/homes/for_sale/{city}/0-1800000_price/` | **Works** | Price filter (max) |
-| `/homes/3-_beds/{city}/` | **Works** | Bed count filter |
-| `/homes/{zip}_rb/` | **Works** | ZIP code search |
-| `/san-francisco-ca/rentals/` | **Works** | Rental listings |
-| `/san-francisco-ca/sold/` | **Works** | Recently sold |
-| `/homedetails/{address}/{zpid}_zpid/` | **403** | Single property detail |
-| `/async-create-search-page-state` | **403** | Internal search API |
-| `/graphql/` | **400/403** | GraphQL endpoint |
-
----
-
-## Rental listings
-
-Rental search pages use the same `__NEXT_DATA__` structure. However, rental listing cards have a **different schema** — individual units are nested, not a flat price:
-
-```python
-html = http_get("https://www.zillow.com/san-francisco-ca/rentals/", headers=HEADERS)
-listings = extract_listings(html)
-
-r = listings[0]
-# Multi-unit buildings:
-# r['units'] = [{'price': '$3,485+', 'beds': '0', 'roomForRent': False}, ...]
-# r['minBaseRent'] = 3485
-# r['maxBaseRent'] = 7130
-# r['availabilityCount'] = 23
-
-# Single-unit rentals:
-# r['price'] = '$2,500/mo'
-# r['unformattedPrice'] = 2500
-
-# Check which type:
-if r.get('isBuilding'):
-    price_range = f"${r['minBaseRent']}–${r['maxBaseRent']}/mo"
-    units = r.get('units', [])
-else:
-    price = r.get('unformattedPrice') or r.get('hdpData', {}).get('homeInfo', {}).get('price')
-```
-
----
-
-## Sold listings
-
-Sold pages (`/sold/`) work identically. Key difference: `statusType` is `"RECENTLY_SOLD"` and price comes from `hdpData.homeInfo.priceForHDP` (not the `price` field which is `None` in sold cards):
-
-```python
-html = http_get("https://www.zillow.com/san-francisco-ca/sold/", headers=HEADERS)
-listings = extract_listings(html)
-
-for l in listings:
-    hi = l.get('hdpData', {}).get('homeInfo', {})
-    sold_price  = hi.get('priceForHDP')      # actual sold price
-    zestimate   = hi.get('zestimate')
-    tax_value   = hi.get('taxAssessedValue')
-    print(l['address'], f"${sold_price:,}", f"zest=${zestimate}")
-# 999 Green St APT 1702, San Francisco, CA 94133 $3,200,000 zest=$3,403,400
-# 1041 Vallejo St, San Francisco, CA 94133 $6,250,000 zest=None
-```
-
-Total sold inventory in San Francisco: **18,109** (all time in Zillow's database, paginated 41/page).
-
----
-
-## Bot detection behavior
-
-- **Zillow detects bot status server-side** and embeds `window.__USER_SESSION_INITIAL_STATE__` and `props.isBot` in the page.
-- In field testing, the page returned `isBot: False` with the Chrome User-Agent — **Zillow does not block the search pages**.
-- The page does embed `captcha` strings in the HTML (for the CAPTCHA challenge widget code), but the challenge is NOT triggered for search pages.
-- **`/homedetails/` pages do trigger blocking** — every property detail URL tested returned HTTP 403. This is enforced before serving HTML, not via JavaScript CAPTCHA.
-- Rate limiting: 3 rapid sequential requests to `/homes/` all succeeded. Observed no 429s. Add `time.sleep(0.5–1.0)` between pages as a courtesy.
-
----
-
-## What you do NOT get from `http_get`
-
-Because property detail pages are blocked (403), you lose:
-
-- Full property description text
-- All listing photos (you only get `imgSrc` thumbnail from search)
-- Detailed home facts (year built, parking, HVAC, school scores)
-- Price history
-- Nearby comparable sales (comps)
-- Agent contact info
-
-**To get these**, you must navigate to the `/homedetails/` URL in a browser session. The browser is not blocked (Zillow relies on JS challenges and fingerprinting that only trigger in browser context).
-
----
-
-## Alternative: Redfin (field-tested, more accessible)
-
-Redfin allows `http_get` with no blocking for both HTML pages and its internal API.
-
-### Redfin JSON-LD per listing (easiest)
-
-Each Redfin search results page embeds one `<script type="application/ld+json">` per listing with structured property data:
-
-```python
-import re, json
-from helpers import http_get
-
-HEADERS = {
-    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
-    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
-    "Accept-Language": "en-US,en;q=0.9",
-}
-
-html = http_get(
-    "https://www.redfin.com/city/17151/CA/San-Francisco/filter/property-type=house",
-    headers=HEADERS
-)
-print(len(html))  # ~1.6 MB
-
-# Extract all SingleFamilyResidence JSON-LD entries
-properties = []
-for s in re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL):
-    try:
-        d = json.loads(s)
-        if isinstance(d, list):
-            for item in d:
-                if item.get('@type') in ('SingleFamilyResidence', 'House', 'Residence', 'Apartment'):
-                    properties.append(item)
-    except Exception:
-        pass
-
-prop = properties[0]
-print("Name:", prop['name'])               # "662 Hampshire St, San Francisco, CA 94110"
-print("Address:", prop['address'])
-# {'@type': 'PostalAddress', 'streetAddress': '662 Hampshire St',
-#  'addressLocality': 'San Francisco', 'addressRegion': 'CA',
-#  'postalCode': '94110', 'addressCountry': 'US'}
-print("Rooms:", prop['numberOfRooms'])     # 3
-print("Floor size:", prop['floorSize'])    # {'@type': 'QuantitativeValue', 'value': 3350, 'unitCode': 'FTK'}
-print("URL:", prop['url'])
-# https://www.redfin.com/CA/San-Francisco/662-Hampshire-St-94110/home/1533754
-```
-
-Note: The JSON-LD schema does NOT include price (Redfin omits `offers` from the LD+JSON). Use the stingray API below for price.
-
-### Redfin stingray API (structured JSON with price)
-
-Redfin's internal GIS/search API returns rich structured data including price, MLS ID, beds, baths, sqft, agent info, and remarks. Responses are prefixed with `{}&&` — strip it before parsing:
-
-```python
-import json
-from helpers import http_get
-
-def redfin_search(region_id, region_type=6, num_homes=20, page=1, uipt="1,2,3,4,5,6"):
-    """
-    region_type: 6=city, 2=zipcode, 5=county
-    uipt: property types (1=house, 2=condo, 3=townhouse, 4=multi-family, 5=land, 6=other)
-    """
-    url = (
-        f"https://www.redfin.com/stingray/api/gis"
-        f"?al=1&num_homes={num_homes}&ord=redfin-recommended-asc"
-        f"&page_number={page}&region_id={region_id}&region_type={region_type}"
-        f"&sf=1,2,3,5,6,7&status=9&uipt={uipt}&v=8"
-    )
-    raw = http_get(url, headers={
-        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
-        "Referer": "https://www.redfin.com/",
-        "Accept": "*/*",
-    })
-    # Strip the {}&& CSRF prefix Redfin prepends to all API responses
-    assert raw.startswith('{}&&'), f"Unexpected prefix: {raw[:10]}"
-    return json.loads(raw[4:])
-
-data = redfin_search(region_id=17151)  # 17151 = San Francisco, CA
-homes = data['payload']['homes']
-
-home = homes[0]
-print("Address:", home['streetLine']['value'])  # "875 California St #703"
-print("City/State/Zip:", home['city'], home['state'], home['zip'])
-print("Price:", home['price']['value'])          # 3300000
-print("Beds:", home['beds'])                     # 3
-print("Baths:", home['baths'])                   # 2.5
-print("Sqft:", home['sqFt']['value'])            # 1828
-print("$/sqft:", home['pricePerSqFt']['value'])  # 1805
-print("Lot size:", home['lotSize']['value'])      # 9448
-print("Year built:", home['yearBuilt']['value'])  # 2021
-print("Days on market:", home['dom']['value'])    # 1
-print("MLS ID:", home['mlsId']['value'])          # "426115342"
-print("MLS Status:", home['mlsStatus'])           # "Active"
-print("Lat/Long:", home['latLong']['value'])
-print("URL:", home['url'])                        # "/CA/San-Francisco/..."
-print("Remarks:", home['listingRemarks'][:100])
-```
-
-### Redfin region IDs
-
-| City | region_id | region_type |
-|---|---|---|
-| San Francisco, CA | `17151` | `6` (city) |
-| Los Angeles, CA | `17152` | `6` |
-| New York, NY | `17834` | `6` |
-| Seattle, WA | `16163` | `6` |
-
-To find other region IDs: search on Redfin, look at the URL (e.g. `/city/17151/CA/San-Francisco`) — the number is the region_id.
-
-### Redfin stingray response structure
-
-```
-data['payload']['homes'][i]
-  .streetLine.value      → street address string
-  .city / .state / .zip  → strings
-  .price.value           → int (asking price in dollars)
-  .sqFt.value            → int (square feet)
-  .pricePerSqFt.value    → int
-  .beds                  → int
-  .baths                 → float (2.5 = 2 full + 1 half)
-  .fullBaths / .partialBaths → ints
-  .lotSize.value         → int (sq ft)
-  .yearBuilt.value       → int
-  .dom.value             → days on market (int)
-  .mlsId.value           → MLS listing number (string)
-  .mlsStatus             → "Active", "Pending", etc.
-  .listingId             → Redfin internal int
-  .propertyId            → Redfin internal int
-  .latLong.value         → {'latitude': float, 'longitude': float}
-  .url                   → relative URL "/CA/San-Francisco/..."
-  .listingRemarks        → description text (may be truncated)
-  .keyFacts              → [{'description': str, 'rank': int}]
-  .listingTags           → ['SWEEPING CITY VIEWS', ...]
-  .hoa.value             → HOA monthly (int)
-  .location.value        → neighborhood name string
-  .sashes                → [{'sashTypeName': 'New'/'Price Drop'/...}]
-  .photos.value          → photo token string
-  .numPictures           → int
-```
-
----
-
-## Alternative APIs (no scraping required)
-
-If you need property data without scraping Zillow or Redfin at scale:
-
-| API | Free tier | Key data |
-|---|---|---|
-| **ATTOM Data** (attomdata.com) | Trial available | Ownership, AVM, tax, sale history, building characteristics |
-| **Rentcast** (rentcastapi.com) | 50 req/mo free | Rental estimates, comps, market data |
-| **RapidAPI: Zillow56** | ~100 req/mo free | Wraps Zillow data (unofficial, use at own risk) |
-| **HouseCanary** | Paid | AVM, market risk, rental value |
-| **Redfin API** (unofficial, above) | Unlimited | MLS listing data |
-| **US Census / HUD** | Free, no key | Median home values by geography, affordability |
-
----
-
-## Gotchas
-
-- **Single User-Agent word triggers 403.** `http_get` passes `"Mozilla/5.0"` as default User-Agent — this gets blocked. Always pass the full Chrome UA via the `headers=` argument.
-
-- **`price` field is `None` for sold and rental multi-unit listings.** Use `unformattedPrice` for for-sale, `hdpData.homeInfo.priceForHDP` for sold, and `minBaseRent`/`maxBaseRent` for rentals.
-
-- **`/homedetails/` is unconditionally blocked.** Tested with full browser headers, Referer, Sec-Fetch-* headers — all return HTTP 403. Only the browser bypasses this.
-
-- **41 listings per page, hardcoded.** Zillow always returns exactly 41 results per page from `listResults`. `mapResults` was empty in all tests (server-side response only).
-
-- **`isBot: False` doesn't mean you're safe.** Zillow correctly identifies server-side requests and blocks `/homedetails/`. The `isBot` flag in `__NEXT_DATA__` is `False` for search pages but the restriction is enforced at route level for detail pages.
-
-- **Captcha strings in HTML do not mean CAPTCHA is active.** The search page includes the captcha widget JavaScript (for lazy loading if needed) but does not serve a challenge — confirmed by successfully parsing listing data from the same HTML.
-
-- **Redfin `{}&&` prefix on all API responses.** Strip with `raw[4:]` before `json.loads()`. If the prefix changes, the assertion fails explicitly.
-
-- **Redfin JSON-LD omits price.** The `SingleFamilyResidence` schema objects do not include an `offers` field — use the stingray API for pricing.
-
-- **Redfin stingray API returns all listing fields wrapped in `{'value': X, 'level': N}` dicts.** Always read `.value` for numeric fields (e.g. `home['price']['value']`, not `home['price']`). Level `1` means data is public; `2` means potentially restricted.
-
-- **Zillow total count can exceed 800 but pagination caps at page ~20.** Zillow caps search results at around 800 listings even if `totalResultCount` shows 1037. Narrow by ZIP code, neighborhood, or price range to stay within bounds.
-
-- **URL filter syntax for Zillow:** Beds: `3-_beds` prefix; price: `0-1800000_price` suffix; ZIP: use `{zip}_rb` instead of city slug. Test by building the URL in a browser and copying the pattern.
+# Zillow — Scraping & Data Extraction
+
+Field-tested against `www.zillow.com` on 2026-04-18 using `http_get` (no browser).
+
+## Quick summary
+
+- **Search listing pages (`/homes/`, `/sold/`, `/rentals/`)** — `http_get` works with full Chrome headers. Returns ~973 KB HTML with all listing data embedded in `__NEXT_DATA__` JSON.
+- **Individual property detail pages (`/homedetails/`)** — `http_get` returns **HTTP 403** unconditionally. No header combination bypasses this.
+- **Internal API endpoints** (`/async-create-search-page-state`, `/graphql/`) — **403** for all server-side requests regardless of headers.
+- **Redfin** — `http_get` works; HTML contains both JSON-LD per listing and a stingray JSON API.
+
+---
+
+## What works: search listing pages via `__NEXT_DATA__`
+
+Zillow search pages embed all listing data in `<script id="__NEXT_DATA__">`. This is standard Next.js SSR output — it is the same data Zillow's React app hydrates from.
+
+**Required headers** — The single-word User-Agent (`"Mozilla/5.0"`) used by `http_get` internally gets 403. You must pass a full Chrome UA plus Accept/Accept-Language headers:
+
+```python
+import re, json
+from helpers import http_get
+
+HEADERS = {
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.9",
+}
+
+def extract_listings(html):
+    """Parse Zillow __NEXT_DATA__ and return list of listing dicts."""
+    m = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
+    if not m:
+        return []
+    d = json.loads(m.group(1))
+    sps = d['props']['pageProps']['searchPageState']
+    return sps['cat1']['searchResults']['listResults']
+
+html = http_get("https://www.zillow.com/homes/San-Francisco,-CA_rb/", headers=HEADERS)
+listings = extract_listings(html)
+print(len(listings))  # 41 — always 41 per page
+```
+
+### Fields available in each listing card
+
+The `listResults` array is the canonical source. Each entry includes:
+
+| Field | Source | Example |
+|---|---|---|
+| `zpid` | listing | `15081707` |
+| `address` | listing | `"212 Spruce St, San Francisco, CA 94118"` |
+| `addressStreet`, `addressCity`, `addressState`, `addressZipcode` | listing | split address components |
+| `price` | listing | `"$4,395,000"` (formatted string) |
+| `unformattedPrice` | listing | `4395000` (int, use for math) |
+| `beds` | listing | `4` |
+| `baths` | listing | `4` |
+| `area` | listing | `4133` (sqft) |
+| `latLong` | listing | `{'latitude': 37.78867, 'longitude': -122.45361}` |
+| `statusType` | listing | `"FOR_SALE"` / `"FOR_RENT"` / `"RECENTLY_SOLD"` |
+| `detailUrl` | listing | full `https://www.zillow.com/homedetails/...` URL |
+| `zestimate` | listing | `4857200` (Zillow AI estimate, int) |
+| `imgSrc` | listing | thumbnail URL |
+| `has3DModel` | listing | `True`/`False` |
+| `hasOpenHouse` | listing | `True`/`False` |
+| `openHouseStartDate`, `openHouseEndDate` | listing | ISO strings |
+| `isFeaturedListing` | listing | sponsored/featured flag |
+| `brokerName` | listing | `"Sotheby's International Realty"` |
+| `statusText` | listing | `"FOR SALE"` display string |
+| `hdpData.homeInfo.price` | nested | raw price int (matches `unformattedPrice`) |
+| `hdpData.homeInfo.zestimate` | nested | raw zestimate int |
+| `hdpData.homeInfo.rentZestimate` | nested | monthly rent estimate |
+| `hdpData.homeInfo.homeType` | nested | `"SINGLE_FAMILY"`, `"CONDO"`, `"TOWNHOUSE"` etc. |
+| `hdpData.homeInfo.daysOnZillow` | nested | int |
+| `hdpData.homeInfo.taxAssessedValue` | nested | int |
+| `hdpData.homeInfo.lotAreaValue` + `lotAreaUnit` | nested | e.g. `2957.724`, `"sqft"` |
+| `hdpData.homeInfo.priceForHDP` | nested | reliable sold price for recently-sold listings |
+
+```python
+# Full extraction snippet
+listing = listings[0]
+hi = listing.get('hdpData', {}).get('homeInfo', {})
+
+record = {
+    "zpid":         listing['zpid'],
+    "address":      listing['address'],
+    "price_raw":    listing.get('unformattedPrice') or hi.get('price'),
+    "beds":         listing.get('beds'),
+    "baths":        listing.get('baths'),
+    "sqft":         listing.get('area'),
+    "lat":          listing['latLong']['latitude'],
+    "lon":          listing['latLong']['longitude'],
+    "status":       listing['statusType'],
+    "zestimate":    listing.get('zestimate'),
+    "rent_zest":    hi.get('rentZestimate'),
+    "home_type":    hi.get('homeType'),
+    "days_listed":  hi.get('daysOnZillow'),
+    "tax_assessed": hi.get('taxAssessedValue'),
+    "url":          listing['detailUrl'],
+}
+```
+
+### Total result count and pagination
+
+```python
+d = json.loads(re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL).group(1))
+sps = d['props']['pageProps']['searchPageState']
+
+# Total listings in this search
+total = sps['categoryTotals']['cat1']['totalResultCount']
+print(total)  # 1037
+
+# Each page returns exactly 41 listings. Add /<N>_p/ for subsequent pages:
+# Page 2: https://www.zillow.com/homes/San-Francisco,-CA_rb/2_p/
+# Page 3: https://www.zillow.com/homes/San-Francisco,-CA_rb/3_p/
+
+max_pages = (total + 40) // 41
+```
+
+### Scrape all pages
+
+```python
+import re, json, time
+from helpers import http_get
+
+HEADERS = {
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.9",
+}
+
+def get_listings(city_slug, page=1):
+    """city_slug: e.g. 'San-Francisco,-CA', 'Seattle,-WA', 'Austin,-TX'"""
+    if page == 1:
+        url = f"https://www.zillow.com/homes/{city_slug}_rb/"
+    else:
+        url = f"https://www.zillow.com/homes/{city_slug}_rb/{page}_p/"
+    html = http_get(url, headers=HEADERS)
+    m = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
+    d = json.loads(m.group(1))
+    sps = d['props']['pageProps']['searchPageState']
+    total = sps['categoryTotals']['cat1']['totalResultCount']
+    listings = sps['cat1']['searchResults']['listResults']
+    return listings, total
+
+all_listings = []
+listings, total = get_listings("San-Francisco,-CA")
+all_listings.extend(listings)
+
+max_pages = (total + 40) // 41
+for page in range(2, min(max_pages + 1, 6)):   # cap at 5 pages for demo
+    time.sleep(1.0)   # polite delay
+    page_listings, _ = get_listings("San-Francisco,-CA", page)
+    all_listings.extend(page_listings)
+
+print(f"Fetched {len(all_listings)} of {total} listings")
+```
+
+---
+
+## URL patterns that work (all confirmed)
+
+| URL pattern | Status | Notes |
+|---|---|---|
+| `/homes/{city}_rb/` | **Works** | For-sale listings |
+| `/homes/{city}_rb/{N}_p/` | **Works** | Pagination |
+| `/homes/for_sale/{city}/0-1800000_price/` | **Works** | Price filter (max) |
+| `/homes/3-_beds/{city}/` | **Works** | Bed count filter |
+| `/homes/{zip}_rb/` | **Works** | ZIP code search |
+| `/san-francisco-ca/rentals/` | **Works** | Rental listings |
+| `/san-francisco-ca/sold/` | **Works** | Recently sold |
+| `/homedetails/{address}/{zpid}_zpid/` | **403** | Single property detail |
+| `/async-create-search-page-state` | **403** | Internal search API |
+| `/graphql/` | **400/403** | GraphQL endpoint |
+
+---
+
+## Rental listings
+
+Rental search pages use the same `__NEXT_DATA__` structure. However, rental listing cards have a **different schema** — individual units are nested, not a flat price:
+
+```python
+html = http_get("https://www.zillow.com/san-francisco-ca/rentals/", headers=HEADERS)
+listings = extract_listings(html)
+
+r = listings[0]
+# Multi-unit buildings:
+# r['units'] = [{'price': '$3,485+', 'beds': '0', 'roomForRent': False}, ...]
+# r['minBaseRent'] = 3485
+# r['maxBaseRent'] = 7130
+# r['availabilityCount'] = 23
+
+# Single-unit rentals:
+# r['price'] = '$2,500/mo'
+# r['unformattedPrice'] = 2500
+
+# Check which type:
+if r.get('isBuilding'):
+    price_range = f"${r['minBaseRent']}–${r['maxBaseRent']}/mo"
+    units = r.get('units', [])
+else:
+    price = r.get('unformattedPrice') or r.get('hdpData', {}).get('homeInfo', {}).get('price')
+```
+
+---
+
+## Sold listings
+
+Sold pages (`/sold/`) work identically. Key difference: `statusType` is `"RECENTLY_SOLD"` and price comes from `hdpData.homeInfo.priceForHDP` (not the `price` field which is `None` in sold cards):
+
+```python
+html = http_get("https://www.zillow.com/san-francisco-ca/sold/", headers=HEADERS)
+listings = extract_listings(html)
+
+for l in listings:
+    hi = l.get('hdpData', {}).get('homeInfo', {})
+    sold_price  = hi.get('priceForHDP')      # actual sold price
+    zestimate   = hi.get('zestimate')
+    tax_value   = hi.get('taxAssessedValue')
+    print(l['address'], f"${sold_price:,}", f"zest=${zestimate}")
+# 999 Green St APT 1702, San Francisco, CA 94133 $3,200,000 zest=$3,403,400
+# 1041 Vallejo St, San Francisco, CA 94133 $6,250,000 zest=None
+```
+
+Total sold inventory in San Francisco: **18,109** (all time in Zillow's database, paginated 41/page).
+
+---
+
+## Bot detection behavior
+
+- **Zillow detects bot status server-side** and embeds `window.__USER_SESSION_INITIAL_STATE__` and `props.isBot` in the page.
+- In field testing, the page returned `isBot: False` with the Chrome User-Agent — **Zillow does not block the search pages**.
+- The page does embed `captcha` strings in the HTML (for the CAPTCHA challenge widget code), but the challenge is NOT triggered for search pages.
+- **`/homedetails/` pages do trigger blocking** — every property detail URL tested returned HTTP 403. This is enforced before serving HTML, not via JavaScript CAPTCHA.
+- Rate limiting: 3 rapid sequential requests to `/homes/` all succeeded. Observed no 429s. Add `time.sleep(0.5–1.0)` between pages as a courtesy.
+
+---
+
+## What you do NOT get from `http_get`
+
+Because property detail pages are blocked (403), you lose:
+
+- Full property description text
+- All listing photos (you only get `imgSrc` thumbnail from search)
+- Detailed home facts (year built, parking, HVAC, school scores)
+- Price history
+- Nearby comparable sales (comps)
+- Agent contact info
+
+**To get these**, you must navigate to the `/homedetails/` URL in a browser session. The browser is not blocked (Zillow relies on JS challenges and fingerprinting that only trigger in browser context).
+
+---
+
+## Alternative: Redfin (field-tested, more accessible)
+
+Redfin allows `http_get` with no blocking for both HTML pages and its internal API.
+
+### Redfin JSON-LD per listing (easiest)
+
+Each Redfin search results page embeds one `<script type="application/ld+json">` per listing with structured property data:
+
+```python
+import re, json
+from helpers import http_get
+
+HEADERS = {
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.9",
+}
+
+html = http_get(
+    "https://www.redfin.com/city/17151/CA/San-Francisco/filter/property-type=house",
+    headers=HEADERS
+)
+print(len(html))  # ~1.6 MB
+
+# Extract all SingleFamilyResidence JSON-LD entries
+properties = []
+for s in re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL):
+    try:
+        d = json.loads(s)
+        if isinstance(d, list):
+            for item in d:
+                if item.get('@type') in ('SingleFamilyResidence', 'House', 'Residence', 'Apartment'):
+                    properties.append(item)
+    except Exception:
+        pass
+
+prop = properties[0]
+print("Name:", prop['name'])               # "662 Hampshire St, San Francisco, CA 94110"
+print("Address:", prop['address'])
+# {'@type': 'PostalAddress', 'streetAddress': '662 Hampshire St',
+#  'addressLocality': 'San Francisco', 'addressRegion': 'CA',
+#  'postalCode': '94110', 'addressCountry': 'US'}
+print("Rooms:", prop['numberOfRooms'])     # 3
+print("Floor size:", prop['floorSize'])    # {'@type': 'QuantitativeValue', 'value': 3350, 'unitCode': 'FTK'}
+print("URL:", prop['url'])
+# https://www.redfin.com/CA/San-Francisco/662-Hampshire-St-94110/home/1533754
+```
+
+Note: The JSON-LD schema does NOT include price (Redfin omits `offers` from the LD+JSON). Use the stingray API below for price.
+
+### Redfin stingray API (structured JSON with price)
+
+Redfin's internal GIS/search API returns rich structured data including price, MLS ID, beds, baths, sqft, agent info, and remarks. Responses are prefixed with `{}&&` — strip it before parsing:
+
+```python
+import json
+from helpers import http_get
+
+def redfin_search(region_id, region_type=6, num_homes=20, page=1, uipt="1,2,3,4,5,6"):
+    """
+    region_type: 6=city, 2=zipcode, 5=county
+    uipt: property types (1=house, 2=condo, 3=townhouse, 4=multi-family, 5=land, 6=other)
+    """
+    url = (
+        f"https://www.redfin.com/stingray/api/gis"
+        f"?al=1&num_homes={num_homes}&ord=redfin-recommended-asc"
+        f"&page_number={page}&region_id={region_id}&region_type={region_type}"
+        f"&sf=1,2,3,5,6,7&status=9&uipt={uipt}&v=8"
+    )
+    raw = http_get(url, headers={
+        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+        "Referer": "https://www.redfin.com/",
+        "Accept": "*/*",
+    })
+    # Strip the {}&& CSRF prefix Redfin prepends to all API responses
+    assert raw.startswith('{}&&'), f"Unexpected prefix: {raw[:10]}"
+    return json.loads(raw[4:])
+
+data = redfin_search(region_id=17151)  # 17151 = San Francisco, CA
+homes = data['payload']['homes']
+
+home = homes[0]
+print("Address:", home['streetLine']['value'])  # "875 California St #703"
+print("City/State/Zip:", home['city'], home['state'], home['zip'])
+print("Price:", home['price']['value'])          # 3300000
+print("Beds:", home['beds'])                     # 3
+print("Baths:", home['baths'])                   # 2.5
+print("Sqft:", home['sqFt']['value'])            # 1828
+print("$/sqft:", home['pricePerSqFt']['value'])  # 1805
+print("Lot size:", home['lotSize']['value'])      # 9448
+print("Year built:", home['yearBuilt']['value'])  # 2021
+print("Days on market:", home['dom']['value'])    # 1
+print("MLS ID:", home['mlsId']['value'])          # "426115342"
+print("MLS Status:", home['mlsStatus'])           # "Active"
+print("Lat/Long:", home['latLong']['value'])
+print("URL:", home['url'])                        # "/CA/San-Francisco/..."
+print("Remarks:", home['listingRemarks'][:100])
+```
+
+### Redfin region IDs
+
+| City | region_id | region_type |
+|---|---|---|
+| San Francisco, CA | `17151` | `6` (city) |
+| Los Angeles, CA | `17152` | `6` |
+| New York, NY | `17834` | `6` |
+| Seattle, WA | `16163` | `6` |
+
+To find other region IDs: search on Redfin, look at the URL (e.g. `/city/17151/CA/San-Francisco`) — the number is the region_id.
+
+### Redfin stingray response structure
+
+```
+data['payload']['homes'][i]
+  .streetLine.value      → street address string
+  .city / .state / .zip  → strings
+  .price.value           → int (asking price in dollars)
+  .sqFt.value            → int (square feet)
+  .pricePerSqFt.value    → int
+  .beds                  → int
+  .baths                 → float (2.5 = 2 full + 1 half)
+  .fullBaths / .partialBaths → ints
+  .lotSize.value         → int (sq ft)
+  .yearBuilt.value       → int
+  .dom.value             → days on market (int)
+  .mlsId.value           → MLS listing number (string)
+  .mlsStatus             → "Active", "Pending", etc.
+  .listingId             → Redfin internal int
+  .propertyId            → Redfin internal int
+  .latLong.value         → {'latitude': float, 'longitude': float}
+  .url                   → relative URL "/CA/San-Francisco/..."
+  .listingRemarks        → description text (may be truncated)
+  .keyFacts              → [{'description': str, 'rank': int}]
+  .listingTags           → ['SWEEPING CITY VIEWS', ...]
+  .hoa.value             → HOA monthly (int)
+  .location.value        → neighborhood name string
+  .sashes                → [{'sashTypeName': 'New'/'Price Drop'/...}]
+  .photos.value          → photo token string
+  .numPictures           → int
+```
+
+---
+
+## Alternative APIs (no scraping required)
+
+If you need property data without scraping Zillow or Redfin at scale:
+
+| API | Free tier | Key data |
+|---|---|---|
+| **ATTOM Data** (attomdata.com) | Trial available | Ownership, AVM, tax, sale history, building characteristics |
+| **Rentcast** (rentcastapi.com) | 50 req/mo free | Rental estimates, comps, market data |
+| **RapidAPI: Zillow56** | ~100 req/mo free | Wraps Zillow data (unofficial, use at own risk) |
+| **HouseCanary** | Paid | AVM, market risk, rental value |
+| **Redfin API** (unofficial, above) | Unlimited | MLS listing data |
+| **US Census / HUD** | Free, no key | Median home values by geography, affordability |
+
+---
+
+## Gotchas
+
+- **Single User-Agent word triggers 403.** `http_get` passes `"Mozilla/5.0"` as default User-Agent — this gets blocked. Always pass the full Chrome UA via the `headers=` argument.
+
+- **`price` field is `None` for sold and rental multi-unit listings.** Use `unformattedPrice` for for-sale, `hdpData.homeInfo.priceForHDP` for sold, and `minBaseRent`/`maxBaseRent` for rentals.
+
+- **`/homedetails/` is unconditionally blocked.** Tested with full browser headers, Referer, Sec-Fetch-* headers — all return HTTP 403. Only the browser bypasses this.
+
+- **41 listings per page, hardcoded.** Zillow always returns exactly 41 results per page from `listResults`. `mapResults` was empty in all tests (server-side response only).
+
+- **`isBot: False` doesn't mean you're safe.** Zillow correctly identifies server-side requests and blocks `/homedetails/`. The `isBot` flag in `__NEXT_DATA__` is `False` for search pages but the restriction is enforced at route level for detail pages.
+
+- **Captcha strings in HTML do not mean CAPTCHA is active.** The search page includes the captcha widget JavaScript (for lazy loading if needed) but does not serve a challenge — confirmed by successfully parsing listing data from the same HTML.
+
+- **Redfin `{}&&` prefix on all API responses.** Strip with `raw[4:]` before `json.loads()`. If the prefix changes, the assertion fails explicitly.
+
+- **Redfin JSON-LD omits price.** The `SingleFamilyResidence` schema objects do not include an `offers` field — use the stingray API for pricing.
+
+- **Redfin stingray API returns all listing fields wrapped in `{'value': X, 'level': N}` dicts.** Always read `.value` for numeric fields (e.g. `home['price']['value']`, not `home['price']`). Level `1` means data is public; `2` means potentially restricted.
+
+- **Zillow total count can exceed 800 but pagination caps at page ~20.** Zillow caps search results at around 800 listings even if `totalResultCount` shows 1037. Narrow by ZIP code, neighborhood, or price range to stay within bounds.
+
+- **URL filter syntax for Zillow:** Beds: `3-_beds` prefix; price: `0-1800000_price` suffix; ZIP: use `{zip}_rb` instead of city slug. Test by building the URL in a browser and copying the pattern.