catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/openstreetmap/scraping.md

@@ -0,0 +1,490 @@
+# OpenStreetMap — Nominatim Geocoding + Overpass API
+
+Two fully public, no-auth APIs. Everything is a direct HTTP call — never need a browser.
+
+- **Nominatim**: geocoding (place name → lat/lon and reverse). Rate limit: 1 req/s.
+- **Overpass API**: spatial query engine over the full OSM dataset. Rate limit: 2 concurrent slots per IP on the public instance.
+
+**Do not use `http_get` without overriding `User-Agent`** — its default `Mozilla/5.0` is blocked by both APIs with HTTP 403. Pass `headers={"User-Agent": "browser-harness/1.0"}` on every call.
+
+---
+
+## Fastest path: forward geocode a place
+
+```python
+import json, urllib.parse
+from helpers import http_get
+
+UA = {"User-Agent": "browser-harness/1.0"}
+
+def geocode(query: str, limit: int = 3) -> list[dict]:
+    q = urllib.parse.quote(query)
+    raw = http_get(
+        f"https://nominatim.openstreetmap.org/search?q={q}&format=json&limit={limit}&addressdetails=1",
+        headers=UA
+    )
+    return json.loads(raw)  # [] when nothing found
+
+results = geocode("Eiffel Tower")
+# results[0]['display_name']  == 'Tour Eiffel, 5, Avenue Anatole France, ..., 75007, France'
+# results[0]['lat']            == '48.8582599'   ← STRING, not float
+# results[0]['lon']            == '2.2945006'    ← STRING, not float
+# results[0]['type']           == 'tower'
+# results[0]['class']          == 'man_made'
+# results[0]['importance']     == 0.6205937724353116
+# results[0]['osm_type']       == 'way'
+# results[0]['osm_id']         == 5013364
+# results[0]['boundingbox']    == ['48.8574753', '48.8590453', '2.2933119', '2.2956897']  ← all strings
+# results[0]['address']['city']     == 'Paris'
+# results[0]['address']['postcode'] == '75007'
+# results[0]['address']['country']  == 'France'
+# results[0]['address']['country_code'] == 'fr'
+```
+
+---
+
+## Nominatim: all three query modes
+
+### 1. Forward geocode (free-text)
+
+```python
+import json, urllib.parse
+from helpers import http_get
+
+UA = {"User-Agent": "browser-harness/1.0"}
+
+raw = http_get(
+    "https://nominatim.openstreetmap.org/search?q=Eiffel+Tower&format=json&limit=3&addressdetails=1",
+    headers=UA
+)
+results = json.loads(raw)
+# Returns [] when nothing found — no exception
+
+# Useful optional params:
+# &addressdetails=1   → adds 'address' dict to each result (city, postcode, road, etc.)
+# &extratags=1        → adds 'extratags' dict (website, wikidata, phone, etc.)
+# &namedetails=1      → adds 'namedetails' dict (name:en, name:fr, etc.)
+# &countrycodes=fr,de → restrict to countries (comma-separated ISO 3166-1 alpha-2)
+# &viewbox=2.2,48.8,2.4,48.9 &bounded=1  → restrict to bounding box (lon_min,lat_min,lon_max,lat_max)
+```
+
+### 2. Reverse geocode (lat/lon → address)
+
+```python
+raw = http_get(
+    "https://nominatim.openstreetmap.org/reverse?lat=48.8584&lon=2.2945&format=json",
+    headers=UA
+)
+result = json.loads(raw)
+# result['display_name']  == 'Avenue Gustave Eiffel, Quartier du Gros-Caillou, ..., France'
+# result['address']['road']         == 'Avenue Gustave Eiffel'
+# result['address']['city']         == 'Paris'
+# result['address']['postcode']     == '75007'
+# result['address']['country']      == 'France'
+# result['address']['country_code'] == 'fr'
+# result['address']['state']        == 'Île-de-France'
+# result['lat'], result['lon']  → strings (not floats)
+
+# Optional: &zoom=N (0-18) controls granularity of the returned address
+# zoom=3 → country, zoom=10 → city, zoom=18 → street/building (default)
+```
+
+### 3. Structured search (field-based)
+
+```python
+raw = http_get(
+    "https://nominatim.openstreetmap.org/search?city=Paris&country=France&format=json&limit=1",
+    headers=UA
+)
+result = json.loads(raw)[0]
+# result['name']          == 'Paris'
+# result['lat']           == '48.8534951'
+# result['lon']           == '2.3483915'
+# result['type']          == 'administrative'
+# result['place_rank']    == 12  (lower = broader: 4=country, 8=state, 12=city, 30=POI)
+# result['addresstype']   == 'city'
+# result['boundingbox']   == ['48.8155755', '48.9021560', '2.2241220', '2.4697602']
+
+# Supported structured params: street, city, county, state, country, postalcode
+```
+
+### 4. Lookup by OSM ID
+
+```python
+# Prefix: N=node, W=way, R=relation
+raw = http_get(
+    "https://nominatim.openstreetmap.org/lookup?osm_ids=W5013364&format=json",
+    headers=UA
+)
+result = json.loads(raw)
+# Returns list. Eiffel Tower way: result[0]['name'] == 'Tour Eiffel'
+# Supports up to 50 IDs: osm_ids=W5013364,N123456,R789
+```
+
+---
+
+## Nominatim response field reference
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `place_id` | int | Internal Nominatim ID — do not cache long-term, can change |
+| `osm_type` | str | `"node"`, `"way"`, or `"relation"` |
+| `osm_id` | int | The OSM element ID |
+| `lat` | **str** | Latitude as string — convert with `float(r['lat'])` |
+| `lon` | **str** | Longitude as string — convert with `float(r['lon'])` |
+| `display_name` | str | Full human-readable address string |
+| `name` | str | Short name of the place |
+| `type` | str | OSM type tag value: `"tower"`, `"administrative"`, `"restaurant"`, etc. |
+| `class` | str | OSM key: `"man_made"`, `"boundary"`, `"amenity"`, `"highway"`, etc. |
+| `addresstype` | str | Semantic category: `"city"`, `"road"`, `"man_made"`, etc. |
+| `place_rank` | int | Hierarchy rank: 4=country, 8=state, 12=city, 16=suburb, 30=POI |
+| `importance` | float | 0–1 relevance score (higher = more notable) |
+| `boundingbox` | list[str] | `[south_lat, north_lat, west_lon, east_lon]` — all strings, note unusual order |
+| `licence` | str | ODbL attribution string — include in user-facing output |
+| `address` | dict | Only present with `&addressdetails=1` or in reverse results |
+
+`address` dict common keys: `road`, `house_number`, `quarter`, `suburb`, `city_district`, `city`, `state`, `postcode`, `country`, `country_code`, `ISO3166-2-lvl4/6`.
+
+---
+
+## Overpass API: query OSM data by tags
+
+Overpass is a read-only query engine over the full OSM planet. It supports finding POIs by tag, radius, bounding box, and combinations.
+
+**Endpoint**: `https://overpass-api.de/api/interpreter`
+**Backup instances** (use when main is overloaded, which happens often):
+- `https://overpass.openstreetmap.fr/api/interpreter` — requires non-Mozilla User-Agent
+
+**http_get works for GET requests** — pass `headers={"User-Agent": "browser-harness/1.0"}`. For POST, use `urllib` directly (see example below).
+
+### GET query (simplest for http_get)
+
+```python
+import json, urllib.parse
+from helpers import http_get
+
+UA = {"User-Agent": "browser-harness/1.0"}
+OVERPASS = "https://overpass.openstreetmap.fr/api/interpreter"
+
+def overpass_get(query: str) -> dict:
+    url = f"{OVERPASS}?data={urllib.parse.quote(query)}"
+    raw = http_get(url, headers=UA)
+    if not raw.startswith("{"):
+        raise RuntimeError(f"Overpass error (HTML returned): {raw[:200]}")
+    return json.loads(raw)
+
+# Find cafes in central Paris (bbox: south_lat, west_lon, north_lat, east_lon)
+r = overpass_get('[out:json][timeout:25];node["amenity"="cafe"](48.855,2.295,48.862,2.308);out 10;')
+# r['version']    == 0.6
+# r['generator']  == 'Overpass API 0.7.62.7 375dc00a'
+# r['elements']   → list of matching OSM elements
+
+for cafe in r['elements']:
+    print(cafe['tags'].get('name'), cafe['lat'], cafe['lon'])
+# 'Café de l\'Alma' 48.8609068 2.3015143
+# 'Le Campanella'   48.8585847 2.3032822
+# 'Kozy Bosquet'    48.855445  2.3054013
+
+# Find restaurants within 500m radius of a point (around filter)
+r = overpass_get(
+    '[out:json][timeout:25];node["amenity"="restaurant"](around:500,37.7749,-122.4194);out 10;'
+)
+for rest in r['elements']:
+    print(rest['tags'].get('name'), rest['tags'].get('cuisine',''))
+# 'Nepalese Indian Cusine' 'indian;nepali'
+# 'Local Diner' 'coffee_shop;italian;burger;seafood'
+# 'Moya Cafe' ''
+```
+
+### POST query (for complex QL, avoids URL length limits)
+
+```python
+import json, urllib.parse, urllib.request, gzip
+from helpers import http_get  # http_get is GET-only; use urllib for POST
+
+OVERPASS = "https://overpass.openstreetmap.fr/api/interpreter"
+
+def overpass_post(query: str) -> dict:
+    """POST to Overpass — no URL length limits, preferred for multi-statement QL."""
+    data = urllib.parse.urlencode({"data": query}).encode()
+    req = urllib.request.Request(
+        OVERPASS, data=data, method="POST",
+        headers={
+            "User-Agent": "browser-harness/1.0",
+            "Content-Type": "application/x-www-form-urlencoded",
+            "Accept-Encoding": "gzip",
+        }
+    )
+    with urllib.request.urlopen(req, timeout=30) as r:
+        body = r.read()
+        if r.headers.get("Content-Encoding") == "gzip":
+            body = gzip.decompress(body)
+    body = body.decode()
+    if not body.startswith("{"):
+        raise RuntimeError(f"Overpass error (HTML): {body[:300]}")
+    return json.loads(body)
+
+# Example: cafes in Paris bbox
+r = overpass_post('[out:json][timeout:25];node["amenity"="cafe"](48.855,2.295,48.862,2.308);out 5;')
+print(len(r['elements']))  # 5 (or up to 5)
+```
+
+### Overpass element structure
+
+Every element in `r['elements']` is a dict with at minimum:
+
+```python
+{
+    "type": "node",          # "node", "way", or "relation"
+    "id": 308684349,         # int — OSM element ID (stable, use for dedup)
+    "lat": 48.8609068,       # float — ONLY present for node type
+    "lon": 2.3015143,        # float — ONLY present for node type
+    "tags": {                # dict — all OSM tags on this element
+        "amenity": "cafe",
+        "name": "Café de l'Alma",
+        "name:fr": "Café de l'Alma",
+        "outdoor_seating": "yes",
+        "payment:credit_cards": "yes",
+        "phone": "+33 1 45 51 56 74",
+        "opening_hours": "Mo-Sa 08:00-23:00; Su 09:00-19:00",  # optional
+        "website": "https://...",                               # optional
+        "wheelchair": "yes"                                     # optional
+    }
+}
+```
+
+For `way` elements, use `out center;` to get a `center` dict with lat/lon instead of a node list:
+
+```python
+# way element with out center:
+{
+    "type": "way",
+    "id": 338411946,
+    "center": {"lat": 48.8660087, "lon": 2.3153233},  # centroid of the polygon
+    "nodes": [3454913623, 3454913707, ...],            # node IDs forming the boundary
+    "tags": {"amenity": "cafe", "name": "Café 1902", ...}
+}
+
+# Query to get both nodes and ways with lat/lon:
+query = '[out:json][timeout:25];(node["amenity"="cafe"](48.85,2.29,48.87,2.32);way["amenity"="cafe"](48.85,2.29,48.87,2.32););out center 20;'
+r = overpass_get(query)
+for el in r['elements']:
+    if el['type'] == 'node':
+        lat, lon = el['lat'], el['lon']
+    else:  # way
+        lat, lon = el['center']['lat'], el['center']['lon']
+    print(el['tags'].get('name'), lat, lon)
+```
+
+### Overpass QL quick reference
+
+```
+[out:json][timeout:25]      # Required header: JSON output, 25s timeout
+[maxsize:52428800]          # Optional: 50MB max result size (default is server limit)
+
+node["amenity"="cafe"](south,west,north,east);out N;
+#  ↑ bbox order: south_lat, west_lon, north_lat, east_lon
+#  Note: DIFFERENT from Nominatim's boundingbox field which is [south,north,west,east]
+
+node["amenity"="cafe"](around:RADIUS_METERS,LAT,LON);out N;
+
+node["amenity"~"cafe|restaurant"](bbox);out N;    # regex match on tag value
+node[!"name"](bbox);out N;                        # elements WITHOUT the 'name' tag
+node["name"~"Star",i](bbox);out N;               # case-insensitive regex
+
+# Union of types:
+(node["amenity"="cafe"](bbox); way["amenity"="cafe"](bbox););out center N;
+
+# Multiple tags (AND logic):
+node["amenity"="cafe"]["outdoor_seating"="yes"](bbox);out N;
+```
+
+---
+
+## OSM tile server (reference only, no scraping)
+
+```
+https://{a,b,c}.tile.openstreetmap.org/{z}/{x}/{y}.png
+```
+
+- Subdomains `a`, `b`, `c` for load balancing
+- `z` = zoom level 0–19, `x`/`y` = tile coordinates
+- Returns 256×256 PNG tiles
+- Policy: max 2 req/s per IP, non-commercial use, must display OSM attribution
+- Tile coordinate calculator: `https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames`
+- Bulk tile downloading is prohibited — use Overpass or data extracts instead
+
+```python
+# Convert lat/lon to tile coordinates
+import math
+
+def lat_lon_to_tile(lat, lon, zoom):
+    n = 2 ** zoom
+    x = int((lon + 180) / 360 * n)
+    y = int((1 - math.log(math.tan(math.radians(lat)) + 1 / math.cos(math.radians(lat))) / math.pi) / 2 * n)
+    return x, y
+
+x, y = lat_lon_to_tile(48.8582, 2.2945, 14)
+url = f"https://a.tile.openstreetmap.org/14/{x}/{y}.png"
+# url == 'https://a.tile.openstreetmap.org/14/8281/5646.png'
+```
+
+---
+
+## Rate limits
+
+| API | Limit | Enforcement | 429 behavior |
+|-----|-------|-------------|--------------|
+| Nominatim | 1 req/s | Soft — rapid requests work but you get delayed/dropped | Returns HTTP 403 if your IP is banned (not 429) |
+| Overpass (main) | 2 concurrent slots per IP | Hard — 3rd concurrent req returns HTML error immediately | HTML error page with `rate_limited` in body |
+| Overpass (main) | Also: query complexity quota | Resets over time (~per hour) | HTML error page with `rate_limited` |
+| Tile server | 2 req/s per IP | Soft/hard | IP block |
+
+**Check your Overpass quota**:
+```python
+raw = http_get("https://overpass-api.de/api/status", headers={"User-Agent": "browser-harness/1.0"})
+print(raw)
+# Connected as: 1728118854
+# Rate limit: 2
+# 2 slots available now.
+# Slot available after: 2026-04-18T11:00:00Z, in 30 seconds.
+```
+
+**Handle rate limiting in production**:
+```python
+import time
+
+def overpass_get_with_retry(query: str, max_retries: int = 3) -> dict:
+    for attempt in range(max_retries):
+        url = f"https://overpass.openstreetmap.fr/api/interpreter?data={urllib.parse.quote(query)}"
+        raw = http_get(url, headers={"User-Agent": "browser-harness/1.0"})
+        if raw.startswith("{"):
+            return json.loads(raw)
+        if "rate_limited" in raw or "too busy" in raw:
+            wait = 2 ** attempt * 10  # 10s, 20s, 40s
+            time.sleep(wait)
+            continue
+        raise RuntimeError(f"Overpass error: {raw[:200]}")
+    raise RuntimeError("Overpass: too many retries")
+```
+
+---
+
+## Complete working example
+
+```python
+import json, time, urllib.parse, urllib.request, gzip
+from helpers import http_get
+
+UA = {"User-Agent": "browser-harness/1.0"}
+NOMINATIM = "https://nominatim.openstreetmap.org"
+OVERPASS   = "https://overpass.openstreetmap.fr/api/interpreter"
+
+def geocode(query: str, limit: int = 1) -> list[dict]:
+    """Forward geocode — returns [] if nothing found."""
+    q = urllib.parse.quote(query)
+    raw = http_get(f"{NOMINATIM}/search?q={q}&format=json&limit={limit}&addressdetails=1", headers=UA)
+    return json.loads(raw)
+
+def reverse_geocode(lat: float, lon: float) -> dict:
+    """Reverse geocode — always returns a result (nearest road/place)."""
+    raw = http_get(f"{NOMINATIM}/reverse?lat={lat}&lon={lon}&format=json", headers=UA)
+    return json.loads(raw)
+
+def overpass_get(query: str) -> list[dict]:
+    """Run an Overpass QL query, return elements list."""
+    url = f"{OVERPASS}?data={urllib.parse.quote(query)}"
+    raw = http_get(url, headers=UA)
+    if not raw.startswith("{"):
+        raise RuntimeError(f"Overpass error: {raw[:200]}")
+    return json.loads(raw)["elements"]
+
+def overpass_post(query: str) -> list[dict]:
+    """POST variant — avoids URL length limits for complex queries."""
+    data = urllib.parse.urlencode({"data": query}).encode()
+    req = urllib.request.Request(
+        OVERPASS, data=data, method="POST",
+        headers={"User-Agent": "browser-harness/1.0",
+                 "Content-Type": "application/x-www-form-urlencoded",
+                 "Accept-Encoding": "gzip"}
+    )
+    with urllib.request.urlopen(req, timeout=30) as r:
+        body = r.read()
+        if r.headers.get("Content-Encoding") == "gzip":
+            body = gzip.decompress(body)
+    body = body.decode()
+    if not body.startswith("{"):
+        raise RuntimeError(f"Overpass error: {body[:300]}")
+    return json.loads(body)["elements"]
+
+# --- Usage examples (validated 2026-04-18) ---
+
+# 1. Geocode a landmark
+places = geocode("Eiffel Tower", limit=3)
+# places[0]['lat']  == '48.8582599'  (string)
+# places[0]['lon']  == '2.2945006'   (string)
+# places[0]['display_name'] == 'Tour Eiffel, 5, Avenue Anatole France, ..., 75007, France'
+# places[0]['address']['city'] == 'Paris'
+lat = float(places[0]['lat'])
+lon = float(places[0]['lon'])
+
+# 2. Reverse geocode the coordinates
+addr = reverse_geocode(lat, lon)
+# addr['address']['road']    == 'Avenue Gustave Eiffel'
+# addr['address']['city']    == 'Paris'
+# addr['address']['postcode']== '75007'
+# addr['address']['country'] == 'France'
+
+# 3. Find nearby cafes (wait 1s between nominatim and overpass if same script)
+time.sleep(1)
+cafes = overpass_get(
+    f"[out:json][timeout:25];node[\"amenity\"=\"cafe\"](around:500,{lat},{lon});out 10;"
+)
+for cafe in cafes:
+    print(f"{cafe['tags'].get('name','?'):30s}  {cafe['lat']:.4f}, {cafe['lon']:.4f}")
+# Café de l'Alma                  48.8609, 2.3015
+# Le Campanella                   48.8586, 2.3033
+
+# 4. Structured city lookup + find restaurants in bounding box
+time.sleep(1)
+paris = geocode("Paris, France")[0]
+bb = paris['boundingbox']  # [south_lat, north_lat, west_lon, east_lon] ← Nominatim order!
+# For Overpass: need (south_lat, west_lon, north_lat, east_lon) ← DIFFERENT order
+south, north, west, east = bb[0], bb[1], bb[2], bb[3]
+# Restrict to center slice to avoid massive result set
+center_bbox = f"48.855,2.295,48.865,2.315"
+rests = overpass_post(
+    f"[out:json][timeout:25];node[\"amenity\"=\"restaurant\"]({center_bbox});out 5;"
+)
+print(f"Found {len(rests)} restaurants near Paris center")
+```
+
+---
+
+## Gotchas
+
+**`http_get` default UA (`Mozilla/5.0`) is blocked by both APIs.** Always pass `headers={"User-Agent": "browser-harness/1.0"}`. The `headers` kwarg in `http_get` does a `.update()` so it properly overrides the default. Confirmed: Mozilla/5.0 → 403 on Nominatim; `browser-harness/1.0` → 200.
+
+**Blocked User-Agent patterns on Nominatim**: `Mozilla/5.0`, `python-requests/*`, `Wget/*`. Accepted: any non-generic app-style UA like `browser-harness/1.0`, `MyApp/2.0`, `curl/7.x`. Nominatim policy requires a descriptive UA with contact info, but in practice any non-library string passes.
+
+**Nominatim lat/lon are strings, Overpass lat/lon are floats.** Always convert Nominatim coordinates: `float(result['lat'])`. Overpass element `lat`/`lon` are native Python floats — no conversion needed.
+
+**Nominatim `boundingbox` field order is `[south_lat, north_lat, west_lon, east_lon]` — NOT `[south, west, north, east]`.** Overpass bbox uses `(south_lat, west_lon, north_lat, east_lon)`. When feeding a Nominatim bounding box into Overpass, you must reorder: `f"({bb[0]},{bb[2]},{bb[1]},{bb[3]})"`.
+
+**`overpass-api.de` main instance is frequently overloaded.** Returns HTTP 504 (timeout) or an HTML error page with `rate_limited` when busy. The FR mirror (`overpass.openstreetmap.fr`) is usually more responsive but also blocks `Mozilla/5.0`. Always detect non-JSON responses: `if not raw.startswith("{")`.
+
+**Overpass error responses are HTML, not JSON.** The API returns HTTP 200 with an HTML error page when rate-limited or when the server is too busy. Always check `raw.startswith("{")` before parsing.
+
+**Overpass rate limit: 2 concurrent slots, NOT 2 requests/s.** You can run 2 queries simultaneously. A 3rd concurrent query immediately returns an error. Sequential queries with no sleep between them work fine as long as each completes before the next starts.
+
+**`out N;` limits results to N elements — use it.** Without a limit, large bounding boxes can return thousands of elements and hit the 512MB memory limit, returning a `maxsize` error. Default safe limit: `out 50;` for exploration, `out 500;` for bulk collection.
+
+**Overpass QL bbox order is `(south, west, north, east)` — latitude FIRST.** This is the opposite of the standard GeoJSON convention `[west, south, east, north]`. The `around:` filter uses `(around:METERS,LAT,LON)` — note lat before lon.
+
+**`name` tag in Overpass is the local-language name.** For Paris cafes this is French. English names may appear under `name:en` but are often absent. Never assume `name` is in English.
+
+**Nominatim `/reverse` always returns the nearest result** — it never returns an empty response (unlike `/search`). If the coordinates are in the ocean, it still returns the nearest coastline or country.
+
+**`place_id` is internal and ephemeral** — do not store it for long-term use. Use `osm_type` + `osm_id` for stable references (e.g., `way/5013364` for the Eiffel Tower).
+
+**Overpass `http_get` POST workaround**: `http_get` only supports GET. For POST requests (needed to avoid URL length limits for complex multi-statement QL), use `urllib.request.Request` directly as shown in the `overpass_post()` example above.