catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/nasa/scraping.md

@@ -0,0 +1,339 @@
+# NASA APIs — Scraping & Data Extraction
+
+`https://api.nasa.gov` — open NASA data APIs. **Never use the browser.** All endpoints return JSON via `http_get`. DEMO_KEY works for low-volume use; register for a free personal key at https://api.nasa.gov/ to raise limits.
+
+## Do this first
+
+**All `api.nasa.gov` endpoints share the same rate-limit pool under DEMO_KEY. EPIC and Exoplanet Archive are on separate domains with no rate limit.**
+
+```python
+import json
+from helpers import http_get
+
+# Simplest call: today's Astronomy Picture of the Day
+apod = json.loads(http_get("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY"))
+print(apod['date'], apod['title'], apod['media_type'])
+# Confirmed output (2026-04-18): 2026-04-18 PanSTARRS and Planets image
+```
+
+Use DEMO_KEY for exploration. Switch to a personal key for any bulk work — DEMO_KEY hits its limit at ~10 req/hour/IP (daily budget around 50; `retry-after` header will show ~22 hours when exhausted).
+
+## Rate limits
+
+| Key type | Limit | Resets |
+|---|---|---|
+| `DEMO_KEY` | 10 req/hour, ~50/day per IP | Hourly window; daily hard stop with `retry-after` ~22h |
+| Personal key (free) | 1,000 req/hour | Hourly window |
+
+Rate limit headers on every `api.nasa.gov` response:
+- `X-Ratelimit-Limit` — your current window limit (e.g. `10`)
+- `X-Ratelimit-Remaining` — calls left this window
+- `Retry-After` — seconds until next window (only on 429)
+
+**EPIC (`epic.gsfc.nasa.gov`) and Exoplanet Archive (`exoplanetarchive.ipac.caltech.edu`) share no rate-limit pool with `api.nasa.gov`.**
+
+## Common workflows
+
+### APOD — single day
+
+```python
+import json
+from helpers import http_get
+
+apod = json.loads(http_get("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY"))
+print(apod['date'])        # '2026-04-18'
+print(apod['title'])       # 'PanSTARRS and Planets'
+print(apod['media_type'])  # 'image' or 'video'
+print(apod['url'])         # full-res or YouTube embed URL
+print(apod['hdurl'])       # HD image URL (absent when media_type='video')
+print(apod.get('copyright'))  # None if public domain
+# Confirmed output (2026-04-18):
+# url:   https://apod.nasa.gov/apod/image/2604/PanstarrsPlanetsPerrotLab1024.jpg
+# hdurl: https://apod.nasa.gov/apod/image/2604/PanstarrsPlanetsPerrot.jpg
+# copyright: Luc Perrot
+```
+
+### APOD — date range (array response)
+
+```python
+import json
+from helpers import http_get
+
+apods = json.loads(http_get(
+    "https://api.nasa.gov/planetary/apod"
+    "?start_date=2024-01-01&end_date=2024-01-07&api_key=DEMO_KEY"
+))
+# Returns a list of 7 dicts — same schema as single-day response
+for a in apods:
+    print(a['date'], a['media_type'], a['title'][:50])
+# Confirmed output (7 items):
+# 2024-01-01 image NGC 1232: A Grand Design Spiral Galaxy
+# 2024-01-02 image Rocket Transits Rippling Moon
+# 2024-01-03 image A SAR Arc from New Zealand
+# 2024-01-04 image Zeta Oph: Runaway Star
+# 2024-01-05 image Trapezium: At the Heart of Orion
+# 2024-01-06 video The Snows of Churyumov-Gerasimenko
+# 2024-01-07 image The Cat's Eye Nebula in Optical and X-ray
+```
+
+Optional params: `date=YYYY-MM-DD` (specific day), `count=N` (N random entries), `thumbs=true` (include `thumbnail_url` for video entries).
+
+### APOD — random sample
+
+```python
+import json
+from helpers import http_get
+
+apods = json.loads(http_get(
+    "https://api.nasa.gov/planetary/apod?count=5&api_key=DEMO_KEY"
+))
+for a in apods:
+    print(a['date'], a['title'][:40])
+# Returns 5 random APOD entries — dates can be any day since 1995-06-16
+```
+
+### NEO — Near Earth Objects feed
+
+```python
+import json
+from helpers import http_get
+
+data = json.loads(http_get(
+    "https://api.nasa.gov/neo/rest/v1/feed"
+    "?start_date=2024-01-01&end_date=2024-01-02&api_key=DEMO_KEY"
+))
+print(data['element_count'])   # 32 (total NEOs across both days)
+neos = data['near_earth_objects']  # dict keyed by date string
+for date, objects in sorted(neos.items()):
+    for neo in objects:
+        ca = neo['close_approach_data'][0]
+        print(
+            neo['name'],
+            'hazardous:', neo['is_potentially_hazardous_asteroid'],
+            'miss km:', ca['miss_distance']['kilometers'][:12],
+            'vel kph:', ca['relative_velocity']['kilometers_per_hour'][:10]
+        )
+# Confirmed output (2 days, 32 total NEOs):
+# 415949 (2001 XY10) hazardous: False miss km: 50452409.34 vel kph: 57205.8951
+# (22+ more objects per day)
+```
+
+NEO object fields:
+- `id`, `name`, `nasa_jpl_url` — identity
+- `estimated_diameter` — dict with `kilometers`, `meters`, `miles`, `feet` sub-dicts, each with `min`/`max`
+- `is_potentially_hazardous_asteroid` — bool
+- `close_approach_data[0]` — `close_approach_date`, `miss_distance` (au/lunar/km/mi), `relative_velocity` (km/s, km/h, mph), `orbiting_body`
+
+Date range is capped at **7 days per request**. For longer ranges, paginate with `start_date` / `end_date` in 7-day steps. `links.next` in the response gives the next 7-day window URL.
+
+### NEO — single asteroid lookup
+
+```python
+import json
+from helpers import http_get
+
+# Asteroid ID comes from the feed's `id` field
+neo = json.loads(http_get(
+    "https://api.nasa.gov/neo/rest/v1/neo/2415949?api_key=DEMO_KEY"
+))
+print(neo['name'])
+print(neo['orbital_data']['orbit_class']['orbit_class_description'])
+# Full orbital history + all close approaches are in `close_approach_data` (long list)
+```
+
+### Mars Rover photos — Curiosity by sol
+
+```python
+import json
+from helpers import http_get
+
+# sol = Martian solar day since landing
+data = json.loads(http_get(
+    "https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos"
+    "?sol=1000&api_key=DEMO_KEY"
+))
+photos = data['photos']
+print(f"Photos on sol 1000: {len(photos)}")
+p = photos[0]
+print(p['earth_date'])          # '2015-05-30'
+print(p['img_src'])             # direct JPEG URL
+print(p['camera']['name'])      # 'FHAZ'
+print(p['camera']['full_name']) # 'Front Hazard Avoidance Camera'
+print(p['rover']['name'])       # 'Curiosity'
+print(p['rover']['status'])     # 'active'
+print(p['rover']['max_sol'])    # highest sol with photos
+
+# Filter by camera
+data = json.loads(http_get(
+    "https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos"
+    "?sol=1000&camera=navcam&api_key=DEMO_KEY"
+))
+```
+
+Available cameras for Curiosity: `fhaz`, `rhaz`, `mast`, `chemcam`, `mahli`, `mardi`, `navcam`. Other rovers: `opportunity`, `spirit`, `perseverance`.
+
+Use `latest_photos` to get the most recent available:
+```python
+data = json.loads(http_get(
+    "https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/latest_photos"
+    "?api_key=DEMO_KEY"
+))
+photos = data['latest_photos']
+```
+
+Add `&page=N` for pagination (25 photos/page by default).
+
+### EPIC — Earth Polychromatic Imaging Camera
+
+EPIC images are served from `epic.gsfc.nasa.gov` — **no `api_key` required, no rate limit.**
+
+```python
+import json
+from helpers import http_get
+
+# Latest available images (natural color)
+images = json.loads(http_get("https://epic.gsfc.nasa.gov/api/natural"))
+print(f"Latest batch: {len(images)} images")  # Confirmed: 4 images on 2026-04-18
+
+img = images[0]
+print(img['identifier'])               # '20260416162050'
+print(img['image'])                    # 'epic_1b_20260416162050'
+print(img['date'])                     # '2026-04-16 16:16:01'
+print(img['centroid_coordinates'])     # {'lat': 13.25, 'lon': -75.59}
+
+# Construct PNG URL from image name + date
+date_str = img['date'].split(' ')[0]   # '2026-04-16'
+year, month, day = date_str.split('-')
+png_url = f"https://epic.gsfc.nasa.gov/archive/natural/{year}/{month}/{day}/png/{img['image']}.png"
+jpg_thumb = f"https://epic.gsfc.nasa.gov/archive/natural/{year}/{month}/{day}/thumbs/{img['image']}.jpg"
+print(png_url)
+# Confirmed: https://epic.gsfc.nasa.gov/archive/natural/2026/04/16/png/epic_1b_20260416162050.png
+```
+
+```python
+# Images for a specific date
+images = json.loads(http_get("https://epic.gsfc.nasa.gov/api/natural/date/2024-01-15"))
+print(len(images))   # 14 images on 2024-01-15
+
+# Enhanced (color-corrected) images — same API, different path
+enhanced = json.loads(http_get("https://epic.gsfc.nasa.gov/api/enhanced/date/2024-01-15"))
+# Enhanced image URL pattern uses 'enhanced' and 'epic_RGB_' prefix:
+img = enhanced[0]
+date_str = img['date'].split(' ')[0]
+year, month, day = date_str.split('-')
+url = f"https://epic.gsfc.nasa.gov/archive/enhanced/{year}/{month}/{day}/png/{img['image']}.png"
+# e.g. .../archive/enhanced/2024/01/15/png/epic_RGB_20240115005515.png
+
+# All available dates
+all_dates = json.loads(http_get("https://epic.gsfc.nasa.gov/api/natural/all"))
+print(f"Available dates: {len(all_dates)}")  # 3477 dates (2015-06-13 to present)
+print(all_dates[0])   # {'date': '2026-04-16'}  (newest first)
+print(all_dates[-1])  # {'date': '2015-06-13'}  (oldest)
+```
+
+### Exoplanet Archive — TAP/ADQL queries
+
+No API key or rate limit. SQL-like ADQL queries over the full archive.
+
+```python
+import json
+from helpers import http_get
+
+# Short-period planets with known radii
+planets = json.loads(http_get(
+    "https://exoplanetarchive.ipac.caltech.edu/TAP/sync"
+    "?query=select+pl_name,hostname,pl_orbper+from+ps+where+pl_orbper+%3C+10"
+    "&format=json"
+))
+print(f"Rows: {len(planets)}")      # 17675 (table 'ps' includes duplicate measurements)
+print(planets[0])
+# {'pl_name': 'GJ 1214 b', 'hostname': 'GJ 1214', 'pl_orbper': 1.58040482}
+```
+
+```python
+# Use 'pscomppars' for one row per planet (composite best-estimate params)
+planets = json.loads(http_get(
+    "https://exoplanetarchive.ipac.caltech.edu/TAP/sync"
+    "?query=select+pl_name,hostname,disc_year,discoverymethod,pl_orbper,pl_rade,pl_masse,pl_eqt,sy_dist"
+    "+from+pscomppars+where+disc_year+%3E+2020+and+pl_rade+is+not+null"
+    "+order+by+disc_year+desc"
+    "&format=json&maxrec=5"
+))
+for p in planets:
+    print(p['pl_name'], p['disc_year'], p['discoverymethod'], f"r={p['pl_rade']}Re")
+# Confirmed output:
+# KMT-2024-BLG-1870L b 2026 Microlensing r=13.8Re
+# LHS 1903 b 2026 Transit r=1.382Re
+# TOI-375 d 2026 Radial Velocity r=13.6Re
+```
+
+Key tables:
+- `ps` — all measurements per planet (multiple rows per planet, all sources)
+- `pscomppars` — one row per confirmed planet (best composite parameters)
+
+Key columns: `pl_name`, `hostname`, `disc_year`, `discoverymethod`, `pl_orbper` (orbital period, days), `pl_rade` (radius in Earth radii), `pl_masse` (mass in Earth masses), `pl_eqt` (equilibrium temp K), `sy_dist` (distance in parsec).
+
+URL-encode operators: `<` = `%3C`, `>` = `%3E`, spaces = `+`.
+
+## URL reference
+
+### api.nasa.gov endpoints
+
+| Endpoint | URL pattern |
+|---|---|
+| APOD today | `https://api.nasa.gov/planetary/apod?api_key=KEY` |
+| APOD by date | `...&date=YYYY-MM-DD` |
+| APOD range | `...&start_date=YYYY-MM-DD&end_date=YYYY-MM-DD` |
+| APOD random N | `...&count=N` |
+| NEO feed | `https://api.nasa.gov/neo/rest/v1/feed?start_date=...&end_date=...&api_key=KEY` |
+| NEO by ID | `https://api.nasa.gov/neo/rest/v1/neo/{id}?api_key=KEY` |
+| Mars photos by sol | `https://api.nasa.gov/mars-photos/api/v1/rovers/{rover}/photos?sol=N&api_key=KEY` |
+| Mars photos by date | `...?earth_date=YYYY-MM-DD&api_key=KEY` |
+| Mars latest | `https://api.nasa.gov/mars-photos/api/v1/rovers/{rover}/latest_photos?api_key=KEY` |
+
+### EPIC (epic.gsfc.nasa.gov — no key, no rate limit)
+
+| Endpoint | URL |
+|---|---|
+| Latest natural images | `https://epic.gsfc.nasa.gov/api/natural` |
+| Natural by date | `https://epic.gsfc.nasa.gov/api/natural/date/YYYY-MM-DD` |
+| Enhanced latest | `https://epic.gsfc.nasa.gov/api/enhanced` |
+| Enhanced by date | `https://epic.gsfc.nasa.gov/api/enhanced/date/YYYY-MM-DD` |
+| All available dates | `https://epic.gsfc.nasa.gov/api/natural/all` |
+| PNG image | `https://epic.gsfc.nasa.gov/archive/natural/YYYY/MM/DD/png/{image}.png` |
+| Thumbnail (JPEG) | `https://epic.gsfc.nasa.gov/archive/natural/YYYY/MM/DD/thumbs/{image}.jpg` |
+| Enhanced PNG | `https://epic.gsfc.nasa.gov/archive/enhanced/YYYY/MM/DD/png/{image}.png` |
+
+### Exoplanet Archive (no key, no rate limit)
+
+```
+https://exoplanetarchive.ipac.caltech.edu/TAP/sync?query=<ADQL>&format=json&maxrec=<N>
+```
+
+## Gotchas
+
+- **DEMO_KEY limit is effectively 10/hour per IP, not 30** — The `X-Ratelimit-Limit` header shows `10` in practice. When the daily budget (~50 req) is exhausted, the `retry-after` header is set to ~80,000 seconds (about 22 hours). Register a free personal key at https://api.nasa.gov/ to get 1,000/hour.
+
+- **All `api.nasa.gov` paths share one rate-limit pool** — APOD, NEO, Mars Rover, and all other `api.nasa.gov` paths draw from the same DEMO_KEY bucket. Calling any one of them depletes the limit for all others.
+
+- **EPIC and Exoplanet Archive are fully free** — `epic.gsfc.nasa.gov` returns no rate-limit headers and is not throttled. `exoplanetarchive.ipac.caltech.edu/TAP/sync` is similarly unrestricted. Use these freely without fear of exhausting DEMO_KEY.
+
+- **NEO date range max is 7 days** — Requests spanning more than 7 days return HTTP 400. Paginate with 7-day windows and use `links.next` from the response to get the next URL.
+
+- **APOD earliest date is 1995-06-16** — Requesting `date` before `1995-06-16` returns HTTP 400 with an error message. No upper date bound other than today.
+
+- **APOD `hdurl` is absent for video entries** — When `media_type` is `video`, the response has `url` (a YouTube embed URL) but no `hdurl`. Always check `media_type` before accessing `hdurl`.
+
+- **Mars Rover `sol` vs `earth_date`** — Both are valid filter params. `sol` is the Martian solar day since rover landing. `earth_date` uses `YYYY-MM-DD`. You cannot mix them in one request.
+
+- **Mars Rover pagination defaults to 25 photos/page** — Large sols (Curiosity sol 1000 has many photos) require `&page=2`, `&page=3`, etc. There is no total count in the response; keep paginating until you get an empty `photos` list.
+
+- **EPIC image name encodes type in the prefix** — Natural images use `epic_1b_` prefix; enhanced color-corrected images use `epic_RGB_` prefix. The API returns the correct filename in `img['image']`; don't guess the prefix.
+
+- **EPIC `/api/natural/all` returns newest-first** — The list of 3,477+ available dates starts from today and goes back to 2015-06-13. Not all days have images (gaps during spacecraft maintenance).
+
+- **Exoplanet `ps` table has multiple rows per planet** — Different publications report different measurements for the same planet. Use `pscomppars` for one-row-per-planet composite parameters. `ps` is useful when you need all reported values or want to filter by specific reference.
+
+- **Exoplanet null values come back as `None` in JSON** — Many fields like `pl_masse` are `null` for planets without mass measurements. Always guard with `if row['pl_masse'] is not None`.
+
+- **`http_get` in helpers.py uses stdlib `urllib`** — On some macOS Python 3.11 installs, SSL certificate verification fails (`CERTIFICATE_VERIFY_FAILED`). If you hit this, run `curl` via `subprocess` as a fallback, or install certifi and patch the default SSL context. The harness's browser CDP connection is not affected; only `http_get` is.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/news-aggregation/multi-source.md

@@ -0,0 +1,205 @@
+# News Aggregation — Multi-Source
+
+Field-tested against TechCrunch, The Verge, Ars Technica, BBC, Guardian, Wired, NPR, HN, Reuters, CNN, NYT (2026-04-18).
+
+## Lead with RSS — fastest and most reliable
+
+For every site that has a feed, `http_get` + XML parsing is faster and more reliable than a browser. Use `ThreadPoolExecutor` for parallel fetches.
+
+**Confirmed working RSS feeds (tested):**
+
+| Source | Feed URL | Format | Items | Fetch time |
+|--------|----------|--------|-------|------------|
+| TechCrunch | `https://techcrunch.com/feed/` | RSS 2.0 | 20 | ~0.08s |
+| Ars Technica | `https://feeds.arstechnica.com/arstechnica/index` | RSS 2.0 | 20 | ~0.10s |
+| BBC News | `http://feeds.bbci.co.uk/news/rss.xml` | RSS 2.0 | 37 | ~0.23s |
+| The Guardian (World) | `https://www.theguardian.com/world/rss` | RSS 2.0 | 45 | ~0.11s |
+| The Guardian (Tech) | `https://www.theguardian.com/technology/rss` | RSS 2.0 | 32 | ~0.25s |
+| Wired | `https://www.wired.com/feed/rss` | RSS 2.0 | 50 | ~0.10s |
+| NPR Top Stories | `https://feeds.npr.org/1001/rss.xml` | RSS 2.0 | 10 | ~0.14s |
+| Hacker News | `https://news.ycombinator.com/rss` | RSS 2.0 | 30 | ~0.16s |
+| CNN Top Stories | `http://rss.cnn.com/rss/cnn_topstories.rss` | RSS 2.0 | 69 | ~0.25s |
+| NYT Homepage | `https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml` | RSS 2.0 | 23 | ~0.12s |
+| The Verge | `https://www.theverge.com/rss/index.xml` | **Atom** | 10 | ~0.15s |
+
+## Parallel fetch pattern (4.3x speedup measured)
+
+Sequential fetch of 7 feeds: **0.70s**. Parallel fetch of same 7 feeds: **0.16s** (4.3x speedup).
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+import xml.etree.ElementTree as ET
+
+RSS_FEEDS = [
+    ("TechCrunch",     "https://techcrunch.com/feed/"),
+    ("Ars Technica",   "https://feeds.arstechnica.com/arstechnica/index"),
+    ("BBC News",       "http://feeds.bbci.co.uk/news/rss.xml"),
+    ("Guardian World", "https://www.theguardian.com/world/rss"),
+    ("Wired",          "https://www.wired.com/feed/rss"),
+    ("NPR",            "https://feeds.npr.org/1001/rss.xml"),
+    ("Wired",          "https://www.wired.com/feed/rss"),
+]
+
+def fetch_rss(name_url):
+    name, url = name_url
+    xml_data = http_get(url)
+    root = ET.fromstring(xml_data)
+    items = root.findall('.//item')
+    return name, items
+
+with ThreadPoolExecutor(max_workers=len(RSS_FEEDS)) as ex:
+    results = list(ex.map(fetch_rss, RSS_FEEDS))
+
+for name, items in results:
+    for item in items[:5]:
+        title = item.find('title').text
+        link  = item.find('link').text
+        print(f"[{name}] {title}")
+```
+
+## The Verge requires Atom namespace parsing
+
+The Verge's feed is Atom format, not RSS 2.0. The naive `.//item` selector returns 0 items. The `title` element uses `type="html"` attribute but its `.text` still contains the plain string.
+
+```python
+import xml.etree.ElementTree as ET
+
+NS = {'atom': 'http://www.w3.org/2005/Atom'}
+
+xml_data = http_get("https://www.theverge.com/rss/index.xml")
+root = ET.fromstring(xml_data)
+entries = root.findall('.//atom:entry', NS)   # 10 entries
+
+for e in entries:
+    title = e.find('atom:title', NS).text
+    link  = e.find('atom:link', NS).get('href')
+    print(title, link)
+```
+
+Do NOT use `root.findall('.//{http://www.w3.org/2005/Atom}entry')` with a bare namespace — the explicit `NS` dict approach above is cleaner. Do NOT call `.text` on a `find()` result without checking for `None` first (the naive RSS path hit this on The Verge).
+
+## Sites that block http_get entirely
+
+**Reuters** returns HTTP 403/Forbidden for all `http_get` calls, even with a real browser `User-Agent` header. Use browser fallback (see below).
+
+```
+Reuters: ERROR HTTP Error 401: HTTP Forbidden   # with AND without User-Agent
+```
+
+Reuters's old RSS feeds (`feeds.reuters.com/reuters/topNews`) resolve to DNS NXDOMAIN — they have been shut down.
+
+## Sites that work fine with http_get + User-Agent
+
+NYT, Guardian, HN, CNN all return full HTML via `http_get` without issues. The User-Agent header (`Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36`) is not required for these but doesn't hurt.
+
+```python
+headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"}
+html = http_get("https://www.nytimes.com", headers=headers)  # 1.1MB, works
+html = http_get("https://news.ycombinator.com")               # 34KB, works without UA
+```
+
+**HN parsing via regex (no HTML parser needed):**
+```python
+import re
+html = http_get("https://news.ycombinator.com")
+stories = re.findall(r'class="titleline"><a href="([^"]+)"[^>]*>([^<]+)<', html)
+# Returns list of (url, title) tuples — 30 stories on the front page
+```
+
+## Browser extraction — use when RSS is unavailable
+
+### BBC (`bbc.com/news`)
+
+No consent banner in headless browser (US region served; GDPR banner only appears for EU IP). Articles use `article h2` selectors.
+
+```python
+goto_url("https://www.bbc.com/news")
+wait_for_load()
+wait(2)
+
+headlines = js("""
+  Array.from(document.querySelectorAll('article h2'))
+    .map(h => ({
+      title: h.innerText.trim(),
+      url: h.closest('a')?.href || h.closest('[href]')?.href || 
+           h.parentElement.querySelector('a')?.href
+    }))
+    .filter(h => h.title.length > 10)
+""")
+# Returns 50+ articles. First one is typically LIVE/breaking.
+```
+
+If running from a EU IP and a consent banner appears:
+```python
+accept = js("""
+  var btns = Array.from(document.querySelectorAll('button'));
+  var btn = btns.find(b => /accept all|agree|continue/i.test(b.innerText));
+  if (btn) { btn.click(); return 'clicked: ' + btn.innerText; }
+  return 'no banner';
+""")
+```
+
+Confirmed: `h3` elements on BBC are site-chrome labels ("The BBC is in multiple languages"), NOT article headlines. Use `article h2` only.
+
+### TechCrunch (`techcrunch.com`)
+
+`article` and `.post-block` selectors return 0 results — TechCrunch changed their layout. Articles are in `h3` elements.
+
+```python
+goto_url("https://techcrunch.com")
+wait_for_load()
+wait(2)
+
+articles = js("""
+  Array.from(document.querySelectorAll('h3'))
+    .map(h => ({
+      title: h.innerText?.trim(),
+      url: h.closest('a')?.href || h.querySelector('a')?.href || 
+           h.parentElement?.querySelector('a')?.href
+    }))
+    .filter(a => a.title && a.title.length > 20)
+""")
+# Returns ~10 articles. RSS is preferred (20 items, no JS required).
+```
+
+RSS is almost always faster for TechCrunch: **0.08s vs 3-5s browser** load. Only fall back to browser if you need paywall/subscriber content.
+
+### Reuters (`reuters.com`)
+
+`http_get` returns 403. Browser loads but the homepage is heavily JS-rendered with delayed hydration. `h3` selectors only return nav elements after standard `wait_for_load()`. Use `wait(3)` plus scroll:
+
+```python
+goto_url("https://www.reuters.com")
+wait_for_load()
+wait(3)
+js("window.scrollTo(0, 500)")
+wait(1)
+# Category links work for topic nav:
+links = js("""
+  Array.from(document.querySelectorAll('a[href*="/world/"], a[href*="/technology/"]'))
+    .filter(a => a.innerText.trim().length > 20)
+    .map(a => ({text: a.innerText.trim(), href: a.href}))
+""")
+```
+
+Reuters headlines are best obtained from the Guardian or AP — Reuters no longer has a public RSS and their JS hydration is slow.
+
+## Decision tree: which approach to use
+
+```
+Does the site have an RSS/Atom feed?
+  YES → http_get + XML parse (fastest, ~0.1s per feed)
+         - RSS 2.0: root.findall('.//item')
+         - Atom:    root.findall('.//atom:entry', {'atom': 'http://www.w3.org/2005/Atom'})
+  NO  → Does http_get return valid HTML (not 403/401/JS shell)?
+          YES → http_get + regex/BeautifulSoup (fast, ~0.2-0.3s)
+          NO  → goto + wait_for_load + wait(2) + js() extraction (slow, 3-8s)
+```
+
+## What to skip
+
+- **Reuters RSS** — DNS dead (`feeds.reuters.com` is NXDOMAIN)
+- **Reuters http_get** — returns 403 regardless of User-Agent
+- **TechCrunch `article`/`.post-block` selectors** — layout changed, use `h3` instead
+- **BBC `h3` for headlines** — those are site-chrome labels; use `article h2`
+- **The Verge `.//item` selector** — feed is Atom, not RSS; use Atom namespace