catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/gutenberg/scraping.md

@@ -0,0 +1,383 @@
+# Project Gutenberg — Scraping & Data Extraction
+
+`https://www.gutenberg.org` — 78 000+ free public-domain ebooks. Every workflow here is pure `http_get` — no browser needed.
+
+## Do this first
+
+**Use the Gutendex REST API (`gutendex.com`) for all search and discovery. It is one call, returns clean JSON, and requires no auth. Go to gutenberg.org URLs only to fetch actual file content.**
+
+```python
+import json
+
+# Search by title/author keyword
+data = json.loads(http_get("https://gutendex.com/books/?search=pride+and+prejudice"))
+# data['count'] = 6 (total matches)
+# data['results'] = list of up to 32 book objects
+book = data['results'][0]
+# book['id'] = 1342  ← use this ID for all further calls
+# book['formats']['text/plain; charset=utf-8'] = direct txt URL
+
+# Fetch the plain-text content of that book
+text = http_get(book['formats']['text/plain; charset=utf-8'])
+# Returns 763 083 chars including Project Gutenberg header/footer boilerplate
+```
+
+For a known book ID, skip search entirely:
+
+```python
+book = json.loads(http_get("https://gutendex.com/books/1342/"))
+```
+
+## Common workflows
+
+### Search by keyword and get the first result
+
+```python
+import json
+
+data = json.loads(http_get("https://gutendex.com/books/?search=frankenstein"))
+if data['results']:
+    b = data['results'][0]
+    print(b['id'], b['title'], b['authors'][0]['name'])
+    # 84  Frankenstein; or, the modern prometheus  Shelley, Mary Wollstonecraft
+    txt_url = b['formats'].get('text/plain; charset=utf-8')
+    if txt_url:
+        text = http_get(txt_url)
+```
+
+### Get the most downloaded books (popularity ranking)
+
+```python
+import json
+
+data = json.loads(http_get("https://gutendex.com/books/?sort=popular"))
+for b in data['results'][:10]:
+    authors = ', '.join(a['name'] for a in b['authors'])
+    print(f"[{b['id']}] {b['title']} — {authors} ({b['download_count']:,} downloads)")
+# [84]    Frankenstein                      — Shelley, Mary Wollstonecraft  (178,271)
+# [45304] The City of God, Volume I         — Augustine, of Hippo, Saint    (147,663)
+# [2701]  Moby Dick; Or, The Whale          — Melville, Herman              (112,302)
+# [1342]  Pride and Prejudice               — Austen, Jane                  (107,502)
+# [768]   Wuthering Heights                 — Brontë, Emily                  (72,775)
+# [1513]  Romeo and Juliet                  — Shakespeare, William           (70,272)
+# [11]    Alice's Adventures in Wonderland  — Carroll, Lewis                 (65,243)
+# [64317] The Great Gatsby                  — Fitzgerald, F. Scott           (60,632)
+# [100]   Complete Works of Shakespeare     — Shakespeare, William           (60,527)
+# [1260]  Jane Eyre: An Autobiography       — Brontë, Charlotte              (57,602)
+```
+
+### Browse by genre / topic
+
+```python
+import json
+
+# 'topic' matches both subjects and bookshelves fields
+data = json.loads(http_get("https://gutendex.com/books/?topic=science+fiction"))
+# data['count'] = 3473 total results, 32 per page
+
+data = json.loads(http_get("https://gutendex.com/books/?topic=detective+fiction"))
+# data['count'] = 111
+# data['results'][0]: id=1661 The Adventures of Sherlock Holmes — Doyle, Arthur Conan
+
+# Filter by language (ISO 639-1 code)
+data = json.loads(http_get("https://gutendex.com/books/?languages=fr&topic=roman"))
+# data['count'] = 254 French books with 'roman' in topic
+```
+
+### Paginate through results
+
+```python
+import json
+
+url = "https://gutendex.com/books/?topic=science+fiction"
+books = []
+while url:
+    data = json.loads(http_get(url))
+    books.extend(data['results'])
+    url = data['next']   # None on last page
+    # data['previous'] is also populated after page 1
+    # e.g. data['next'] = "https://gutendex.com/books/?page=3&topic=science+fiction"
+# All 3473 sci-fi books loaded across ~109 pages of 32 each
+```
+
+### Fetch multiple specific books by ID
+
+```python
+import json
+
+data = json.loads(http_get("https://gutendex.com/books/?ids=1342,11,84"))
+# Returns exactly those 3 books, count=3
+for b in data['results']:
+    print(b['id'], b['title'])
+# 84    Frankenstein; or, the modern prometheus
+# 1342  Pride and Prejudice
+# 11    Alice's Adventures in Wonderland
+```
+
+### Read the plain text of a book (boilerplate stripped)
+
+```python
+raw = http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.txt")
+# 763 083 chars total including PG licence header and footer
+
+START = "*** START OF THE PROJECT GUTENBERG EBOOK"
+END   = "*** END OF THE PROJECT GUTENBERG EBOOK"
+s = raw.find(START)
+e = raw.find(END)
+if s != -1:
+    content = raw[raw.index('\n', s) + 1 : e].strip()
+    # 743 241 chars of actual novel text
+```
+
+The cache URL is the most reliable direct path. The `formats` dict in Gutendex also provides a redirect URL that resolves to the same file:
+
+```python
+# Both of these return identical content (763 083 chars):
+http_get("https://www.gutenberg.org/ebooks/1342.txt.utf-8")          # redirect
+http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.txt")     # direct cache
+```
+
+### Download formats available per book
+
+Every book's `formats` dict maps MIME type to URL. All URLs resolve to `/cache/epub/{id}/` files via redirect.
+
+| MIME type | URL pattern (after redirect) | Typical size |
+|---|---|---|
+| `text/plain; charset=utf-8` | `pg{id}.txt` | ~750 KB |
+| `text/html` | `pg{id}-images.html` | ~850 KB |
+| `application/epub+zip` | `pg{id}-images-3.epub` | ~25 MB |
+| `application/x-mobipocket-ebook` | `pg{id}-images-kf8.mobi` | ~25 MB |
+| `application/rdf+xml` | `{id}.rdf` via gutenberg.org | metadata XML |
+| `image/jpeg` | `pg{id}.cover.medium.jpg` | cover image |
+| `application/octet-stream` | `pg{id}-h.zip` | HTML+images zip |
+
+```python
+import json
+
+b = json.loads(http_get("https://gutendex.com/books/1342/"))
+# Grab every downloadable format URL:
+for mime, url in b['formats'].items():
+    print(mime, '->', url)
+# text/html                         -> https://www.gutenberg.org/ebooks/1342.html.images
+# application/epub+zip              -> https://www.gutenberg.org/ebooks/1342.epub3.images
+# application/x-mobipocket-ebook   -> https://www.gutenberg.org/ebooks/1342.kf8.images
+# application/rdf+xml               -> https://www.gutenberg.org/ebooks/1342.rdf
+# image/jpeg                        -> https://www.gutenberg.org/cache/epub/1342/pg1342.cover.medium.jpg
+# application/octet-stream          -> https://www.gutenberg.org/cache/epub/1342/pg1342-h.zip
+# text/plain; charset=utf-8         -> https://www.gutenberg.org/ebooks/1342.txt.utf-8
+```
+
+### Fetch RDF/XML metadata for a book
+
+```python
+import re
+
+rdf = http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.rdf")
+# Also available as: http_get("https://www.gutenberg.org/ebooks/1342.rdf")
+
+title     = re.search(r'<dcterms:title>(.*?)</dcterms:title>', rdf, re.DOTALL)
+creator   = re.findall(r'<pgterms:name>(.*?)</pgterms:name>', rdf)
+birth     = re.findall(r'<pgterms:birthdate[^>]*>(\d+)', rdf)
+death     = re.findall(r'<pgterms:deathdate[^>]*>(\d+)', rdf)
+issued    = re.search(r'<dcterms:issued[^>]*>(.*?)</dcterms:issued>', rdf)
+rights    = re.search(r'<dcterms:rights>(.*?)</dcterms:rights>', rdf)
+downloads = re.search(r'<pgterms:downloads[^>]*>(\d+)</pgterms:downloads>', rdf)
+language  = re.search(r'<dcterms:language>.*?<rdf:value>(.*?)</rdf:value>', rdf, re.DOTALL)
+subjects  = re.findall(r'<dcterms:subject>.*?<rdf:value>(.*?)</rdf:value>.*?</dcterms:subject>', rdf, re.DOTALL)
+
+print(title.group(1))          # Pride and Prejudice
+print(creator)                 # ['Austen, Jane']
+print(birth, death)            # ['1775'] ['1817']
+print(issued.group(1))         # 1998-06-01
+print(rights.group(1))         # Public domain in the USA.
+print(int(downloads.group(1))) # 107502
+print(subjects[:3])            # ['England -- Fiction', 'Young women -- Fiction', 'Love stories']
+```
+
+Note: `<dcterms:language>` value is a subject string, not a language code. For language codes use the Gutendex `languages` field instead.
+
+### Search the HTML catalog (25 results per page)
+
+Use this only when you need to leverage Gutenberg's own search index (author:, title:, subject: prefix syntax).
+
+```python
+import re, json
+
+html = http_get(
+    "https://www.gutenberg.org/ebooks/search/"
+    "?query=shakespeare&sort_order=downloads"
+)
+# sort_order options: downloads, title, release_date, last_update, random
+
+entries = re.findall(r'<li class="booklink">(.*?)</li>', html, re.DOTALL)
+books = []
+for e in entries:
+    book_id   = re.search(r'/ebooks/(\d+)', e)
+    title     = re.search(r'<span class="title">(.*?)</span>', e)
+    author    = re.search(r'<span class="subtitle">(.*?)</span>', e)
+    downloads = re.search(r'<span class="extra">([^<]+)</span>', e)
+    books.append({
+        'id':        int(book_id.group(1)) if book_id else None,
+        'title':     title.group(1) if title else '',
+        'author':    author.group(1) if author else '',
+        'downloads': downloads.group(1).strip() if downloads else '',
+    })
+
+# books[0] = {'id': 1513, 'title': 'Romeo and Juliet',
+#             'author': 'William Shakespeare', 'downloads': '74316 downloads'}
+
+# Paginate with start_index (25 per page)
+html_p2 = http_get(
+    "https://www.gutenberg.org/ebooks/search/"
+    "?query=shakespeare&sort_order=downloads&start_index=26"
+)
+```
+
+### Browse a bookshelf (curated genre list)
+
+```python
+import re
+
+# Bookshelf 68 = Science Fiction
+html = http_get("https://www.gutenberg.org/ebooks/bookshelf/68")
+titles = re.findall(r'<span class="title">(.*?)</span>', html)
+# ['Twenty Thousand Leagues under the Sea', 'The War of the Worlds',
+#  'The Time Machine', 'Thuvia, Maid of Mars', ...]
+```
+
+### OPDS catalog (machine-readable Atom feed)
+
+```python
+import re
+
+feed = http_get("https://www.gutenberg.org/ebooks/search.opds/?query=dracula")
+# Returns Atom XML, 7 entries per page (including 1 metadata entry)
+entries = re.findall(r'<entry>(.*?)</entry>', feed, re.DOTALL)
+for e in entries:
+    title = re.search(r'<title>(.*?)</title>', e)
+    entry_id = re.search(r'<id>(.*?)</id>', e)
+    if title and entry_id and 'opds' in entry_id.group(1):
+        book_id = re.search(r'/ebooks/(\d+)\.opds', entry_id.group(1))
+        print(book_id.group(1), title.group(1))
+# 345  Dracula
+```
+
+## Gutendex API — full response schema
+
+Validated against a real call to `GET https://gutendex.com/books/1342/`:
+
+```json
+{
+  "id": 1342,
+  "title": "Pride and Prejudice",
+  "authors": [
+    {"name": "Austen, Jane", "birth_year": 1775, "death_year": 1817}
+  ],
+  "summaries": ["...automatically generated summary..."],
+  "editors": [],
+  "translators": [],
+  "subjects": [
+    "Courtship -- Fiction",
+    "Domestic fiction",
+    "England -- Fiction",
+    "Love stories",
+    "Sisters -- Fiction",
+    "Women -- England -- Fiction",
+    "Young women -- Fiction"
+  ],
+  "bookshelves": [
+    "Best Books Ever Listings",
+    "Category: British Literature",
+    "Category: Classics of Literature",
+    "Category: Novels",
+    "Category: Romance",
+    "Harvard Classics"
+  ],
+  "languages": ["en"],
+  "copyright": false,
+  "media_type": "Text",
+  "formats": {
+    "text/html":                       "https://www.gutenberg.org/ebooks/1342.html.images",
+    "application/epub+zip":            "https://www.gutenberg.org/ebooks/1342.epub3.images",
+    "application/x-mobipocket-ebook": "https://www.gutenberg.org/ebooks/1342.kf8.images",
+    "application/rdf+xml":             "https://www.gutenberg.org/ebooks/1342.rdf",
+    "image/jpeg":                      "https://www.gutenberg.org/cache/epub/1342/pg1342.cover.medium.jpg",
+    "application/octet-stream":        "https://www.gutenberg.org/cache/epub/1342/pg1342-h.zip",
+    "text/plain; charset=utf-8":       "https://www.gutenberg.org/ebooks/1342.txt.utf-8"
+  },
+  "download_count": 107502
+}
+```
+
+List response wrapper (from `GET /books/`):
+
+```json
+{
+  "count": 6,
+  "next": null,
+  "previous": null,
+  "results": [...]
+}
+```
+
+`count` is the total across all pages. `next` / `previous` are fully-formed URLs ready to pass to `http_get`, or `null` when absent.
+
+## Gutendex query parameters
+
+All parameters combine freely.
+
+| Parameter | Example | Notes |
+|---|---|---|
+| `search` | `search=moby+dick` | Matches title and author |
+| `ids` | `ids=1342,11,84` | Comma-separated; returns only those books |
+| `languages` | `languages=fr` | ISO 639-1 code; comma-separated for multiple |
+| `topic` | `topic=science+fiction` | Matches subjects + bookshelves |
+| `author_year_start` | `author_year_start=1800` | Author born on/after year |
+| `author_year_end` | `author_year_end=1850` | Author born on/before year |
+| `copyright` | `copyright=false` | `false`=public domain, `true`=copyrighted |
+| `sort` | `sort=popular` | `popular` (default), `ascending`, `descending` |
+| `page` | `page=2` | 1-based; 32 results per page (not configurable) |
+
+`page_size` is not supported — always 32 results per page regardless.
+
+## Finding book IDs
+
+Three ways, in order of preference:
+
+1. **Gutendex search** — returns `id` directly in JSON.
+2. **Gutenberg HTML catalog** — `book_id = re.search(r'/ebooks/(\d+)', entry)`. IDs in the URL.
+3. **URL pattern** — `https://www.gutenberg.org/ebooks/{id}` — if you already know the ID from any source.
+
+Notable IDs validated in tests: `84` (Frankenstein), `1342` (Pride and Prejudice), `11` (Alice in Wonderland), `2701` (Moby Dick), `64317` (The Great Gatsby), `1513` (Romeo and Juliet), `100` (Complete Works of Shakespeare), `1661` (Adventures of Sherlock Holmes), `345` (Dracula).
+
+## Rate limits
+
+Gutendex (`gutendex.com`) returns no `X-RateLimit-*` headers. Server is Apache/2.4.58 on Ubuntu. Rapid sequential calls can trigger connection resets — observed a timeout on the second call in a tight loop. Add a small delay between calls when paginating:
+
+```python
+import time, json
+
+url = "https://gutendex.com/books/?sort=popular"
+while url:
+    data = json.loads(http_get(url))
+    # ... process data['results'] ...
+    url = data['next']
+    if url:
+        time.sleep(0.5)   # be respectful — no published rate limit but timeouts observed
+```
+
+For gutenberg.org file downloads (txt, epub, etc.) there is no documented rate limit but Gutenberg asks not to use automated bulk downloading; use their [offline catalogs](https://www.gutenberg.org/ebooks/offline_catalogs.html) for bulk access.
+
+## Gotchas
+
+- **`.opf` 404**: `https://www.gutenberg.org/cache/epub/1342/pg1342.opf` returns 404. Use `.rdf` instead — same path prefix, same data in RDF/XML.
+- **`formats` URLs redirect**: URLs like `https://www.gutenberg.org/ebooks/1342.txt.utf-8` are redirect endpoints that resolve to `/cache/epub/1342/pg1342.txt`. Either form works with `http_get` (urllib follows redirects automatically), but the `/cache/epub/` direct URL avoids an extra round trip.
+- **Two text files**: `/files/1342/1342-0.txt` (older Project Gutenberg edition, 729 KB) and `/cache/epub/1342/pg1342.txt` (modern edition, 763 KB) contain different versions of the same book. The Gutendex `formats` entry always points to the cache/modern version.
+- **Boilerplate**: Every `.txt` file opens with a PG licence header and closes with a footer. Strip with `START`/`END` markers (see "Read the plain text" section above).
+- **`summaries` field is AI-generated**: The `summaries` array in Gutendex responses contains automatically generated summaries, not the author's original blurb.
+- **`copyright: false`** means public domain in the USA. Non-US copyright status is not tracked.
+- **`page_size` ignored**: Passing `?page_size=5` to Gutendex has no effect — always returns 32 results.
+- **Gutendex `sort=ascending/descending`** sorts by ID (oldest/newest book in the catalog), not by title or author name.
+- **Catalog search `author:` prefix**: `?query=author:dickens` searches within author names but Gutenberg's relevance ranking is fuzzy and can return unexpected results. For precise author lookup use Gutendex `?search=charles+dickens`.
+- **OPDS pagination**: Only 7 entries per page (1 metadata + 6 books). Slow for bulk extraction — use Gutendex instead.
+- **HTML catalog `start_index`**: Pagination is 25 per page. Next page = `start_index=26`, then `51`, `76`, etc. The value appears in the rendered HTML (`re.findall(r'start_index=(\d+)', html)` returns the next page's value).

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/hackernews/scraping.md

@@ -0,0 +1,243 @@
+# Hacker News — Data Extraction
+
+`https://news.ycombinator.com` — YCombinator's link aggregator. Three access paths tested: `http_get` DOM scraping, Algolia search API, and the official HN Firebase API. All work without a browser.
+
+## Do this first: pick your access path
+
+| Goal | Best approach | Latency |
+|------|--------------|---------|
+| Current front page (30 stories, real-time) | `http_get` + regex | ~170ms |
+| Historical / keyword search | Algolia search API | ~400ms |
+| Full comment tree (nested) | Algolia items API | ~300ms |
+| Specific item by ID | Firebase API | ~200ms |
+| 500 ranked story IDs | Firebase topstories | ~200ms (+ ~190ms/item after) |
+
+**Never use a browser for read-only HN tasks.** Everything is accessible over HTTP with no auth, no JS rendering needed.
+
+---
+
+## Path 1: http_get front page (fastest for real-time data)
+
+The front page HTML is ~34KB. Story order matches Firebase `/topstories.json` exactly — confirmed identical on 2026-04-18.
+
+```python
+import re, html as htmllib
+
+page = http_get("https://news.ycombinator.com")
+
+# Extract all 30 story IDs (in rank order)
+story_ids = re.findall(r'<tr class="athing submission" id="(\d+)">', page)
+
+# Extract titles + URLs (same order as IDs)
+titles_urls = re.findall(
+    r'class="titleline"[^>]*><a href="([^"]*)"[^>]*>(.*?)</a>', page
+)
+
+# Extract scores keyed by story ID (job posts have no score row)
+scores_by_id = {
+    m.group(1): int(m.group(2))
+    for m in re.finditer(
+        r'<span class="score" id="score_(\d+)">(\d+) points</span>', page
+    )
+}
+
+# Extract authors keyed by story ID (anchor on score span)
+authors_by_id = {}
+for m in re.finditer(
+    r'<span class="score" id="score_(\d+)">\d+ points</span>'
+    r'.*?class="hnuser">(.*?)</a>',
+    page, re.DOTALL
+):
+    authors_by_id[m.group(1)] = m.group(2)
+
+# Extract comment counts keyed by story ID
+comments_by_id = {
+    m.group(1): int(m.group(2))
+    for m in re.finditer(
+        r'href="item\?id=(\d+)">(\d+)&nbsp;comments</a>', page
+    )
+}
+
+stories = []
+for i, sid in enumerate(story_ids):
+    url, raw_title = titles_urls[i] if i < len(titles_urls) else ('', '')
+    stories.append({
+        'rank': i + 1,
+        'id': sid,
+        'title': htmllib.unescape(raw_title),   # MUST unescape — titles contain &#x27; etc.
+        'url': url,
+        'score': scores_by_id.get(sid),          # None for job posts
+        'author': authors_by_id.get(sid),
+        'comments': comments_by_id.get(sid, 0),
+    })
+```
+
+**Gotchas:**
+- Titles contain HTML entities (`&#x27;` `&amp;` `&quot;` `&gt;`). Always call `html.unescape()`.
+- `<tr class="athing submission" id="...">` — the class is `athing submission`, not just `athing`. The `athing comtr` class is for comment rows.
+- Job/hiring posts (YC ads) appear in the list but have no score or author. `scores_by_id.get(sid)` returns `None` for them — check before comparing.
+- `re.DOTALL` multi-line patterns can cross story boundaries. Use ID-anchored patterns (as above) instead of positional zip for score/author.
+- The page only serves page 1 (30 items). Pages 2–4 exist at `?p=2` etc. but require a login cookie for page 3+.
+
+---
+
+## Path 2: Algolia search API (best for historical / keyword search)
+
+No rate limiting observed. Returns up to 1000 hits per query (`hitsPerPage` max is capped at ~1000 per Algolia plan).
+
+```python
+import json
+
+# Keyword search — sorted by relevance
+data = json.loads(http_get(
+    "https://hn.algolia.com/api/v1/search"
+    "?query=llm&tags=story&hitsPerPage=20"
+))
+
+# Date-sorted (most recent first)
+data = json.loads(http_get(
+    "https://hn.algolia.com/api/v1/search_by_date"
+    "?tags=story&hitsPerPage=20"
+))
+
+# Paginate: add &page=N (0-indexed), up to data['nbPages']-1
+```
+
+**Fields returned per story hit:**
+```
+objectID, title, url, author, points, num_comments,
+created_at (ISO 8601), created_at_i (unix ts), story_id,
+children (list of comment IDs — flat, not tree),
+_tags, _highlightResult
+```
+
+**Fields returned per comment hit:**
+```
+objectID, comment_text, author, story_id, story_title, story_url,
+parent_id, created_at, created_at_i, points
+```
+Note: comment hits use `comment_text`, NOT `text`. Story hits use `story_text` for self-post body.
+
+### Tag filters
+
+Tags are AND by default, OR with parentheses:
+
+```python
+# Story types
+"tags=story"           # regular link/self posts
+"tags=show_hn"         # Show HN
+"tags=ask_hn"          # Ask HN
+"tags=poll"            # polls
+"tags=job"             # job posts
+
+# Combined AND
+"tags=story,front_page"          # currently on front page
+"tags=story,author_pg"           # stories submitted by pg
+
+# OR
+"tags=(ask_hn,show_hn),story"    # Ask OR Show HN
+
+# By story ID (gets story + all its comments)
+"tags=story_47806725"
+```
+
+### Numeric filters
+
+```python
+# Date range (unix timestamps)
+"numericFilters=created_at_i>1745000000"
+"numericFilters=created_at_i>1700000000,created_at_i<1750000000"
+
+# Point threshold
+"numericFilters=points>100"
+"numericFilters=points>500,points<1000"
+```
+
+### Full Algolia items API (nested comment tree)
+
+```python
+import json
+
+thread = json.loads(http_get(
+    "https://hn.algolia.com/api/v1/items/47806725"
+))
+# thread['children'] = list of top-level comment objects
+# Each comment: author, text (HTML), created_at, children (nested replies)
+# Recursively walk children for full thread
+
+# Total comment count (recursive walk with stack):
+stack = list(thread.get('children', []))
+total = 0
+while stack:
+    node = stack.pop()
+    total += 1
+    stack.extend(node.get('children', []))
+```
+
+Confirmed: Algolia items returns 653 total comments for a 659-comment thread (some deleted). `text` field in items API is HTML with `<p>` tags and `<a>` links — may need to strip tags.
+
+---
+
+## Path 3: Official HN Firebase API
+
+Clean JSON, no scraping. Use for fetching specific items or building live feeds.
+
+```python
+import json
+
+# Ranked story ID lists (no metadata — just IDs)
+top   = json.loads(http_get("https://hacker-news.firebaseio.com/v0/topstories.json"))  # 500 IDs
+new   = json.loads(http_get("https://hacker-news.firebaseio.com/v0/newstories.json"))  # 500 IDs
+best  = json.loads(http_get("https://hacker-news.firebaseio.com/v0/beststories.json")) # 200 IDs
+ask   = json.loads(http_get("https://hacker-news.firebaseio.com/v0/askstories.json"))  # ~32 IDs
+show  = json.loads(http_get("https://hacker-news.firebaseio.com/v0/showstories.json")) # ~119 IDs
+jobs  = json.loads(http_get("https://hacker-news.firebaseio.com/v0/jobstories.json"))  # ~31 IDs
+
+# Fetch a single item
+item = json.loads(http_get(
+    "https://hacker-news.firebaseio.com/v0/item/47806725.json"
+))
+# Fields: id, type, by, title, url, score, descendants (total comment count),
+#         time (unix ts), kids (list of top-level comment IDs), text (self-post body)
+
+# Fetch a user profile
+user = json.loads(http_get(
+    "https://hacker-news.firebaseio.com/v0/user/pg.json"
+))
+# Fields: id, karma, created (unix ts), about (HTML), submitted (list of item IDs)
+
+# Highest current item ID (useful for polling new items)
+maxid = json.loads(http_get("https://hacker-news.firebaseio.com/v0/maxitem.json"))
+```
+
+**Firebase vs Algolia tradeoff:**
+- Firebase `topstories` gives you 500 IDs in one call but then requires one HTTP call per item (~190ms each). Fetching all 500 items sequentially would take ~100 seconds.
+- Algolia returns full story data (title, points, author, comments) in one call for up to ~1000 results.
+- For "top 30 stories with full metadata": use `http_get` front page scrape (170ms total). For "top 500 stories with full metadata": use Algolia with `tags=front_page` or loop pages.
+
+---
+
+## Comment thread HTML (item page)
+
+For a large thread, the item page HTML (~1MB for 659 comments) loads ALL comments flat in a single request — no pagination, no JS required.
+
+```python
+import re, html as htmllib
+
+page = http_get("https://news.ycombinator.com/item?id=47806725")
+
+# Count all comment IDs
+comment_ids = re.findall(r'<tr class="athing comtr" id="(\d+)">', page)
+# len(comment_ids) matches total comment count
+
+# Extract comment texts (careful: text spans multiple lines with <p> tags)
+# Use Algolia items API instead for structured access
+```
+
+For structured comment access prefer Algolia items API — it returns a proper nested tree. The HTML item page is useful only when you need approximate comment count without an API call.
+
+---
+
+## Do NOT use a browser for HN
+
+All data is in plain HTML or JSON APIs. `goto_url()` + `wait_for_load()` takes 3–8 seconds; `http_get` takes 170–400ms. The JS `querySelectorAll` approach works (tested, returns correct data) but is 20–50x slower with no benefit.