@pencil-agent/nano-pencil 2.0.0-beta.8 → 2.0.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/gutenberg/scraping.md

@@ -1,383 +1,383 @@
-# Project Gutenberg — Scraping & Data Extraction
-
-`https://www.gutenberg.org` — 78 000+ free public-domain ebooks. Every workflow here is pure `http_get` — no browser needed.
-
-## Do this first
-
-**Use the Gutendex REST API (`gutendex.com`) for all search and discovery. It is one call, returns clean JSON, and requires no auth. Go to gutenberg.org URLs only to fetch actual file content.**
-
-```python
-import json
-
-# Search by title/author keyword
-data = json.loads(http_get("https://gutendex.com/books/?search=pride+and+prejudice"))
-# data['count'] = 6 (total matches)
-# data['results'] = list of up to 32 book objects
-book = data['results'][0]
-# book['id'] = 1342  ← use this ID for all further calls
-# book['formats']['text/plain; charset=utf-8'] = direct txt URL
-
-# Fetch the plain-text content of that book
-text = http_get(book['formats']['text/plain; charset=utf-8'])
-# Returns 763 083 chars including Project Gutenberg header/footer boilerplate
-```
-
-For a known book ID, skip search entirely:
-
-```python
-book = json.loads(http_get("https://gutendex.com/books/1342/"))
-```
-
-## Common workflows
-
-### Search by keyword and get the first result
-
-```python
-import json
-
-data = json.loads(http_get("https://gutendex.com/books/?search=frankenstein"))
-if data['results']:
-    b = data['results'][0]
-    print(b['id'], b['title'], b['authors'][0]['name'])
-    # 84  Frankenstein; or, the modern prometheus  Shelley, Mary Wollstonecraft
-    txt_url = b['formats'].get('text/plain; charset=utf-8')
-    if txt_url:
-        text = http_get(txt_url)
-```
-
-### Get the most downloaded books (popularity ranking)
-
-```python
-import json
-
-data = json.loads(http_get("https://gutendex.com/books/?sort=popular"))
-for b in data['results'][:10]:
-    authors = ', '.join(a['name'] for a in b['authors'])
-    print(f"[{b['id']}] {b['title']} — {authors} ({b['download_count']:,} downloads)")
-# [84]    Frankenstein                      — Shelley, Mary Wollstonecraft  (178,271)
-# [45304] The City of God, Volume I         — Augustine, of Hippo, Saint    (147,663)
-# [2701]  Moby Dick; Or, The Whale          — Melville, Herman              (112,302)
-# [1342]  Pride and Prejudice               — Austen, Jane                  (107,502)
-# [768]   Wuthering Heights                 — Brontë, Emily                  (72,775)
-# [1513]  Romeo and Juliet                  — Shakespeare, William           (70,272)
-# [11]    Alice's Adventures in Wonderland  — Carroll, Lewis                 (65,243)
-# [64317] The Great Gatsby                  — Fitzgerald, F. Scott           (60,632)
-# [100]   Complete Works of Shakespeare     — Shakespeare, William           (60,527)
-# [1260]  Jane Eyre: An Autobiography       — Brontë, Charlotte              (57,602)
-```
-
-### Browse by genre / topic
-
-```python
-import json
-
-# 'topic' matches both subjects and bookshelves fields
-data = json.loads(http_get("https://gutendex.com/books/?topic=science+fiction"))
-# data['count'] = 3473 total results, 32 per page
-
-data = json.loads(http_get("https://gutendex.com/books/?topic=detective+fiction"))
-# data['count'] = 111
-# data['results'][0]: id=1661 The Adventures of Sherlock Holmes — Doyle, Arthur Conan
-
-# Filter by language (ISO 639-1 code)
-data = json.loads(http_get("https://gutendex.com/books/?languages=fr&topic=roman"))
-# data['count'] = 254 French books with 'roman' in topic
-```
-
-### Paginate through results
-
-```python
-import json
-
-url = "https://gutendex.com/books/?topic=science+fiction"
-books = []
-while url:
-    data = json.loads(http_get(url))
-    books.extend(data['results'])
-    url = data['next']   # None on last page
-    # data['previous'] is also populated after page 1
-    # e.g. data['next'] = "https://gutendex.com/books/?page=3&topic=science+fiction"
-# All 3473 sci-fi books loaded across ~109 pages of 32 each
-```
-
-### Fetch multiple specific books by ID
-
-```python
-import json
-
-data = json.loads(http_get("https://gutendex.com/books/?ids=1342,11,84"))
-# Returns exactly those 3 books, count=3
-for b in data['results']:
-    print(b['id'], b['title'])
-# 84    Frankenstein; or, the modern prometheus
-# 1342  Pride and Prejudice
-# 11    Alice's Adventures in Wonderland
-```
-
-### Read the plain text of a book (boilerplate stripped)
-
-```python
-raw = http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.txt")
-# 763 083 chars total including PG licence header and footer
-
-START = "*** START OF THE PROJECT GUTENBERG EBOOK"
-END   = "*** END OF THE PROJECT GUTENBERG EBOOK"
-s = raw.find(START)
-e = raw.find(END)
-if s != -1:
-    content = raw[raw.index('\n', s) + 1 : e].strip()
-    # 743 241 chars of actual novel text
-```
-
-The cache URL is the most reliable direct path. The `formats` dict in Gutendex also provides a redirect URL that resolves to the same file:
-
-```python
-# Both of these return identical content (763 083 chars):
-http_get("https://www.gutenberg.org/ebooks/1342.txt.utf-8")          # redirect
-http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.txt")     # direct cache
-```
-
-### Download formats available per book
-
-Every book's `formats` dict maps MIME type to URL. All URLs resolve to `/cache/epub/{id}/` files via redirect.
-
-| MIME type | URL pattern (after redirect) | Typical size |
-|---|---|---|
-| `text/plain; charset=utf-8` | `pg{id}.txt` | ~750 KB |
-| `text/html` | `pg{id}-images.html` | ~850 KB |
-| `application/epub+zip` | `pg{id}-images-3.epub` | ~25 MB |
-| `application/x-mobipocket-ebook` | `pg{id}-images-kf8.mobi` | ~25 MB |
-| `application/rdf+xml` | `{id}.rdf` via gutenberg.org | metadata XML |
-| `image/jpeg` | `pg{id}.cover.medium.jpg` | cover image |
-| `application/octet-stream` | `pg{id}-h.zip` | HTML+images zip |
-
-```python
-import json
-
-b = json.loads(http_get("https://gutendex.com/books/1342/"))
-# Grab every downloadable format URL:
-for mime, url in b['formats'].items():
-    print(mime, '->', url)
-# text/html                         -> https://www.gutenberg.org/ebooks/1342.html.images
-# application/epub+zip              -> https://www.gutenberg.org/ebooks/1342.epub3.images
-# application/x-mobipocket-ebook   -> https://www.gutenberg.org/ebooks/1342.kf8.images
-# application/rdf+xml               -> https://www.gutenberg.org/ebooks/1342.rdf
-# image/jpeg                        -> https://www.gutenberg.org/cache/epub/1342/pg1342.cover.medium.jpg
-# application/octet-stream          -> https://www.gutenberg.org/cache/epub/1342/pg1342-h.zip
-# text/plain; charset=utf-8         -> https://www.gutenberg.org/ebooks/1342.txt.utf-8
-```
-
-### Fetch RDF/XML metadata for a book
-
-```python
-import re
-
-rdf = http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.rdf")
-# Also available as: http_get("https://www.gutenberg.org/ebooks/1342.rdf")
-
-title     = re.search(r'<dcterms:title>(.*?)</dcterms:title>', rdf, re.DOTALL)
-creator   = re.findall(r'<pgterms:name>(.*?)</pgterms:name>', rdf)
-birth     = re.findall(r'<pgterms:birthdate[^>]*>(\d+)', rdf)
-death     = re.findall(r'<pgterms:deathdate[^>]*>(\d+)', rdf)
-issued    = re.search(r'<dcterms:issued[^>]*>(.*?)</dcterms:issued>', rdf)
-rights    = re.search(r'<dcterms:rights>(.*?)</dcterms:rights>', rdf)
-downloads = re.search(r'<pgterms:downloads[^>]*>(\d+)</pgterms:downloads>', rdf)
-language  = re.search(r'<dcterms:language>.*?<rdf:value>(.*?)</rdf:value>', rdf, re.DOTALL)
-subjects  = re.findall(r'<dcterms:subject>.*?<rdf:value>(.*?)</rdf:value>.*?</dcterms:subject>', rdf, re.DOTALL)
-
-print(title.group(1))          # Pride and Prejudice
-print(creator)                 # ['Austen, Jane']
-print(birth, death)            # ['1775'] ['1817']
-print(issued.group(1))         # 1998-06-01
-print(rights.group(1))         # Public domain in the USA.
-print(int(downloads.group(1))) # 107502
-print(subjects[:3])            # ['England -- Fiction', 'Young women -- Fiction', 'Love stories']
-```
-
-Note: `<dcterms:language>` value is a subject string, not a language code. For language codes use the Gutendex `languages` field instead.
-
-### Search the HTML catalog (25 results per page)
-
-Use this only when you need to leverage Gutenberg's own search index (author:, title:, subject: prefix syntax).
-
-```python
-import re, json
-
-html = http_get(
-    "https://www.gutenberg.org/ebooks/search/"
-    "?query=shakespeare&sort_order=downloads"
-)
-# sort_order options: downloads, title, release_date, last_update, random
-
-entries = re.findall(r'<li class="booklink">(.*?)</li>', html, re.DOTALL)
-books = []
-for e in entries:
-    book_id   = re.search(r'/ebooks/(\d+)', e)
-    title     = re.search(r'<span class="title">(.*?)</span>', e)
-    author    = re.search(r'<span class="subtitle">(.*?)</span>', e)
-    downloads = re.search(r'<span class="extra">([^<]+)</span>', e)
-    books.append({
-        'id':        int(book_id.group(1)) if book_id else None,
-        'title':     title.group(1) if title else '',
-        'author':    author.group(1) if author else '',
-        'downloads': downloads.group(1).strip() if downloads else '',
-    })
-
-# books[0] = {'id': 1513, 'title': 'Romeo and Juliet',
-#             'author': 'William Shakespeare', 'downloads': '74316 downloads'}
-
-# Paginate with start_index (25 per page)
-html_p2 = http_get(
-    "https://www.gutenberg.org/ebooks/search/"
-    "?query=shakespeare&sort_order=downloads&start_index=26"
-)
-```
-
-### Browse a bookshelf (curated genre list)
-
-```python
-import re
-
-# Bookshelf 68 = Science Fiction
-html = http_get("https://www.gutenberg.org/ebooks/bookshelf/68")
-titles = re.findall(r'<span class="title">(.*?)</span>', html)
-# ['Twenty Thousand Leagues under the Sea', 'The War of the Worlds',
-#  'The Time Machine', 'Thuvia, Maid of Mars', ...]
-```
-
-### OPDS catalog (machine-readable Atom feed)
-
-```python
-import re
-
-feed = http_get("https://www.gutenberg.org/ebooks/search.opds/?query=dracula")
-# Returns Atom XML, 7 entries per page (including 1 metadata entry)
-entries = re.findall(r'<entry>(.*?)</entry>', feed, re.DOTALL)
-for e in entries:
-    title = re.search(r'<title>(.*?)</title>', e)
-    entry_id = re.search(r'<id>(.*?)</id>', e)
-    if title and entry_id and 'opds' in entry_id.group(1):
-        book_id = re.search(r'/ebooks/(\d+)\.opds', entry_id.group(1))
-        print(book_id.group(1), title.group(1))
-# 345  Dracula
-```
-
-## Gutendex API — full response schema
-
-Validated against a real call to `GET https://gutendex.com/books/1342/`:
-
-```json
-{
-  "id": 1342,
-  "title": "Pride and Prejudice",
-  "authors": [
-    {"name": "Austen, Jane", "birth_year": 1775, "death_year": 1817}
-  ],
-  "summaries": ["...automatically generated summary..."],
-  "editors": [],
-  "translators": [],
-  "subjects": [
-    "Courtship -- Fiction",
-    "Domestic fiction",
-    "England -- Fiction",
-    "Love stories",
-    "Sisters -- Fiction",
-    "Women -- England -- Fiction",
-    "Young women -- Fiction"
-  ],
-  "bookshelves": [
-    "Best Books Ever Listings",
-    "Category: British Literature",
-    "Category: Classics of Literature",
-    "Category: Novels",
-    "Category: Romance",
-    "Harvard Classics"
-  ],
-  "languages": ["en"],
-  "copyright": false,
-  "media_type": "Text",
-  "formats": {
-    "text/html":                       "https://www.gutenberg.org/ebooks/1342.html.images",
-    "application/epub+zip":            "https://www.gutenberg.org/ebooks/1342.epub3.images",
-    "application/x-mobipocket-ebook": "https://www.gutenberg.org/ebooks/1342.kf8.images",
-    "application/rdf+xml":             "https://www.gutenberg.org/ebooks/1342.rdf",
-    "image/jpeg":                      "https://www.gutenberg.org/cache/epub/1342/pg1342.cover.medium.jpg",
-    "application/octet-stream":        "https://www.gutenberg.org/cache/epub/1342/pg1342-h.zip",
-    "text/plain; charset=utf-8":       "https://www.gutenberg.org/ebooks/1342.txt.utf-8"
-  },
-  "download_count": 107502
-}
-```
-
-List response wrapper (from `GET /books/`):
-
-```json
-{
-  "count": 6,
-  "next": null,
-  "previous": null,
-  "results": [...]
-}
-```
-
-`count` is the total across all pages. `next` / `previous` are fully-formed URLs ready to pass to `http_get`, or `null` when absent.
-
-## Gutendex query parameters
-
-All parameters combine freely.
-
-| Parameter | Example | Notes |
-|---|---|---|
-| `search` | `search=moby+dick` | Matches title and author |
-| `ids` | `ids=1342,11,84` | Comma-separated; returns only those books |
-| `languages` | `languages=fr` | ISO 639-1 code; comma-separated for multiple |
-| `topic` | `topic=science+fiction` | Matches subjects + bookshelves |
-| `author_year_start` | `author_year_start=1800` | Author born on/after year |
-| `author_year_end` | `author_year_end=1850` | Author born on/before year |
-| `copyright` | `copyright=false` | `false`=public domain, `true`=copyrighted |
-| `sort` | `sort=popular` | `popular` (default), `ascending`, `descending` |
-| `page` | `page=2` | 1-based; 32 results per page (not configurable) |
-
-`page_size` is not supported — always 32 results per page regardless.
-
-## Finding book IDs
-
-Three ways, in order of preference:
-
-1. **Gutendex search** — returns `id` directly in JSON.
-2. **Gutenberg HTML catalog** — `book_id = re.search(r'/ebooks/(\d+)', entry)`. IDs in the URL.
-3. **URL pattern** — `https://www.gutenberg.org/ebooks/{id}` — if you already know the ID from any source.
-
-Notable IDs validated in tests: `84` (Frankenstein), `1342` (Pride and Prejudice), `11` (Alice in Wonderland), `2701` (Moby Dick), `64317` (The Great Gatsby), `1513` (Romeo and Juliet), `100` (Complete Works of Shakespeare), `1661` (Adventures of Sherlock Holmes), `345` (Dracula).
-
-## Rate limits
-
-Gutendex (`gutendex.com`) returns no `X-RateLimit-*` headers. Server is Apache/2.4.58 on Ubuntu. Rapid sequential calls can trigger connection resets — observed a timeout on the second call in a tight loop. Add a small delay between calls when paginating:
-
-```python
-import time, json
-
-url = "https://gutendex.com/books/?sort=popular"
-while url:
-    data = json.loads(http_get(url))
-    # ... process data['results'] ...
-    url = data['next']
-    if url:
-        time.sleep(0.5)   # be respectful — no published rate limit but timeouts observed
-```
-
-For gutenberg.org file downloads (txt, epub, etc.) there is no documented rate limit but Gutenberg asks not to use automated bulk downloading; use their [offline catalogs](https://www.gutenberg.org/ebooks/offline_catalogs.html) for bulk access.
-
-## Gotchas
-
-- **`.opf` 404**: `https://www.gutenberg.org/cache/epub/1342/pg1342.opf` returns 404. Use `.rdf` instead — same path prefix, same data in RDF/XML.
-- **`formats` URLs redirect**: URLs like `https://www.gutenberg.org/ebooks/1342.txt.utf-8` are redirect endpoints that resolve to `/cache/epub/1342/pg1342.txt`. Either form works with `http_get` (urllib follows redirects automatically), but the `/cache/epub/` direct URL avoids an extra round trip.
-- **Two text files**: `/files/1342/1342-0.txt` (older Project Gutenberg edition, 729 KB) and `/cache/epub/1342/pg1342.txt` (modern edition, 763 KB) contain different versions of the same book. The Gutendex `formats` entry always points to the cache/modern version.
-- **Boilerplate**: Every `.txt` file opens with a PG licence header and closes with a footer. Strip with `START`/`END` markers (see "Read the plain text" section above).
-- **`summaries` field is AI-generated**: The `summaries` array in Gutendex responses contains automatically generated summaries, not the author's original blurb.
-- **`copyright: false`** means public domain in the USA. Non-US copyright status is not tracked.
-- **`page_size` ignored**: Passing `?page_size=5` to Gutendex has no effect — always returns 32 results.
-- **Gutendex `sort=ascending/descending`** sorts by ID (oldest/newest book in the catalog), not by title or author name.
-- **Catalog search `author:` prefix**: `?query=author:dickens` searches within author names but Gutenberg's relevance ranking is fuzzy and can return unexpected results. For precise author lookup use Gutendex `?search=charles+dickens`.
-- **OPDS pagination**: Only 7 entries per page (1 metadata + 6 books). Slow for bulk extraction — use Gutendex instead.
-- **HTML catalog `start_index`**: Pagination is 25 per page. Next page = `start_index=26`, then `51`, `76`, etc. The value appears in the rendered HTML (`re.findall(r'start_index=(\d+)', html)` returns the next page's value).
+# Project Gutenberg — Scraping & Data Extraction
+
+`https://www.gutenberg.org` — 78 000+ free public-domain ebooks. Every workflow here is pure `http_get` — no browser needed.
+
+## Do this first
+
+**Use the Gutendex REST API (`gutendex.com`) for all search and discovery. It is one call, returns clean JSON, and requires no auth. Go to gutenberg.org URLs only to fetch actual file content.**
+
+```python
+import json
+
+# Search by title/author keyword
+data = json.loads(http_get("https://gutendex.com/books/?search=pride+and+prejudice"))
+# data['count'] = 6 (total matches)
+# data['results'] = list of up to 32 book objects
+book = data['results'][0]
+# book['id'] = 1342  ← use this ID for all further calls
+# book['formats']['text/plain; charset=utf-8'] = direct txt URL
+
+# Fetch the plain-text content of that book
+text = http_get(book['formats']['text/plain; charset=utf-8'])
+# Returns 763 083 chars including Project Gutenberg header/footer boilerplate
+```
+
+For a known book ID, skip search entirely:
+
+```python
+book = json.loads(http_get("https://gutendex.com/books/1342/"))
+```
+
+## Common workflows
+
+### Search by keyword and get the first result
+
+```python
+import json
+
+data = json.loads(http_get("https://gutendex.com/books/?search=frankenstein"))
+if data['results']:
+    b = data['results'][0]
+    print(b['id'], b['title'], b['authors'][0]['name'])
+    # 84  Frankenstein; or, the modern prometheus  Shelley, Mary Wollstonecraft
+    txt_url = b['formats'].get('text/plain; charset=utf-8')
+    if txt_url:
+        text = http_get(txt_url)
+```
+
+### Get the most downloaded books (popularity ranking)
+
+```python
+import json
+
+data = json.loads(http_get("https://gutendex.com/books/?sort=popular"))
+for b in data['results'][:10]:
+    authors = ', '.join(a['name'] for a in b['authors'])
+    print(f"[{b['id']}] {b['title']} — {authors} ({b['download_count']:,} downloads)")
+# [84]    Frankenstein                      — Shelley, Mary Wollstonecraft  (178,271)
+# [45304] The City of God, Volume I         — Augustine, of Hippo, Saint    (147,663)
+# [2701]  Moby Dick; Or, The Whale          — Melville, Herman              (112,302)
+# [1342]  Pride and Prejudice               — Austen, Jane                  (107,502)
+# [768]   Wuthering Heights                 — Brontë, Emily                  (72,775)
+# [1513]  Romeo and Juliet                  — Shakespeare, William           (70,272)
+# [11]    Alice's Adventures in Wonderland  — Carroll, Lewis                 (65,243)
+# [64317] The Great Gatsby                  — Fitzgerald, F. Scott           (60,632)
+# [100]   Complete Works of Shakespeare     — Shakespeare, William           (60,527)
+# [1260]  Jane Eyre: An Autobiography       — Brontë, Charlotte              (57,602)
+```
+
+### Browse by genre / topic
+
+```python
+import json
+
+# 'topic' matches both subjects and bookshelves fields
+data = json.loads(http_get("https://gutendex.com/books/?topic=science+fiction"))
+# data['count'] = 3473 total results, 32 per page
+
+data = json.loads(http_get("https://gutendex.com/books/?topic=detective+fiction"))
+# data['count'] = 111
+# data['results'][0]: id=1661 The Adventures of Sherlock Holmes — Doyle, Arthur Conan
+
+# Filter by language (ISO 639-1 code)
+data = json.loads(http_get("https://gutendex.com/books/?languages=fr&topic=roman"))
+# data['count'] = 254 French books with 'roman' in topic
+```
+
+### Paginate through results
+
+```python
+import json
+
+url = "https://gutendex.com/books/?topic=science+fiction"
+books = []
+while url:
+    data = json.loads(http_get(url))
+    books.extend(data['results'])
+    url = data['next']   # None on last page
+    # data['previous'] is also populated after page 1
+    # e.g. data['next'] = "https://gutendex.com/books/?page=3&topic=science+fiction"
+# All 3473 sci-fi books loaded across ~109 pages of 32 each
+```
+
+### Fetch multiple specific books by ID
+
+```python
+import json
+
+data = json.loads(http_get("https://gutendex.com/books/?ids=1342,11,84"))
+# Returns exactly those 3 books, count=3
+for b in data['results']:
+    print(b['id'], b['title'])
+# 84    Frankenstein; or, the modern prometheus
+# 1342  Pride and Prejudice
+# 11    Alice's Adventures in Wonderland
+```
+
+### Read the plain text of a book (boilerplate stripped)
+
+```python
+raw = http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.txt")
+# 763 083 chars total including PG licence header and footer
+
+START = "*** START OF THE PROJECT GUTENBERG EBOOK"
+END   = "*** END OF THE PROJECT GUTENBERG EBOOK"
+s = raw.find(START)
+e = raw.find(END)
+if s != -1:
+    content = raw[raw.index('\n', s) + 1 : e].strip()
+    # 743 241 chars of actual novel text
+```
+
+The cache URL is the most reliable direct path. The `formats` dict in Gutendex also provides a redirect URL that resolves to the same file:
+
+```python
+# Both of these return identical content (763 083 chars):
+http_get("https://www.gutenberg.org/ebooks/1342.txt.utf-8")          # redirect
+http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.txt")     # direct cache
+```
+
+### Download formats available per book
+
+Every book's `formats` dict maps MIME type to URL. All URLs resolve to `/cache/epub/{id}/` files via redirect.
+
+| MIME type | URL pattern (after redirect) | Typical size |
+|---|---|---|
+| `text/plain; charset=utf-8` | `pg{id}.txt` | ~750 KB |
+| `text/html` | `pg{id}-images.html` | ~850 KB |
+| `application/epub+zip` | `pg{id}-images-3.epub` | ~25 MB |
+| `application/x-mobipocket-ebook` | `pg{id}-images-kf8.mobi` | ~25 MB |
+| `application/rdf+xml` | `{id}.rdf` via gutenberg.org | metadata XML |
+| `image/jpeg` | `pg{id}.cover.medium.jpg` | cover image |
+| `application/octet-stream` | `pg{id}-h.zip` | HTML+images zip |
+
+```python
+import json
+
+b = json.loads(http_get("https://gutendex.com/books/1342/"))
+# Grab every downloadable format URL:
+for mime, url in b['formats'].items():
+    print(mime, '->', url)
+# text/html                         -> https://www.gutenberg.org/ebooks/1342.html.images
+# application/epub+zip              -> https://www.gutenberg.org/ebooks/1342.epub3.images
+# application/x-mobipocket-ebook   -> https://www.gutenberg.org/ebooks/1342.kf8.images
+# application/rdf+xml               -> https://www.gutenberg.org/ebooks/1342.rdf
+# image/jpeg                        -> https://www.gutenberg.org/cache/epub/1342/pg1342.cover.medium.jpg
+# application/octet-stream          -> https://www.gutenberg.org/cache/epub/1342/pg1342-h.zip
+# text/plain; charset=utf-8         -> https://www.gutenberg.org/ebooks/1342.txt.utf-8
+```
+
+### Fetch RDF/XML metadata for a book
+
+```python
+import re
+
+rdf = http_get("https://www.gutenberg.org/cache/epub/1342/pg1342.rdf")
+# Also available as: http_get("https://www.gutenberg.org/ebooks/1342.rdf")
+
+title     = re.search(r'<dcterms:title>(.*?)</dcterms:title>', rdf, re.DOTALL)
+creator   = re.findall(r'<pgterms:name>(.*?)</pgterms:name>', rdf)
+birth     = re.findall(r'<pgterms:birthdate[^>]*>(\d+)', rdf)
+death     = re.findall(r'<pgterms:deathdate[^>]*>(\d+)', rdf)
+issued    = re.search(r'<dcterms:issued[^>]*>(.*?)</dcterms:issued>', rdf)
+rights    = re.search(r'<dcterms:rights>(.*?)</dcterms:rights>', rdf)
+downloads = re.search(r'<pgterms:downloads[^>]*>(\d+)</pgterms:downloads>', rdf)
+language  = re.search(r'<dcterms:language>.*?<rdf:value>(.*?)</rdf:value>', rdf, re.DOTALL)
+subjects  = re.findall(r'<dcterms:subject>.*?<rdf:value>(.*?)</rdf:value>.*?</dcterms:subject>', rdf, re.DOTALL)
+
+print(title.group(1))          # Pride and Prejudice
+print(creator)                 # ['Austen, Jane']
+print(birth, death)            # ['1775'] ['1817']
+print(issued.group(1))         # 1998-06-01
+print(rights.group(1))         # Public domain in the USA.
+print(int(downloads.group(1))) # 107502
+print(subjects[:3])            # ['England -- Fiction', 'Young women -- Fiction', 'Love stories']
+```
+
+Note: `<dcterms:language>` value is a subject string, not a language code. For language codes use the Gutendex `languages` field instead.
+
+### Search the HTML catalog (25 results per page)
+
+Use this only when you need to leverage Gutenberg's own search index (author:, title:, subject: prefix syntax).
+
+```python
+import re, json
+
+html = http_get(
+    "https://www.gutenberg.org/ebooks/search/"
+    "?query=shakespeare&sort_order=downloads"
+)
+# sort_order options: downloads, title, release_date, last_update, random
+
+entries = re.findall(r'<li class="booklink">(.*?)</li>', html, re.DOTALL)
+books = []
+for e in entries:
+    book_id   = re.search(r'/ebooks/(\d+)', e)
+    title     = re.search(r'<span class="title">(.*?)</span>', e)
+    author    = re.search(r'<span class="subtitle">(.*?)</span>', e)
+    downloads = re.search(r'<span class="extra">([^<]+)</span>', e)
+    books.append({
+        'id':        int(book_id.group(1)) if book_id else None,
+        'title':     title.group(1) if title else '',
+        'author':    author.group(1) if author else '',
+        'downloads': downloads.group(1).strip() if downloads else '',
+    })
+
+# books[0] = {'id': 1513, 'title': 'Romeo and Juliet',
+#             'author': 'William Shakespeare', 'downloads': '74316 downloads'}
+
+# Paginate with start_index (25 per page)
+html_p2 = http_get(
+    "https://www.gutenberg.org/ebooks/search/"
+    "?query=shakespeare&sort_order=downloads&start_index=26"
+)
+```
+
+### Browse a bookshelf (curated genre list)
+
+```python
+import re
+
+# Bookshelf 68 = Science Fiction
+html = http_get("https://www.gutenberg.org/ebooks/bookshelf/68")
+titles = re.findall(r'<span class="title">(.*?)</span>', html)
+# ['Twenty Thousand Leagues under the Sea', 'The War of the Worlds',
+#  'The Time Machine', 'Thuvia, Maid of Mars', ...]
+```
+
+### OPDS catalog (machine-readable Atom feed)
+
+```python
+import re
+
+feed = http_get("https://www.gutenberg.org/ebooks/search.opds/?query=dracula")
+# Returns Atom XML, 7 entries per page (including 1 metadata entry)
+entries = re.findall(r'<entry>(.*?)</entry>', feed, re.DOTALL)
+for e in entries:
+    title = re.search(r'<title>(.*?)</title>', e)
+    entry_id = re.search(r'<id>(.*?)</id>', e)
+    if title and entry_id and 'opds' in entry_id.group(1):
+        book_id = re.search(r'/ebooks/(\d+)\.opds', entry_id.group(1))
+        print(book_id.group(1), title.group(1))
+# 345  Dracula
+```
+
+## Gutendex API — full response schema
+
+Validated against a real call to `GET https://gutendex.com/books/1342/`:
+
+```json
+{
+  "id": 1342,
+  "title": "Pride and Prejudice",
+  "authors": [
+    {"name": "Austen, Jane", "birth_year": 1775, "death_year": 1817}
+  ],
+  "summaries": ["...automatically generated summary..."],
+  "editors": [],
+  "translators": [],
+  "subjects": [
+    "Courtship -- Fiction",
+    "Domestic fiction",
+    "England -- Fiction",
+    "Love stories",
+    "Sisters -- Fiction",
+    "Women -- England -- Fiction",
+    "Young women -- Fiction"
+  ],
+  "bookshelves": [
+    "Best Books Ever Listings",
+    "Category: British Literature",
+    "Category: Classics of Literature",
+    "Category: Novels",
+    "Category: Romance",
+    "Harvard Classics"
+  ],
+  "languages": ["en"],
+  "copyright": false,
+  "media_type": "Text",
+  "formats": {
+    "text/html":                       "https://www.gutenberg.org/ebooks/1342.html.images",
+    "application/epub+zip":            "https://www.gutenberg.org/ebooks/1342.epub3.images",
+    "application/x-mobipocket-ebook": "https://www.gutenberg.org/ebooks/1342.kf8.images",
+    "application/rdf+xml":             "https://www.gutenberg.org/ebooks/1342.rdf",
+    "image/jpeg":                      "https://www.gutenberg.org/cache/epub/1342/pg1342.cover.medium.jpg",
+    "application/octet-stream":        "https://www.gutenberg.org/cache/epub/1342/pg1342-h.zip",
+    "text/plain; charset=utf-8":       "https://www.gutenberg.org/ebooks/1342.txt.utf-8"
+  },
+  "download_count": 107502
+}
+```
+
+List response wrapper (from `GET /books/`):
+
+```json
+{
+  "count": 6,
+  "next": null,
+  "previous": null,
+  "results": [...]
+}
+```
+
+`count` is the total across all pages. `next` / `previous` are fully-formed URLs ready to pass to `http_get`, or `null` when absent.
+
+## Gutendex query parameters
+
+All parameters combine freely.
+
+| Parameter | Example | Notes |
+|---|---|---|
+| `search` | `search=moby+dick` | Matches title and author |
+| `ids` | `ids=1342,11,84` | Comma-separated; returns only those books |
+| `languages` | `languages=fr` | ISO 639-1 code; comma-separated for multiple |
+| `topic` | `topic=science+fiction` | Matches subjects + bookshelves |
+| `author_year_start` | `author_year_start=1800` | Author born on/after year |
+| `author_year_end` | `author_year_end=1850` | Author born on/before year |
+| `copyright` | `copyright=false` | `false`=public domain, `true`=copyrighted |
+| `sort` | `sort=popular` | `popular` (default), `ascending`, `descending` |
+| `page` | `page=2` | 1-based; 32 results per page (not configurable) |
+
+`page_size` is not supported — always 32 results per page regardless.
+
+## Finding book IDs
+
+Three ways, in order of preference:
+
+1. **Gutendex search** — returns `id` directly in JSON.
+2. **Gutenberg HTML catalog** — `book_id = re.search(r'/ebooks/(\d+)', entry)`. IDs in the URL.
+3. **URL pattern** — `https://www.gutenberg.org/ebooks/{id}` — if you already know the ID from any source.
+
+Notable IDs validated in tests: `84` (Frankenstein), `1342` (Pride and Prejudice), `11` (Alice in Wonderland), `2701` (Moby Dick), `64317` (The Great Gatsby), `1513` (Romeo and Juliet), `100` (Complete Works of Shakespeare), `1661` (Adventures of Sherlock Holmes), `345` (Dracula).
+
+## Rate limits
+
+Gutendex (`gutendex.com`) returns no `X-RateLimit-*` headers. Server is Apache/2.4.58 on Ubuntu. Rapid sequential calls can trigger connection resets — observed a timeout on the second call in a tight loop. Add a small delay between calls when paginating:
+
+```python
+import time, json
+
+url = "https://gutendex.com/books/?sort=popular"
+while url:
+    data = json.loads(http_get(url))
+    # ... process data['results'] ...
+    url = data['next']
+    if url:
+        time.sleep(0.5)   # be respectful — no published rate limit but timeouts observed
+```
+
+For gutenberg.org file downloads (txt, epub, etc.) there is no documented rate limit but Gutenberg asks not to use automated bulk downloading; use their [offline catalogs](https://www.gutenberg.org/ebooks/offline_catalogs.html) for bulk access.
+
+## Gotchas
+
+- **`.opf` 404**: `https://www.gutenberg.org/cache/epub/1342/pg1342.opf` returns 404. Use `.rdf` instead — same path prefix, same data in RDF/XML.
+- **`formats` URLs redirect**: URLs like `https://www.gutenberg.org/ebooks/1342.txt.utf-8` are redirect endpoints that resolve to `/cache/epub/1342/pg1342.txt`. Either form works with `http_get` (urllib follows redirects automatically), but the `/cache/epub/` direct URL avoids an extra round trip.
+- **Two text files**: `/files/1342/1342-0.txt` (older Project Gutenberg edition, 729 KB) and `/cache/epub/1342/pg1342.txt` (modern edition, 763 KB) contain different versions of the same book. The Gutendex `formats` entry always points to the cache/modern version.
+- **Boilerplate**: Every `.txt` file opens with a PG licence header and closes with a footer. Strip with `START`/`END` markers (see "Read the plain text" section above).
+- **`summaries` field is AI-generated**: The `summaries` array in Gutendex responses contains automatically generated summaries, not the author's original blurb.
+- **`copyright: false`** means public domain in the USA. Non-US copyright status is not tracked.
+- **`page_size` ignored**: Passing `?page_size=5` to Gutendex has no effect — always returns 32 results.
+- **Gutendex `sort=ascending/descending`** sorts by ID (oldest/newest book in the catalog), not by title or author name.
+- **Catalog search `author:` prefix**: `?query=author:dickens` searches within author names but Gutenberg's relevance ranking is fuzzy and can return unexpected results. For precise author lookup use Gutendex `?search=charles+dickens`.
+- **OPDS pagination**: Only 7 entries per page (1 metadata + 6 books). Slow for bulk extraction — use Gutendex instead.
+- **HTML catalog `start_index`**: Pagination is 25 per page. Next page = `start_index=26`, then `51`, `76`, etc. The value appears in the rendered HTML (`re.findall(r'start_index=(\d+)', html)` returns the next page's value).