catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/open-library/scraping.md

@@ -0,0 +1,472 @@
+# Open Library — Book Data Extraction
+
+`https://openlibrary.org` — Internet Archive's free book catalog. All endpoints are public JSON APIs — no auth, no browser, no scraping required.
+
+## Do this first
+
+**Every task is a direct HTTP call — never open the browser.**
+
+```python
+import json
+from helpers import http_get
+
+# Search by title
+results = json.loads(http_get("https://openlibrary.org/search.json?q=dune&limit=5"))
+# results['numFound']  == 49090
+# results['docs']      == list of work objects
+# results['start']     == 0  (offset for pagination)
+```
+
+The search API is your entry point for everything. It returns work-level records (all editions grouped). To get edition details, follow the `key` to the Works or Books API.
+
+---
+
+## Common workflows
+
+### Search by query, author, title, or ISBN
+
+```python
+import json
+from helpers import http_get
+
+# Free-text search
+r = json.loads(http_get("https://openlibrary.org/search.json?q=dune+frank+herbert&limit=5"))
+
+# Author search
+r = json.loads(http_get(
+    "https://openlibrary.org/search.json?author=tolkien&limit=5"
+    "&fields=title,author_name,first_publish_year,isbn"
+))
+# fields=* returns all available fields; default returns ~15
+
+# Title + author combined
+r = json.loads(http_get(
+    "https://openlibrary.org/search.json?title=dune&author=frank+herbert&limit=3"
+    "&fields=title,author_name,edition_count,first_publish_year"
+))
+# r['docs'][0]['title']              == 'Dune'
+# r['docs'][0]['author_name']        == ['Frank Herbert']
+# r['docs'][0]['first_publish_year'] == 1965
+# r['docs'][0]['edition_count']      == 120
+
+# ISBN lookup (returns 0–2 results for the same work)
+r = json.loads(http_get("https://openlibrary.org/search.json?isbn=9780743273565"))
+# r['numFound']            == 2
+# r['docs'][0]['title']    == 'The Great Gatsby'
+# r['docs'][0]['key']      == '/works/OL468431W'
+```
+
+**Sort options** (`&sort=`): `new` (recently added), `old`, `random`, `editions` (most editions), `scans` (most scans). Default is relevance.
+
+**Language filter**: `&language=fre` (ISO 639-2/B codes: `eng`, `fre`, `ger`, `spa`, `ita`, etc.)
+
+**Pagination**: `&limit=N&offset=N`. Max limit not enforced but keep under 100 for reliability.
+
+#### Search doc fields (default — ~15 keys always present)
+
+| Field | Type | Notes |
+|---|---|---|
+| `key` | str | `/works/OL893415W` — use for Works API |
+| `title` | str | Work title |
+| `author_name` | list[str] | e.g. `['Frank Herbert']` |
+| `author_key` | list[str] | e.g. `['OL79034A']` |
+| `first_publish_year` | int | |
+| `edition_count` | int | Number of editions across all languages |
+| `cover_i` | int | Cover image ID — use with covers API |
+| `cover_edition_key` | str | e.g. `OL7353617M` |
+| `language` | list[str] | ISO codes of all editions |
+| `ia` | list[str] | Internet Archive identifiers (when has_fulltext=true) |
+| `ebook_access` | str | `'public'`, `'borrowable'`, `'no_ebook'` |
+| `has_fulltext` | bool | |
+
+#### Extra fields with `&fields=*`
+
+```python
+# With fields=* you also get:
+# 'isbn'                   list[str]   All ISBNs across editions
+# 'publisher'              list[str]   All publishers ever
+# 'publish_date'           list[str]   All publish dates (strings, inconsistent formats)
+# 'publish_year'           list[int]   Parsed years
+# 'subject'                list[str]   Subject headings
+# 'person'                 list[str]   Subject persons (e.g. 'Big Brother')
+# 'place'                  list[str]   Subject places
+# 'time'                   list[str]   Subject times
+# 'number_of_pages_median' int         Median page count across editions
+# 'ratings_average'        float       e.g. 4.29
+# 'ratings_count'          int
+# 'want_to_read_count'     int
+# 'already_read_count'     int
+# 'currently_reading_count'int
+# 'readinglog_count'       int         Total of all reading log entries
+# 'first_sentence'         list[str]
+# 'id_goodreads'           list[str]
+# 'id_librarything'        list[str]
+# 'id_wikidata'            list[str]
+# 'ddc'                    list[str]   Dewey Decimal Classification
+# 'lcc'                    list[str]   Library of Congress Classification
+# 'lccn'                   list[str]
+```
+
+---
+
+### Bulk ISBN lookups (parallel)
+
+```python
+import json
+from helpers import http_get
+from concurrent.futures import ThreadPoolExecutor
+
+isbns = ['9780743273565', '9780451524935', '9780618346257']
+
+def lookup_isbn(isbn):
+    url = f"https://openlibrary.org/search.json?isbn={isbn}&fields=title,author_name,first_publish_year,key"
+    r = json.loads(http_get(url))
+    if r['docs']:
+        d = r['docs'][0]
+        return {'isbn': isbn, 'title': d.get('title'), 'author': d.get('author_name', [None])[0],
+                'year': d.get('first_publish_year'), 'key': d.get('key')}
+    return {'isbn': isbn, 'found': False}
+
+with ThreadPoolExecutor(max_workers=5) as ex:
+    books = list(ex.map(lookup_isbn, isbns))
+
+# [{'isbn': '9780743273565', 'title': 'The Great Gatsby', 'author': 'F. Scott Fitzgerald', 'year': 1920, ...},
+#  {'isbn': '9780451524935', 'title': 'Nineteen Eighty-Four', 'author': 'George Orwell',      'year': 1949, ...},
+#  {'isbn': '9780618346257', 'title': 'The Fellowship of the Ring', 'author': 'J.R.R. Tolkien', 'year': 1954, ...}]
+```
+
+---
+
+### Works API (editions grouped by title)
+
+Returns all metadata for a work (all editions combined). Get the work ID from `key` in search results.
+
+```python
+import json
+from helpers import http_get
+
+work_id = 'OL893415W'  # from search doc['key'] = '/works/OL893415W'
+work = json.loads(http_get(f"https://openlibrary.org/works/{work_id}.json"))
+
+# work['title']           == 'Dune'
+# work['key']             == '/works/OL893415W'
+# work['covers']          == [11481354, 12375564, 11157826]  ← cover IDs for covers API
+# work['subjects']        == ['Dune (Imaginary place)', 'Fiction', ...]
+# work['subject_places']  == [...]   ← geographic subjects (may be absent)
+# work['subject_people']  == [...]   ← person subjects (may be absent)
+# work['subject_times']   == [...]   ← time subjects (may be absent)
+# work['authors']         == [{'author': {'key': '/authors/OL79034A'}, 'type': {...}}]
+# work['description']     → either str OR {'type': '/type/text', 'value': str}  ← see gotchas
+# work['created']         == {'type': '/type/datetime', 'value': '2009-10-15T11:34:21.437031'}
+# work['last_modified']   same shape as created
+```
+
+Helper for the description field (which has two possible shapes):
+
+```python
+def get_description(work: dict) -> str:
+    desc = work.get('description', '')
+    if isinstance(desc, dict):
+        return desc.get('value', '')
+    return desc or ''
+```
+
+#### Works editions (paginated list of all editions)
+
+```python
+editions_resp = json.loads(http_get(
+    f"https://openlibrary.org/works/{work_id}/editions.json?limit=10&offset=0"
+))
+# editions_resp['size']    == 120      (total edition count)
+# editions_resp['entries'] == [...]    (up to limit items)
+# editions_resp['links']   == {'self': '...', 'work': '...', 'next': '...', 'prev': '...'}
+# ← use links['next'] for pagination when offset+limit < size
+
+e = editions_resp['entries'][0]
+# e['title']           == 'Duna'
+# e['publishers']      == ['Editora Aleph']
+# e['publish_date']    == '19/08/2017'   ← inconsistent format, string
+# e['isbn_13']         == ['9788576573135']
+# e['isbn_10']         == ['857657313X']
+# e['covers']          == [10368109]
+# e['number_of_pages'] == 680
+# e['languages']       == [{'key': '/languages/por'}]
+# e['key']             == '/books/OL28969075M'
+# e['physical_format'] == 'Paperback'   (often missing)
+# e['notes']           → str or {'value': str}  (often missing)
+```
+
+---
+
+### Books API (specific edition)
+
+Two sub-APIs: direct JSON for raw data, or `api/books` for enriched data.
+
+#### Direct edition JSON
+
+```python
+import json
+from helpers import http_get
+
+edition_id = 'OL7353617M'  # from editions list e['key'] or cover_edition_key in search
+edition = json.loads(http_get(f"https://openlibrary.org/books/{edition_id}.json"))
+
+# edition['title']           == 'Fantastic Mr. Fox'
+# edition['publishers']      == ['Puffin']
+# edition['publish_date']    == 'October 1, 1988'
+# edition['isbn_13']         == ['9780140328721']
+# edition['isbn_10']         == ['0140328726']
+# edition['number_of_pages'] == 96
+# edition['covers']          == [...]   ← cover IDs
+# edition['languages']       == [{'key': '/languages/eng'}]
+# edition['works']           == [{'key': '/works/OL45804W'}]
+# edition['authors']         == [{'key': '/authors/OL34184A'}]
+# edition['identifiers']     == {'goodreads': [...], 'librarything': [...]}
+# edition['first_sentence']  == {'value': '...'} or str  (often missing)
+# edition['ocaid']           == 'fantast00dahl'  ← Internet Archive ID (if available)
+```
+
+#### Bibkeys API (enriched, multiple books at once)
+
+```python
+# jscmd=data: cleaned up dict with cover URLs pre-built
+r = json.loads(http_get(
+    "https://openlibrary.org/api/books"
+    "?bibkeys=ISBN:9780743273565,ISBN:9780451524935"
+    "&format=json&jscmd=data"
+))
+# r == {'ISBN:9780743273565': {...}, 'ISBN:9780451524935': {...}}
+
+book = r['ISBN:9780743273565']
+# book['title']           == 'The Great Gatsby'
+# book['authors']         == [{'url': '...', 'name': 'F. Scott Fitzgerald'}]
+# book['publish_date']    == '2021'
+# book['publishers']      == [{'name': 'Independently Published'}]
+# book['number_of_pages'] == 208
+# book['url']             == 'http://openlibrary.org/books/OL46773254M/The_Great_Gatsby'
+# book['key']             == '/books/OL46773254M'
+# book['cover']           == {'small': '...S.jpg', 'medium': '...M.jpg', 'large': '...L.jpg'}
+# book['identifiers']     == {'isbn_13': [...], 'openlibrary': [...]}
+# book['subjects']        == [{'name': 'Modern fiction', 'url': '...'}, ...]
+# book['subject_places']  == None  ← often null even with jscmd=data
+
+# jscmd=details: raw edition JSON + extra fields
+r2 = json.loads(http_get(
+    "https://openlibrary.org/api/books"
+    "?bibkeys=ISBN:9780743273565&format=json&jscmd=details"
+))
+item = r2['ISBN:9780743273565']
+# item['bib_key']      == 'ISBN:9780743273565'
+# item['info_url']     == 'http://openlibrary.org/books/OL...'
+# item['preview']      == 'noview' | 'restricted' | 'full'
+# item['preview_url']  == URL to read on OL or IA
+# item['thumbnail_url']== 'https://covers.openlibrary.org/b/id/14314120-S.jpg'
+# item['details']      → raw edition JSON (same as /books/OL...M.json)
+
+# Supported bibkey prefixes: ISBN:, OCLC:, LCCN:, OLID: (e.g. OLID:OL46773254M)
+```
+
+---
+
+### Authors API
+
+```python
+import json
+from helpers import http_get
+
+# Lookup by known author key
+author = json.loads(http_get("https://openlibrary.org/authors/OL26320A.json"))
+# OL26320A is J.R.R. Tolkien (note: not Frank Herbert as originally stated — verify with search)
+
+# author['name']           == 'J.R.R. Tolkien'
+# author['fuller_name']    == 'John Ronald Reuel Tolkien'
+# author['personal_name']  == 'J. R. R. Tolkien'
+# author['birth_date']     == '3 January 1892'   ← string, not parsed
+# author['death_date']     == '2 September 1973'
+# author['bio']            → str or {'type': '/type/text', 'value': str}
+# author['photos']         == [6155606, 6433524, ...]   ← photo IDs for covers API
+# author['links']          == [{'title': '...', 'url': '...', 'type': {...}}, ...]
+# author['remote_ids']     == {'wikidata': 'Q892', 'viaf': '95218067', ...}
+# author['alternate_names']== ['J. R. R. Tolkien', 'TOLKIEN', ...]
+# author['key']            == '/authors/OL26320A'
+# author['wikipedia']      → URL string or None
+
+# Author works (paginated)
+works = json.loads(http_get("https://openlibrary.org/authors/OL26320A/works.json?limit=5"))
+# works['size']    == 415
+# works['entries'] == [{title, key, covers, authors, created, ...}, ...]
+# works['links']   == {'self': '...', 'next': '...'}
+```
+
+#### Author search
+
+```python
+r = json.loads(http_get("https://openlibrary.org/search/authors.json?q=tolkien"))
+# r['numFound'] == 40
+# r['docs'][0]:
+#   'name'                   == 'Christopher Tolkien'
+#   'key'                    == 'OL2623360A'     ← NOTE: no /authors/ prefix here
+#   'birth_date'             == '21 November 1924'
+#   'death_date'             == '16 January 2020'
+#   'top_work'               == 'The War of the Ring'
+#   'work_count'             == 43
+#   'top_subjects'           == [...]
+#   'alternate_names'        == [...]
+#   'ratings_average'        float
+#   'want_to_read_count'     int
+#   'already_read_count'     int
+#   'currently_reading_count'int
+```
+
+---
+
+### Cover images
+
+Covers are served directly as JPEG — redirect to Internet Archive CDN. No auth needed.
+
+```
+# Book covers — three key types:
+https://covers.openlibrary.org/b/id/{cover_id}-{size}.jpg      # by cover ID (most reliable)
+https://covers.openlibrary.org/b/isbn/{isbn}-{size}.jpg        # by ISBN
+https://covers.openlibrary.org/b/olid/{edition_id}-{size}.jpg  # by edition OLID (unreliable — see gotchas)
+
+# Author photos:
+https://covers.openlibrary.org/a/id/{photo_id}-{size}.jpg
+
+# Sizes: S (small), M (medium), L (large)
+```
+
+```python
+import urllib.request
+
+def get_cover_bytes(cover_id: int, size: str = 'M') -> bytes | None:
+    """Fetch cover image bytes. Returns None if no cover (43-byte GIF placeholder)."""
+    url = f"https://covers.openlibrary.org/b/id/{cover_id}-{size}.jpg"
+    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=15) as resp:
+        data = resp.read()
+    return None if len(data) == 43 else data   # 43-byte GIF = no cover placeholder
+
+# Or just get the URL for embedding:
+def cover_url(cover_id: int, size: str = 'M') -> str:
+    return f"https://covers.openlibrary.org/b/id/{cover_id}-{size}.jpg"
+
+# Usage:
+from helpers import http_get
+import json
+work = json.loads(http_get("https://openlibrary.org/works/OL893415W.json"))
+if work.get('covers'):
+    img = get_cover_bytes(work['covers'][0], 'L')   # first cover, large
+    # img is ~20–80KB JPEG bytes, redirected from ia*.archive.org
+```
+
+To get cover by ISBN directly (e.g. for UI without a full book lookup):
+```python
+# Medium-size cover by ISBN:
+url = f"https://covers.openlibrary.org/b/isbn/{isbn}-M.jpg"
+# Redirects to Internet Archive CDN, content-type: image/jpeg
+# Use ?default=false to get 404 instead of 1×1 GIF placeholder for missing covers
+url_safe = f"https://covers.openlibrary.org/b/isbn/{isbn}-M.jpg?default=false"
+```
+
+---
+
+### Subjects API
+
+```python
+import json
+from helpers import http_get
+
+# Subject slugs: lowercase, underscores for spaces
+r = json.loads(http_get("https://openlibrary.org/subjects/science_fiction.json?limit=5"))
+# r['name']          == 'science fiction'
+# r['subject_type']  == 'subject'   ← also: 'person', 'place', 'time'
+# r['work_count']    == 20973
+# r['works']         == [{title, key, cover_id, authors, edition_count, ...}, ...]
+
+w = r['works'][0]
+# w['title']           == 'Alice\'s Adventures in Wonderland'
+# w['key']             == '/works/OL138052W'
+# w['cover_id']        == 10527843
+# w['cover_edition_key']== 'OL...'
+# w['authors']         == [{'key': '/authors/OL22098A', 'name': 'Lewis Carroll'}]
+# w['edition_count']   == 3546
+# w['first_publish_year']== ...
+# w['has_fulltext']    == True | False
+# w['ia']              == 'identifier'   (Internet Archive ID when available)
+
+# Pagination: &offset=N
+# Place subject:
+r2 = json.loads(http_get("https://openlibrary.org/subjects/place:london.json?limit=5"))
+# r2['subject_type'] == 'place', r2['work_count'] == 23927
+
+# Person subject:
+# https://openlibrary.org/subjects/person:napoleon.json?limit=5
+# Time subject:
+# https://openlibrary.org/subjects/time:middle_ages.json?limit=5
+
+# Combine with ebooks=true to filter to only freely readable books:
+r3 = json.loads(http_get("https://openlibrary.org/subjects/science_fiction.json?limit=5&ebooks=true"))
+# r3['works'][i]['has_fulltext'] == True for all results
+```
+
+---
+
+### Trending books
+
+```python
+import json
+from helpers import http_get
+
+for period in ['daily', 'weekly', 'monthly']:
+    r = json.loads(http_get(f"https://openlibrary.org/trending/{period}.json?limit=10"))
+    # r['works']  == list of search-doc-style objects
+    # r['days']   == int (time window)
+    # r['hours']  == int
+    # Same fields as search docs (title, author_name, cover_i, key, ...)
+    print(period, r['works'][0]['title'])  # e.g. 'Atomic Habits'
+```
+
+---
+
+## Rate limits
+
+No authentication required. No API key. No explicit rate limit published.
+
+Observed in testing: 5 requests completed in ~1 second with no throttling, no 429s. The API is served from CDN/Solr — in practice you can make 10–20 parallel requests without issue. For bulk operations (hundreds of ISBNs), use `ThreadPoolExecutor(max_workers=5)` to be a good citizen.
+
+**No `User-Agent` override needed** — the default `Mozilla/5.0` from `http_get` is accepted by all Open Library endpoints (unlike Nominatim which blocks it).
+
+---
+
+## Gotchas
+
+**`description` field has two shapes.** Both are real — check at runtime:
+```python
+desc = work.get('description', '')
+text = desc.get('value', '') if isinstance(desc, dict) else (desc or '')
+```
+
+**`/works/OL45804W` is Fantastic Mr. Fox, not Dune.** The OL IDs in the original prompt were placeholders. Always resolve real IDs via the search API rather than hardcoding them.
+
+**Author search `key` has no prefix.** `/search/authors.json` returns `key: 'OL26320A'`, but the Authors API and all other APIs use `/authors/OL26320A`. Add the prefix manually when constructing follow-up URLs.
+
+**Missing cover → 43-byte GIF placeholder, not 404.** Without `?default=false`, the covers API returns a 1×1 transparent GIF instead of HTTP 404 for unknown IDs. Check `len(data) == 43` to detect missing covers.
+
+**`covers.openlibrary.org/b/olid/{work_id}` is unreliable.** OLID-based cover URLs for work IDs (OL...W) return the placeholder even when covers exist. Always use `b/id/{cover_id}` (from `work['covers'][0]`) or `b/isbn/{isbn}` instead.
+
+**Bibkeys API picks one edition per ISBN.** When the same ISBN appears on multiple editions (reprint, reissue), `api/books?bibkeys=ISBN:...` returns one — and it may not be the most common edition.
+
+**`publish_date` is a raw string.** Values like `'October 1, 1988'`, `'19/08/2017'`, `'2021'`, and `'1965-01-01'` all appear. Don't parse without normalization.
+
+**`/works/.../editions.json` pagination uses `links.next`.** Unlike search (which uses `offset=`), check `links['next']` in the response to know if more pages exist:
+```python
+resp = json.loads(http_get("https://openlibrary.org/works/OL893415W/editions.json?limit=50"))
+while 'next' in resp.get('links', {}):
+    resp = json.loads(http_get("https://openlibrary.org" + resp['links']['next']))
+    # process resp['entries']
+```
+
+**404 for non-existent IDs.** `/works/OL99999999W.json`, `/books/OL99999999M.json`, and `/authors/OL99999999A.json` all raise `HTTPError: HTTP Error 404: Not Found`. Wrap in try/except.
+
+**Search `docs` default fields are minimal.** The default response includes ~15 fields. Add `&fields=*` to get all 100+ Solr fields (ratings, ISBNs, publishers, subjects, Goodreads IDs, etc.). Alternatively specify exactly what you need: `&fields=title,isbn,ratings_average`.