catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/crossref/scraping.md

@@ -0,0 +1,568 @@
+# CrossRef — Scraping & Data Extraction
+
+`https://api.crossref.org` — scholarly DOI and citation metadata. **Never use the browser for CrossRef.** Completely free, no auth required. All workflows use `http_get`.
+
+## Do this first
+
+**Always add `mailto=your@email.com` to every request** — it moves you into the polite pool, which doubles the rate limit and concurrency allowance. The difference is measurable and the cost is zero.
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"  # set once, append to every URL
+
+# Single DOI lookup — fastest way to get metadata for a known paper
+data = json.loads(http_get(f"https://api.crossref.org/works/10.1038/s41586-021-03819-2?{MAILTO}"))
+msg = data['message']
+# msg keys: DOI, title, author, published, type, container-title, volume, issue,
+#           page, is-referenced-by-count, references-count, abstract (optional), ...
+```
+
+## Common workflows
+
+### DOI lookup — single paper
+
+```python
+from helpers import http_get
+import json, re
+
+MAILTO = "mailto=your@email.com"
+
+def fetch_work(doi):
+    data = json.loads(http_get(f"https://api.crossref.org/works/{doi}?{MAILTO}"))
+    return data['message']
+
+def parse_date(d):
+    """[[2021, 7, 15]] -> '2021-7-15'. Handles partial dates like [[2021]]."""
+    if not d: return None
+    parts = d.get('date-parts', [[]])[0]
+    return '-'.join(str(p) for p in parts if p is not None)
+
+def clean_abstract(raw):
+    """Strip JATS XML tags. Abstract field contains tags like <jats:p>, <jats:italic>."""
+    return re.sub(r'<[^>]+>', ' ', raw).strip() if raw else None
+
+w = fetch_work("10.1038/s41586-021-03819-2")  # AlphaFold2
+
+print("DOI:", w['DOI'])                                    # 10.1038/s41586-021-03819-2
+print("Title:", w['title'][0])                             # Highly accurate protein structure...
+print("Type:", w['type'])                                  # journal-article
+print("Publisher:", w['publisher'])                        # Springer Science and Business Media LLC
+print("Journal:", w.get('container-title', [''])[0])      # Nature
+print("Volume:", w.get('volume'))                          # 596
+print("Issue:", w.get('issue'))                            # 7873
+print("Page:", w.get('page'))                              # 583-589
+print("published:", parse_date(w.get('published')))        # 2021-7-15  (online date)
+print("published-online:", parse_date(w.get('published-online')))  # 2021-7-15
+print("published-print:", parse_date(w.get('published-print')))    # 2021-8-26
+print("Citations:", w.get('is-referenced-by-count'))       # 40260
+print("References:", w.get('references-count'))            # 84
+print("Abstract:", clean_abstract(w.get('abstract', ''))[:100] if w.get('abstract') else None)
+# Confirmed output (2026-04-18):
+# DOI: 10.1038/s41586-021-03819-2
+# Title: Highly accurate protein structure prediction with AlphaFold
+# Type: journal-article
+# Journal: Nature
+# Volume: 596 | Issue: 7873 | Page: 583-589
+# published: 2021-7-15 | published-print: 2021-8-26
+# Citations: 40260
+```
+
+### DOI lookup — extract authors with ORCID
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+data = json.loads(http_get(f"https://api.crossref.org/works/10.1038/s41586-021-03819-2?{MAILTO}"))
+authors = data['message'].get('author', [])
+
+for a in authors[:3]:
+    name = f"{a.get('given', '')} {a.get('family', '')}".strip()
+    # ORCID is a full URL, not a bare ID — strip the prefix
+    orcid_url = a.get('ORCID')  # e.g. 'https://orcid.org/0000-0001-6169-6580'
+    orcid_id = orcid_url.replace('https://orcid.org/', '') if orcid_url else None
+    authenticated = a.get('authenticated-orcid', False)  # False = self-reported, True = verified
+    affiliations = [aff.get('name', '') for aff in a.get('affiliation', [])]
+    print(f"{name} | ORCID: {orcid_id} | auth={authenticated} | seq={a['sequence']}")
+# Confirmed output:
+# John Jumper | ORCID: 0000-0001-6169-6580 | auth=False | seq=first
+# Richard Evans | ORCID: None | auth=False | seq=additional
+# Alexander Pritzel | ORCID: None | auth=False | seq=additional
+```
+
+### Batch DOI lookup (parallel — 5 calls in ~0.3s)
+
+```python
+from helpers import http_get
+from concurrent.futures import ThreadPoolExecutor
+import json
+
+MAILTO = "mailto=your@email.com"
+
+def fetch_work(doi):
+    try:
+        data = json.loads(http_get(f"https://api.crossref.org/works/{doi}?{MAILTO}"))
+        msg = data['message']
+        return {
+            'doi': doi,
+            'title': msg.get('title', [''])[0],
+            'year': (msg.get('published', {}).get('date-parts') or [[None]])[0][0],
+            'citations': msg.get('is-referenced-by-count'),
+            'type': msg.get('type'),
+        }
+    except Exception as e:
+        return {'doi': doi, 'error': str(e)}
+
+dois = [
+    "10.1038/nature12345",
+    "10.1038/s41586-021-03819-2",
+    "10.1056/NEJMoa2034577",
+    "10.1126/science.1260419",
+    "10.1038/s41586-024-07487-w",
+]
+
+# max_workers=5 safe; polite pool: 10 req/s, concurrency=3 (see Rate limits)
+with ThreadPoolExecutor(max_workers=5) as ex:
+    results = list(ex.map(fetch_work, dois))
+
+for r in results:
+    print(r['year'], f"cites={r['citations']}", r['title'][:50])
+# Confirmed output (2026-04-18, ~0.296s total):
+# 2013 cites=465 LRG1 promotes angiogenesis by modulating endotheli
+# 2021 cites=40260 Highly accurate protein structure prediction with
+# 2020 cites=13752 Safety and Efficacy of the BNT162b2 mRNA Covid-19
+# 2015 cites=13553 Tissue-based map of the human proteome
+# 2024 cites=12037 Accurate structure prediction of biomolecular inte
+```
+
+### Search works by keyword
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+# Broad keyword search
+data = json.loads(http_get(
+    f"https://api.crossref.org/works?query=machine+learning&rows=5&{MAILTO}"
+))
+msg = data['message']
+print("Total results:", msg['total-results'])   # 2,805,391
+for item in msg['items']:
+    title = item.get('title', ['(no title)'])[0][:60]
+    doi   = item.get('DOI', '')
+    year  = (item.get('published', {}).get('date-parts') or [[None]])[0][0]
+    type_ = item.get('type', '')
+    print(f"  [{type_}] {year} {title}")
+    print(f"    DOI: {doi}")
+```
+
+### Search by author + title (targeted)
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+data = json.loads(http_get(
+    f"https://api.crossref.org/works?query.author=Lecun&query.title=deep+learning&rows=5&{MAILTO}"
+))
+msg = data['message']
+print("Total results:", msg['total-results'])   # 62
+for item in msg['items'][:3]:
+    title   = item.get('title', [''])[0][:60]
+    authors = ', '.join(a.get('family', '') for a in item.get('author', [])[:2])
+    year    = (item.get('published', {}).get('date-parts') or [[None]])[0][0]
+    print(f"  {year} {title}")
+    print(f"    Authors: {authors}  DOI: {item.get('DOI')}")
+# Confirmed output:
+# 2015 Deep learning & convolutional networks
+#   Authors: LeCun  DOI: 10.1109/hotchips.2015.7477328
+```
+
+### Filter by date, type, and sort by citations
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+data = json.loads(http_get(
+    f"https://api.crossref.org/works"
+    f"?filter=from-pub-date:2024-01-01,type:journal-article"
+    f"&rows=5&sort=is-referenced-by-count&order=desc&{MAILTO}"
+))
+msg = data['message']
+print("Total 2024+ journal articles:", msg['total-results'])   # 14,565,456
+for item in msg['items'][:3]:
+    title  = item.get('title', [''])[0][:60]
+    cites  = item.get('is-referenced-by-count', 0)
+    year   = (item.get('published', {}).get('date-parts') or [[None]])[0][0]
+    print(f"  {year} cites={cites} {title}")
+# Confirmed output:
+# 2024 cites=17371 Global cancer statistics 2022: GLOBOCAN estimates...
+# 2024 cites=12037 Accurate structure prediction of biomolecular int...
+```
+
+### Filter with `has-abstract:true`
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+# Only return works that have an abstract (useful since ~30-70% do not)
+data = json.loads(http_get(
+    f"https://api.crossref.org/works"
+    f"?filter=from-pub-date:2023-01-01,until-pub-date:2023-12-31"
+    f",type:journal-article,has-abstract:true"
+    f"&rows=3&sort=is-referenced-by-count&order=desc&{MAILTO}"
+))
+msg = data['message']
+print("2023 journal articles with abstract:", msg['total-results'])   # 3,041,841
+for item in msg['items']:
+    print(item.get('title', [''])[0][:60], '| cites:', item.get('is-referenced-by-count'))
+# Confirmed output:
+# Cancer statistics, 2023 | cites: 12919
+# Evolutionary-scale prediction of atomic-level protein struct | cites: 4352
+```
+
+### Cursor pagination (large result sets)
+
+Standard offset pagination (`start=`) caps at a few thousand results. Use cursor for full sweeps.
+
+```python
+from helpers import http_get
+from urllib.parse import quote
+import json
+
+MAILTO = "mailto=your@email.com"
+
+# First page: cursor=*
+data = json.loads(http_get(
+    f"https://api.crossref.org/works?query=covid&rows=100&cursor=*&{MAILTO}"
+))
+msg = data['message']
+print("Total results:", msg['total-results'])   # 897,660
+items = msg['items']
+next_cursor = msg['next-cursor']   # base64 string like "DnF1ZXJ5VGhlbkZldGNoJA..."
+
+# Next pages: pass URL-encoded cursor
+while next_cursor and items:
+    data = json.loads(http_get(
+        f"https://api.crossref.org/works?query=covid&rows=100"
+        f"&cursor={quote(next_cursor)}&{MAILTO}"
+    ))
+    msg = data['message']
+    items = msg.get('items', [])
+    next_cursor = msg.get('next-cursor')
+    # process items...
+    break  # remove for full sweep
+```
+
+### Fetch specific fields only (`select=`)
+
+Reduces response size significantly for bulk operations:
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+data = json.loads(http_get(
+    f"https://api.crossref.org/works?query=cancer&rows=5"
+    f"&select=DOI,title,author&{MAILTO}"
+))
+# Warning: if a field is absent for a record, it simply won't appear in that item
+for item in data['message']['items']:
+    print(list(item.keys()))   # only ['DOI', 'title'] or ['DOI', 'title', 'author']
+    # Note: select= does NOT guarantee the field appears — absent fields are just omitted
+```
+
+### Count by type using facets
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+data = json.loads(http_get(
+    f"https://api.crossref.org/works?query=machine+learning&rows=0"
+    f"&facet=type-name:*&{MAILTO}"
+))
+msg = data['message']
+type_facet = msg['facets']['type-name']
+for k, v in sorted(type_facet['values'].items(), key=lambda x: -x[1]):
+    print(f"  {k}: {v:,}")
+# Confirmed output (all CrossRef, 2026-04-18):
+# Journal Article: 1,628,997 (for query=machine+learning scope)
+# Conference Paper: 501,433
+# Chapter: 455,907
+# Posted Content: 87,937
+# ...
+```
+
+### Journal info by ISSN
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+# Nature (ISSN 0028-0836)
+data = json.loads(http_get(f"https://api.crossref.org/journals/0028-0836?{MAILTO}"))
+msg = data['message']
+print("Title:", msg['title'])                          # Nature
+print("Publisher:", msg['publisher'])                  # Springer Science and Business Media LLC
+print("ISSN:", msg['ISSN'])                            # ['0028-0836', '1476-4687']
+print("Total DOIs:", msg['counts']['total-dois'])       # 445,417
+print("Subjects:", msg.get('subjects', []))             # [] (not always populated)
+
+# Search journals by name
+data2 = json.loads(http_get(f"https://api.crossref.org/journals?query=nature&rows=3&{MAILTO}"))
+for j in data2['message']['items']:
+    print(f"{j.get('title')} | ISSN: {j.get('ISSN')} | DOIs: {j.get('counts', {}).get('total-dois')}")
+# Confirmed output:
+# NatureJobs | ISSN: [] | DOIs: 0
+# Naturen | ISSN: ['0028-0887', '1504-3118'] | DOIs: 1055
+```
+
+### Funder search
+
+```python
+from helpers import http_get
+import json
+
+MAILTO = "mailto=your@email.com"
+
+data = json.loads(http_get(
+    f"https://api.crossref.org/funders?query=national+science+foundation&rows=3&{MAILTO}"
+))
+msg = data['message']
+print("Total funders:", msg['total-results'])   # 108
+for f in msg['items']:
+    print(f"  ID: {f['id']} | {f['name']}")
+    print(f"    Alt names: {f.get('alt-names', [])[:2]}")
+    print(f"    URI: {f.get('uri')}")
+# Confirmed output:
+# ID: 501100001711 | Schweizerischer Nationalfonds zur Förderung...
+# ID: 100000143 | Division of Computing and Communication Foundations
+```
+
+### DOI content negotiation (alternative, no CrossRef API needed)
+
+The `doi.org` resolver can return formatted metadata directly via `Accept` header:
+
+```python
+import urllib.request, json
+
+def doi_to_csl(doi):
+    """Fetch CSL-JSON via DOI content negotiation. Same data as CrossRef API."""
+    req = urllib.request.Request(
+        f"https://doi.org/{doi}",
+        headers={"Accept": "application/vnd.citationstyles.csl+json",
+                 "User-Agent": "Mozilla/5.0"}
+    )
+    with urllib.request.urlopen(req, timeout=20) as r:
+        return json.loads(r.read().decode())
+
+def doi_to_bibtex(doi):
+    """Fetch BibTeX via DOI content negotiation."""
+    req = urllib.request.Request(
+        f"https://doi.org/{doi}",
+        headers={"Accept": "application/x-bibtex", "User-Agent": "Mozilla/5.0"}
+    )
+    with urllib.request.urlopen(req, timeout=20) as r:
+        return r.read().decode()
+
+csl = doi_to_csl("10.1038/nature12345")
+print("Title:", csl['title'])   # LRG1 promotes angiogenesis...
+print("Type:", csl['type'])     # journal-article
+
+bib = doi_to_bibtex("10.1038/nature12345")
+print(bib[:200])
+# @article{Wang_2013, title={LRG1 promotes angiogenesis...
+```
+
+## Field reference
+
+### Work object — complete field list
+
+All fields are potentially absent unless marked required. Fields marked (R) are always present.
+
+| Field | Type | Notes |
+|---|---|---|
+| `DOI` (R) | string | e.g. `"10.1038/s41586-021-03819-2"` |
+| `URL` (R) | string | `"https://doi.org/10.1038/s41586-021-03819-2"` |
+| `title` (R) | list[str] | Always a list; access `title[0]` |
+| `type` (R) | string | e.g. `"journal-article"` — see type table below |
+| `publisher` | string | |
+| `container-title` | list[str] | Journal name; access `[0]` |
+| `short-container-title` | list[str] | Abbreviated journal name |
+| `ISSN` | list[str] | May contain print and online ISSN |
+| `volume` | string | Note: string not int (`"596"`) |
+| `issue` | string | |
+| `page` | string | e.g. `"583-589"` |
+| `author` | list[object] | See author fields below |
+| `published` | date-object | Best single date — use this |
+| `published-online` | date-object | Online-first date |
+| `published-print` | date-object | Print edition date |
+| `issued` | date-object | Usually same as `published` |
+| `is-referenced-by-count` | int | Inbound citations to this work |
+| `references-count` | int | Outbound references from this work |
+| `reference` | list[object] | Full reference list (when deposited) |
+| `abstract` | string | JATS XML markup; ~30-70% of works; strip tags before use |
+| `subject` | list[str] | Subject classification (often empty) |
+| `language` | string | e.g. `"en"` |
+| `license` | list[object] | Each: `{URL, start, delay-in-days, content-version}` |
+| `funder` | list[object] | Each: `{name, DOI, award}` |
+| `link` | list[object] | Full-text links |
+| `relation` | object | Related DOIs (e.g. preprint → article) |
+| `assertion` | list[object] | Publisher-specific metadata |
+| `alternative-id` | list[str] | Publisher's internal IDs |
+| `member` | string | CrossRef member ID |
+| `prefix` | string | DOI prefix |
+| `score` | float | Relevance score (search results only) |
+| `source` | string | e.g. `"Crossref"` |
+| `indexed` | date-object | When CrossRef indexed this record |
+| `deposited` | date-object | When publisher last deposited metadata |
+| `created` | date-object | When CrossRef record was first created |
+
+### Author object fields
+
+| Field | Notes |
+|---|---|
+| `given` | Given/first name |
+| `family` | Family/last name |
+| `sequence` | `"first"` or `"additional"` |
+| `affiliation` | list of `{name, place}` — usually `[]` |
+| `ORCID` | Full URL `"https://orcid.org/0000-0001-..."` — strip prefix to get bare ID |
+| `authenticated-orcid` | `true` = verified via ORCID OAuth; `false` = self-reported |
+| `name` | Used instead of given/family for organizations |
+
+### Date object structure
+
+```python
+# All date fields share this structure:
+date_obj = {
+    "date-parts": [[2021, 7, 15]],  # [[year, month, day]] — month/day may be absent
+    "date-time": "2021-07-15T00:00:00Z",  # not always present
+    "timestamp": 1626307200000               # not always present
+}
+
+# Safe extraction (handles [[2021]] or [[2021, 7]] partial dates):
+def parse_date(d):
+    if not d: return None
+    parts = (d.get('date-parts') or [[]])[0]
+    return '-'.join(str(p) for p in parts if p is not None)
+```
+
+### Type identifiers (filter param values vs facet display names)
+
+Use these exact strings in `filter=type:...`. The facet `type-name` values are display names only.
+
+| filter `type:` value | Facet display name | Count (all CrossRef) |
+|---|---|---|
+| `journal-article` | Journal Article | 121,030,194 |
+| `book-chapter` | Chapter | 24,359,059 |
+| `proceedings-article` | Conference Paper | 9,744,754 |
+| `dataset` | Dataset | 3,424,142 |
+| `posted-content` | Posted Content (preprints) | 3,203,320 |
+| `dissertation` | Dissertation | 1,044,461 |
+| `peer-review` | Peer Review | 1,028,287 |
+| `report` | Report | 906,301 |
+| `book` | Book | 870,949 |
+| `monograph` | Monograph | 788,401 |
+
+### Query parameters reference
+
+| Parameter | Notes |
+|---|---|
+| `query` | Full-text keyword search across title, abstract, author |
+| `query.author` | Author name search only |
+| `query.title` | Title search only |
+| `query.bibliographic` | Combined title + author + journal search |
+| `rows` | Results per page (default 20, max 1000) |
+| `offset` | Offset for pagination (max ~10,000 effective) |
+| `cursor` | Use `cursor=*` for first page, then URL-encode `next-cursor` value |
+| `sort` | `relevance`, `is-referenced-by-count`, `published`, `indexed` |
+| `order` | `asc` or `desc` |
+| `filter` | Comma-separated `key:value` pairs (see filters below) |
+| `select` | Comma-separated field names to return |
+| `facet` | `type-name:*` for type counts; `publisher-name:10` for top publishers |
+| `mailto` | Your email — enables polite pool (higher limits) |
+
+### Filter keys reference
+
+| Filter key | Example | Notes |
+|---|---|---|
+| `doi` | `doi:10.1038/nature12345` | Exact DOI match |
+| `type` | `type:journal-article` | See type table above for valid values |
+| `from-pub-date` | `from-pub-date:2024-01-01` | ISO date or `YYYY` |
+| `until-pub-date` | `until-pub-date:2024-12-31` | |
+| `from-index-date` | `from-index-date:2024-01-01` | When CrossRef indexed it |
+| `has-abstract` | `has-abstract:true` | Only works with deposited abstract |
+| `has-orcid` | `has-orcid:true` | At least one author has ORCID |
+| `has-full-text` | `has-full-text:true` | Has full-text link |
+| `has-references` | `has-references:true` | Has deposited reference list |
+| `is-update` | `is-update:true` | Corrections, retractions |
+| `issn` | `issn:0028-0836` | Filter by journal ISSN |
+| `publisher-name` | `publisher-name:elsevier` | Partial match |
+| `funder` | `funder:100000001` | Funder DOI or CrossRef funder ID |
+
+## Rate limits
+
+CrossRef has two pools based on whether `mailto=` is present:
+
+| Pool | Triggered by | Rate limit | Concurrency |
+|---|---|---|---|
+| **polite** | `mailto=` param present | 10 req/s | 3 concurrent |
+| **public** | no `mailto=` | 5 req/s | 1 concurrent |
+
+Headers returned: `x-rate-limit-limit`, `x-rate-limit-interval`, `x-concurrency-limit`, `x-api-pool`.
+
+In practice with polite pool: 10 rapid sequential calls complete in ~2.7s (avg 0.27s/req) with no throttling. 5 parallel calls complete in ~0.3s. Stay at `max_workers=5` to respect the concurrency limit.
+
+No per-day or per-hour cap. If you exceed limits, responses slow or return HTTP 429. No ban. Add `time.sleep(0.1)` between calls for sustained bulk crawls.
+
+## Gotchas
+
+- **`mailto=` doubles your rate limit and concurrency.** Public pool: 5 req/s, concurrency=1. Polite pool: 10 req/s, concurrency=3. Always add `?mailto=your@email.com` to every request — confirmed by reading `x-api-pool` response header.
+
+- **`title`, `container-title`, `ISSN` are always lists, not strings.** Access with `title[0]`, `container-title[0]` etc. Do not rely on there being only one entry — `container-title` can have multiple values.
+
+- **Abstract contains JATS XML markup.** The `abstract` field is not plain text — it contains tags like `<jats:p>`, `<jats:italic>`, `<jats:sup>`. Strip with `re.sub(r'<[^>]+>', ' ', abstract)`. About 30-70% of works have an abstract at all; journal articles 2023 with `has-abstract:true` filter: 3,041,841 / ~5.5M total = ~55%.
+
+- **ORCID is a full URL, not just the ID.** `a['ORCID']` = `"https://orcid.org/0000-0001-6169-6580"`. Strip with `.replace('https://orcid.org/', '')` to get the bare ID. `authenticated-orcid: false` means self-asserted (not verified via OAuth).
+
+- **`published` vs `published-print` vs `published-online`.** Online-first is common in journals — a paper may be online months before its print issue. `published` is CrossRef's best single date and equals `published-online` when both exist. For preprints (`posted-content` type), look for `posted` instead of `published-print` — it may only have `posted` and `published`. Partial dates like `[[2023]]` (year only) are valid — always use `parse_date()` to handle missing month/day.
+
+- **404 raises `HTTPError`, not a JSON error response.** An invalid DOI (e.g. `10.9999/doesnotexist`) raises `urllib.error.HTTPError: HTTP Error 404: Not Found`. Wrap `fetch_work()` in try/except for any untrusted DOI list.
+
+- **`volume` and `issue` are strings, not integers.** CrossRef stores them as strings — `"596"`, not `596`. Don't compare with `==` to an int.
+
+- **Filter type values are hyphenated lowercase, not the facet display names.** `filter=type:journal-article` works. `filter=type:journal article`, `filter=type:Journal Article`, and `filter=type:conference-paper` all return HTTP 400. Conference papers are `proceedings-article`.
+
+- **`select=` does not guarantee field presence.** When you `select=DOI,title,author`, a record that has no author still omits the `author` key — it doesn't return `author: []`. Always use `.get()`.
+
+- **Cursor pagination required for >10,000 results.** Offset pagination (`offset=`) is limited to around 10,000 results. For bulk sweeps, use `cursor=*` for the first page, then URL-encode the returned `next-cursor` value with `urllib.parse.quote()`. The cursor expires if unused for too long.
+
+- **`rows` max is 1000 per call.** Requesting more silently returns 1000. For cursor-based sweeps of large result sets (millions of records), `rows=1000` with cursor is the most efficient approach.
+
+- **HTML entities in titles.** Titles may contain HTML entities like `&amp;` — `"Deep learning &amp; convolutional networks"`. Decode with `html.unescape()` if needed.
+
+- **`funder` search `works-count` field is `None`.** The funder search result object has a `works-count` key that is always `None` in the search response. To get actual work counts for a funder, fetch the funder directly: `GET /funders/{id}`.
+
+- **`subject` is often an empty list.** The `subject` field in works is populated inconsistently — many journal articles have `subject: []` even for well-indexed journals like Nature.
+
+- **Affiliation is usually empty.** `author[i]['affiliation']` is `[]` for the majority of records, even for papers published in 2024. CrossRef has been working on affiliation deposit, but coverage is inconsistent.