catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/arxiv/scraping.md

@@ -0,0 +1,311 @@
+# ArXiv — Scraping & Data Extraction
+
+`https://arxiv.org` — open-access preprint server. **Never use the browser for ArXiv.** All data is reachable via `http_get` using the Atom API or HTML meta tags. No API key required.
+
+## Do this first
+
+**Use the Atom API for any paper search or metadata fetch — one call, XML response, no auth.**
+
+```python
+import xml.etree.ElementTree as ET
+from helpers import http_get
+
+NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}
+
+xml = http_get("http://export.arxiv.org/api/query?search_query=ti:transformer+AND+cat:cs.LG&max_results=5&sortBy=submittedDate&sortOrder=descending")
+root = ET.fromstring(xml)
+entries = root.findall('atom:entry', NS)
+```
+
+Use `id_list` for known paper IDs — supports comma-separated batch fetch in a single call.
+
+Use `http_get` on `https://arxiv.org/abs/{id}` + regex for `citation_*` meta tags when you need the full abstract from an HTML page.
+
+## Common workflows
+
+### Search papers (API)
+
+```python
+import xml.etree.ElementTree as ET
+from helpers import http_get
+
+NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}
+
+xml = http_get(
+    "http://export.arxiv.org/api/query"
+    "?search_query=ti:transformer+AND+cat:cs.LG"
+    "&max_results=5&sortBy=submittedDate&sortOrder=descending"
+)
+root = ET.fromstring(xml)
+entries = root.findall('atom:entry', NS)
+for e in entries:
+    title     = e.find('atom:title', NS).text.strip().replace('\n', ' ')
+    arxiv_id  = e.find('atom:id', NS).text.split('/')[-1]   # e.g. '2604.15259v1'
+    published = e.find('atom:published', NS).text[:10]       # '2026-04-16'
+    updated   = e.find('atom:updated', NS).text[:10]
+    abstract  = e.find('atom:summary', NS).text.strip()
+    authors   = [a.find('atom:name', NS).text for a in e.findall('atom:author', NS)]
+    cats      = [c.get('term') for c in e.findall('atom:category', NS)]
+    primary   = e.find('arxiv:primary_category', NS).get('term')
+    comment   = e.find('arxiv:comment', NS)
+    pdf_link  = next((l.get('href') for l in e.findall('atom:link', NS) if l.get('title') == 'pdf'), None)
+    abs_link  = next((l.get('href') for l in e.findall('atom:link', NS) if l.get('rel') == 'alternate'), None)
+    print(arxiv_id, published, title[:60])
+    print("  Authors:", authors[:2])
+    print("  PDF:", pdf_link)
+# Confirmed output (2026-04-18):
+# 2604.15259v1 2026-04-16 Stability and Generalization in Looped Transformers
+#   Authors: ['Asher Labovich']
+#   PDF: https://arxiv.org/pdf/2604.15259v1
+```
+
+### Fetch single paper by ID (API)
+
+```python
+import xml.etree.ElementTree as ET
+from helpers import http_get
+
+NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}
+
+xml = http_get("http://export.arxiv.org/api/query?id_list=1706.03762")
+root = ET.fromstring(xml)
+e = root.find('atom:entry', NS)
+title      = e.find('atom:title', NS).text.strip()
+abstract   = e.find('atom:summary', NS).text.strip()
+categories = [c.get('term') for c in e.findall('atom:category', NS)]
+pdf_link   = next((l.get('href') for l in e.findall('atom:link', NS) if l.get('title') == 'pdf'), None)
+print("Title:", title)
+print("Categories:", categories)
+print("PDF:", pdf_link)
+print("Abstract:", abstract[:200])
+# Confirmed output:
+# Title: Attention Is All You Need
+# Categories: ['cs.CL', 'cs.LG']
+# PDF: https://arxiv.org/pdf/1706.03762v7
+# Abstract: The dominant sequence transduction models are based on complex recurrent...
+```
+
+### Batch fetch by comma-separated IDs (single call — fast)
+
+Fetching 10 IDs in one call takes ~2s. Prefer this over parallel single-ID fetches.
+
+```python
+import xml.etree.ElementTree as ET
+from helpers import http_get
+
+NS = {'atom': 'http://www.w3.org/2005/Atom'}
+
+ids = ['1706.03762', '1810.04805', '2005.14165']  # Transformer, BERT, GPT-3
+xml = http_get(f"http://export.arxiv.org/api/query?id_list={','.join(ids)}&max_results={len(ids)}")
+root = ET.fromstring(xml)
+for e in root.findall('atom:entry', NS):
+    arxiv_id  = e.find('atom:id', NS).text.split('/')[-1]
+    title     = e.find('atom:title', NS).text.strip()
+    published = e.find('atom:published', NS).text[:10]
+    print(arxiv_id, published, title[:60])
+# Confirmed output:
+# 1512.03385v1 2015-12-10 Deep Residual Learning for Image Recognition
+# 1706.03762v7 2017-06-12 Attention Is All You Need
+# 2005.14165v4 2020-05-28 Language Models are Few-Shot Learners
+# 1810.04805v2 2018-10-11 BERT: Pre-training of Deep Bidirectional Transformers...
+# Note: order returned may differ from order requested
+```
+
+### Parallel fetch (ThreadPoolExecutor for independent IDs)
+
+Use only when IDs are not known upfront or when mixing with other work. For pure batch, single comma-separated `id_list` call is faster.
+
+```python
+import xml.etree.ElementTree as ET
+from concurrent.futures import ThreadPoolExecutor
+from helpers import http_get
+
+NS = {'atom': 'http://www.w3.org/2005/Atom'}
+
+def fetch_paper(arxiv_id):
+    xml = http_get(f"http://export.arxiv.org/api/query?id_list={arxiv_id}")
+    root = ET.fromstring(xml)
+    e = root.find('atom:entry', NS)
+    if e is None:
+        return None
+    return {
+        'id': arxiv_id,
+        'title': e.find('atom:title', NS).text.strip(),
+        'published': e.find('atom:published', NS).text[:10],
+    }
+
+ids = ['1706.03762', '1810.04805', '2005.14165']
+with ThreadPoolExecutor(max_workers=3) as ex:
+    papers = list(ex.map(fetch_paper, ids))
+for p in papers:
+    print(p['id'], p['published'], p['title'][:60])
+# Confirmed working — max_workers=3 is safe; don't exceed 5 for continuous crawling
+```
+
+### HTML abstract page — citation_* meta tags
+
+Use this when you want the full abstract or the versionless PDF URL without parsing Atom XML.
+
+```python
+import re
+from helpers import http_get
+
+html = http_get("https://arxiv.org/abs/1706.03762", headers={"User-Agent": "Mozilla/5.0"})
+# HTML page is ~48 KB, fully static, no JS required
+
+title   = re.search(r'<meta name="citation_title" content="([^"]+)"', html)
+pdf_url = re.search(r'<meta name="citation_pdf_url" content="([^"]+)"', html)
+authors = re.findall(r'<meta name="citation_author" content="([^"]+)"', html)
+date    = re.search(r'<meta name="citation_date" content="([^"]+)"', html)
+arxiv_id = re.search(r'<meta name="citation_arxiv_id" content="([^"]+)"', html)
+abstract = re.search(r'<meta name="citation_abstract" content="([^"]+)"', html)
+
+print("Title:", title.group(1) if title else None)
+print("PDF:", pdf_url.group(1) if pdf_url else None)
+print("Authors:", authors[:3])
+print("Date:", date.group(1) if date else None)
+print("ID:", arxiv_id.group(1) if arxiv_id else None)
+# Confirmed output for 1706.03762:
+# Title: Attention Is All You Need
+# PDF: https://arxiv.org/pdf/1706.03762   (no version suffix — always latest)
+# Authors: ['Vaswani, Ashish', 'Shazeer, Noam', 'Parmar, Niki']
+# Date: 2017/06/12
+# ID: 1706.03762
+```
+
+All `citation_*` meta tags present on the abs page:
+- `citation_title` — paper title
+- `citation_author` — one tag per author, format `"Last, First"`
+- `citation_date` — submission date `YYYY/MM/DD`
+- `citation_online_date` — latest version date `YYYY/MM/DD`
+- `citation_pdf_url` — versionless PDF URL (redirects to latest)
+- `citation_arxiv_id` — bare ID without version suffix
+- `citation_abstract` — full abstract text
+
+### Category search with pagination
+
+```python
+import xml.etree.ElementTree as ET
+from helpers import http_get
+
+NS = {
+    'atom': 'http://www.w3.org/2005/Atom',
+    'opensearch': 'http://a9.com/-/spec/opensearch/1.1/',
+}
+
+# Page 1
+xml = http_get(
+    "http://export.arxiv.org/api/query"
+    "?search_query=cat:cs.AI"
+    "&max_results=10&start=0&sortBy=lastUpdatedDate&sortOrder=descending"
+)
+root = ET.fromstring(xml)
+total   = root.find('opensearch:totalResults', NS).text   # e.g. '172726'
+start_i = root.find('opensearch:startIndex', NS).text
+per_pg  = root.find('opensearch:itemsPerPage', NS).text
+print(f"Total cs.AI papers: {total}")  # Confirmed: 172726 (2026-04-18)
+
+entries = root.findall('atom:entry', NS)
+
+# Page 2: increment start
+xml2 = http_get(
+    "http://export.arxiv.org/api/query"
+    "?search_query=cat:cs.AI"
+    "&max_results=10&start=10&sortBy=lastUpdatedDate&sortOrder=descending"
+)
+```
+
+## URL and ID reference
+
+### API base URL
+
+```
+http://export.arxiv.org/api/query
+```
+
+HTTPS also works: `https://export.arxiv.org/api/query`
+
+### Query parameters
+
+| Parameter | Values | Notes |
+|---|---|---|
+| `search_query` | `ti:word`, `au:name`, `abs:phrase`, `cat:cs.LG`, combine with `AND`/`OR`/`ANDNOT` | URL-encode spaces as `+` |
+| `id_list` | `1706.03762` or `1706.03762,1810.04805` | Comma-separated; version suffix optional |
+| `max_results` | integer (default 10, max 2000) | |
+| `start` | integer (default 0) | Offset for pagination |
+| `sortBy` | `relevance`, `lastUpdatedDate`, `submittedDate` | |
+| `sortOrder` | `ascending`, `descending` | |
+
+### Search field prefixes
+
+| Prefix | Searches |
+|---|---|
+| `ti:` | Title |
+| `au:` | Author name |
+| `abs:` | Abstract |
+| `co:` | Comment |
+| `jr:` | Journal reference |
+| `cat:` | Category (e.g. `cat:cs.LG`) |
+| `all:` | All fields |
+
+### PDF and abstract URL construction
+
+```python
+import re
+
+arxiv_id = "1706.03762v7"                        # from API atom:id field
+bare_id = re.sub(r'v\d+$', '', arxiv_id)          # strip version: '1706.03762'
+
+pdf_versioned   = f"https://arxiv.org/pdf/{arxiv_id}"   # specific version
+pdf_latest      = f"https://arxiv.org/pdf/{bare_id}"    # always redirects to latest
+abs_versioned   = f"https://arxiv.org/abs/{arxiv_id}"
+abs_latest      = f"https://arxiv.org/abs/{bare_id}"
+```
+
+The API's `atom:link[@title='pdf']` href includes the version suffix. The HTML `citation_pdf_url` meta tag does not — it always resolves to the latest.
+
+### Category codes (confirmed paper counts, 2026-04-18)
+
+| Code | Area | Papers |
+|---|---|---|
+| `cs.LG` | Machine Learning | 261,782 |
+| `cs.CV` | Computer Vision | 189,049 |
+| `cs.AI` | Artificial Intelligence | 172,726 |
+| `cs.CL` | Computation and Language (NLP) | 106,724 |
+| `stat.ML` | Statistics - Machine Learning | 76,902 |
+| `math.OC` | Optimization and Control | 60,669 |
+| `eess.AS` | Audio and Speech Processing | 21,288 |
+| `cs.NE` | Neural and Evolutionary Computing | 17,475 |
+| `q-bio.NC` | Neurons and Cognition | 11,903 |
+
+Full category taxonomy: https://arxiv.org/category_taxonomy
+
+## Gotchas
+
+- **Never use the browser for ArXiv.** The abstract page (`/abs/`) and search results are fully server-side rendered static HTML. `http_get` is sufficient for everything including full abstracts, author lists, and PDF URLs.
+
+- **Always define the namespace dict.** Without `NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}`, `findall('atom:entry')` silently returns `[]`. All ArXiv Atom elements live in the `http://www.w3.org/2005/Atom` namespace; ArXiv-specific fields (`comment`, `primary_category`, `journal_ref`, `doi`) live in `http://arxiv.org/schemas/atom`.
+
+- **Batch single `id_list` call is faster than ThreadPoolExecutor.** A comma-separated `id_list` with 10 IDs resolved in one call (1.91s) vs. 10 separate `ThreadPoolExecutor` calls (6.34s). Use the batch form when you already have the IDs.
+
+- **`atom:id` contains a URL, not a bare ID.** The element text is `http://arxiv.org/abs/1706.03762v7` — always split on `/` and take `[-1]` to get the bare ID with version. Strip version with `re.sub(r'v\d+$', '', id)` if needed.
+
+- **Batch `id_list` returns entries in unpredictable order.** When fetching `1706.03762,1810.04805,2005.14165`, entries came back ordered by publication date, not by the order given in the request. Index by ID, not position.
+
+- **`max_results` must be set explicitly when using `id_list` batches.** If you request 10 IDs but omit `max_results`, the API defaults to 10, which happens to work — but set it explicitly to `len(ids)` to be safe.
+
+- **Nonexistent IDs return zero entries, not an error.** `id_list=9999.99999` gives `totalResults=0` and an empty `atom:entry` list. Always check `len(entries) > 0` before accessing `entries[0]`.
+
+- **`arxiv:comment` and `arxiv:journal_ref` / `arxiv:doi` may be absent.** Not all papers have these fields. Use `e.find('arxiv:comment', NS)` and check `if el is not None and el.text`.
+
+- **Rate limit: 3 seconds between requests recommended for bulk crawling.** In practice, rapid bursts of 10 individual requests complete in ~6s (avg 0.63s/req) without being blocked. For sustained crawls over hundreds of papers, insert `time.sleep(3)` between requests. The API does not return rate limit headers — it just starts slowing responses or returns HTTP 503 silently.
+
+- **`citation_author` tags are in `"Last, First"` format**, not `"First Last"` like the Atom API. The Atom `atom:author/atom:name` field gives `"First Last"` order. Pick the format that matches your downstream use.
+
+- **The `arxiv:affiliation` sub-element of `atom:author` is rarely populated.** Most institutional affiliations are absent from the API response even when listed on the paper. The HTML abs page doesn't expose them in meta tags either.
+
+- **`sortBy=relevance` applies only with `search_query`.** Using `sortBy=relevance` with `id_list` has no effect — results still come back in date order.
+
+- **`max_results` cap is 2000 per call.** For bulk harvesting of a category, use `start` offset pagination and add 3s sleep between pages. `opensearch:totalResults` tells you the total so you can compute how many pages are needed.
+
+- **HTML `citation_abstract` meta tag contains the full abstract.** Unlike the Atom `atom:summary` which can have trailing whitespace and embedded newlines, the meta tag version is a single clean string — no `.strip()` needed.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/arxiv-bulk/scraping.md

@@ -0,0 +1,333 @@
+# arXiv Bulk Harvest + Semantic Scholar — OAI-PMH & Citation Enrichment
+
+Companion to `domain-skills/arxiv/scraping.md`. Use the **arxiv** skill for search-and-fetch workflows. Use **this skill** when you need:
+
+- Bulk-harvesting all papers in a subject area or date window (OAI-PMH)
+- Citation counts, influential-citation scores, and cross-database IDs (Semantic Scholar)
+- Per-paper version history and submitter info (`arXivRaw` metadata)
+
+No API key required for either endpoint. Both return JSON or XML over plain HTTP.
+
+---
+
+## OAI-PMH bulk harvest
+
+### Endpoint (confirmed 2026-04-19)
+
+```
+https://oaipmh.arxiv.org/oai
+```
+
+`https://export.arxiv.org/oai2` is the old URL — it 301-redirects to the new one. Use the new URL directly to avoid the extra round-trip.
+
+### Harvest all cs papers from a date window
+
+```python
+import xml.etree.ElementTree as ET
+from helpers import http_get
+
+OAI_NS = {
+    'oai': 'http://www.openarchives.org/OAI/2.0/',
+    'arXiv': 'http://arxiv.org/OAI/arXiv/',
+}
+
+def fetch_oai_page(url):
+    """Fetch one OAI-PMH page; return (records_xml_list, next_token_or_None)."""
+    xml = http_get(url)
+    root = ET.fromstring(xml)
+    records = root.findall('.//oai:record', OAI_NS)
+    token_el = root.find('.//oai:resumptionToken', OAI_NS)
+    token = token_el.text if token_el is not None and token_el.text else None
+    return records, token
+
+def parse_arxiv_record(rec):
+    """Extract fields from one <record> element (metadataPrefix=arXiv)."""
+    header = rec.find('oai:header', OAI_NS)
+    meta   = rec.find('.//arXiv:arXiv', OAI_NS)
+    if meta is None:
+        return None   # deleted record (header has status="deleted")
+    authors_el = meta.findall('arXiv:authors/arXiv:author', OAI_NS)
+    authors = []
+    for a in authors_el:
+        fn = (a.findtext('arXiv:forenames', namespaces=OAI_NS) or '').strip()
+        ln = (a.findtext('arXiv:keyname',   namespaces=OAI_NS) or '').strip()
+        authors.append(f"{fn} {ln}".strip())
+    return {
+        'id':           meta.findtext('arXiv:id', namespaces=OAI_NS),
+        'datestamp':    header.findtext('oai:datestamp', namespaces=OAI_NS),
+        'created':      meta.findtext('arXiv:created',  namespaces=OAI_NS),
+        'updated':      meta.findtext('arXiv:updated',  namespaces=OAI_NS),
+        'title':        (meta.findtext('arXiv:title',    namespaces=OAI_NS) or '').strip(),
+        'authors':      authors,
+        'categories':   (meta.findtext('arXiv:categories', namespaces=OAI_NS) or '').split(),
+        'abstract':     (meta.findtext('arXiv:abstract',   namespaces=OAI_NS) or '').strip(),
+        'doi':          meta.findtext('arXiv:doi',         namespaces=OAI_NS),
+        'journal_ref':  meta.findtext('arXiv:journal-ref', namespaces=OAI_NS),
+        'license':      meta.findtext('arXiv:license',     namespaces=OAI_NS),
+    }
+
+# --- Main harvest loop ---
+import time
+
+BASE = 'https://oaipmh.arxiv.org/oai'
+first_url = (
+    f"{BASE}?verb=ListRecords"
+    f"&metadataPrefix=arXiv"
+    f"&set=cs"
+    f"&from=2024-01-01"
+    f"&until=2024-01-02"
+)
+
+papers = []
+url = first_url
+while url:
+    records, token = fetch_oai_page(url)
+    for rec in records:
+        p = parse_arxiv_record(rec)
+        if p:
+            papers.append(p)
+    print(f"  fetched {len(records)} records, total so far: {len(papers)}")
+    if token:
+        url = f"{BASE}?verb=ListRecords&resumptionToken={token}"
+        time.sleep(5)   # OAI-PMH policy: >=5s between pages
+    else:
+        url = None
+
+print(f"Done. {len(papers)} papers harvested.")
+# Confirmed output for cs, 2024-01-01 to 2024-01-02:
+# fetched 44 records, total so far: 44
+# Done. 44 papers harvested.
+# For 2024-01-01 to 2024-01-07 (cs): multiple pages, resumptionToken issued when >~200 records
+```
+
+### Available verbs
+
+| Verb | Purpose | Key params |
+|---|---|---|
+| `Identify` | Repository info, earliest datestamp (`2005-09-16`) | — |
+| `ListSets` | All harvestable sets (see table below) | — |
+| `ListMetadataFormats` | `oai_dc`, `arXiv`, `arXivOld`, `arXivRaw` | — |
+| `ListRecords` | Bulk harvest with date/set filter | `metadataPrefix`, `set`, `from`, `until` |
+| `GetRecord` | Single record by OAI identifier | `identifier`, `metadataPrefix` |
+
+### Top-level sets (confirmed)
+
+| setSpec | Name |
+|---|---|
+| `cs` | Computer Science (all) |
+| `cs:cs` | Computer Science (subset notation — same scope) |
+| `math` | Mathematics |
+| `physics` | Physics |
+| `stat` | Statistics |
+| `eess` | Electrical Engineering and Systems Science |
+| `econ` | Economics |
+| `q-bio` | Quantitative Biology |
+| `q-fin` | Quantitative Finance |
+
+Subset sets use `topic:topic:SUBCATEGORY` notation, e.g. `cs:cs:LG` for Machine Learning. List all with `verb=ListSets`.
+
+### Available metadata formats
+
+- `arXiv` — rich: id, created/updated dates, authors (keyname + forenames separately), categories, abstract, doi, journal-ref, license. **Use this.**
+- `arXivRaw` — adds `<submitter>`, per-version history (`<version version="v1">` with date and file size), author list as flat string. Use when you need version history.
+- `oai_dc` — Dublin Core, minimal. Skip unless you need cross-system compatibility.
+- `arXivOld` — legacy format pre-2007. Skip.
+
+### GetRecord + arXivRaw (version history)
+
+```python
+import xml.etree.ElementTree as ET
+from helpers import http_get
+
+RAW_NS = {
+    'oai': 'http://www.openarchives.org/OAI/2.0/',
+    'raw': 'http://arxiv.org/OAI/arXivRaw/',
+}
+
+xml = http_get(
+    "https://oaipmh.arxiv.org/oai"
+    "?verb=GetRecord"
+    "&metadataPrefix=arXivRaw"
+    "&identifier=oai:arXiv.org:1706.03762"
+)
+root = ET.fromstring(xml)
+meta = root.find('.//raw:arXivRaw', RAW_NS)
+
+title     = meta.findtext('raw:title',     namespaces=RAW_NS)
+submitter = meta.findtext('raw:submitter', namespaces=RAW_NS)
+versions  = meta.findall('raw:version',    RAW_NS)
+for v in versions:
+    print(v.get('version'), v.findtext('raw:date', namespaces=RAW_NS))
+# Confirmed output for 1706.03762 ("Attention Is All You Need"):
+# v1 Mon, 12 Jun 2017 17:57:34 GMT
+# v2 Mon, 19 Jun 2017 16:49:45 GMT
+# ...
+# v7 Wed, 02 Aug 2023 00:41:18 GMT
+# submitter: Llion Jones
+```
+
+---
+
+## Semantic Scholar — citation enrichment for arXiv papers
+
+No API key required (unauthenticated: 1 req/s, 5000 req/day). With a free key the limit rises to 100 req/s.
+
+Base URL: `https://api.semanticscholar.org/graph/v1/`
+
+### Single paper lookup by arXiv ID
+
+```python
+import json
+from helpers import http_get
+
+paper = json.loads(http_get(
+    "https://api.semanticscholar.org/graph/v1/paper/arXiv:1706.03762"
+    "?fields=title,year,venue,publicationDate,citationCount,"
+    "influentialCitationCount,authors,abstract,externalIds"
+))
+print(paper['title'])                    # "Attention is All you Need"
+print(paper['citationCount'])            # 173155  (confirmed 2026-04-19)
+print(paper['influentialCitationCount']) # 19629
+print(paper['venue'])                    # "Neural Information Processing Systems"
+print(paper['externalIds']['ArXiv'])     # "1706.03762"
+print(paper['externalIds']['DOI'])       # missing if no DOI
+for a in paper['authors']:
+    print(a['name'], a['authorId'])
+```
+
+The ID format `arXiv:NNNN.NNNNN` is accepted directly — no conversion needed.
+
+### Batch lookup (up to 500 IDs per POST)
+
+```python
+import json
+from helpers import http_get
+import urllib.request
+
+ids = ["arXiv:1706.03762", "arXiv:1810.04805", "arXiv:2005.14165"]
+fields = "paperId,externalIds,title,year,citationCount,influentialCitationCount"
+
+body = json.dumps({"ids": ids}).encode()
+req = urllib.request.Request(
+    f"https://api.semanticscholar.org/graph/v1/paper/batch?fields={fields}",
+    data=body,
+    headers={"Content-Type": "application/json"},
+    method="POST",
+)
+with urllib.request.urlopen(req, timeout=20) as r:
+    results = json.loads(r.read())
+
+for p in results:
+    print(p['externalIds'].get('ArXiv'), p['citationCount'], p['title'][:50])
+# Confirmed output (2026-04-19):
+# 1706.03762  173155  Attention is All you Need
+# 1810.04805  113138  BERT: Pre-training of Deep Bidirectional Tran...
+# 2005.14165  (varies)  Language Models are Few-Shot Learners
+```
+
+Note: `helpers.http_get` only does GET. For POST use `urllib.request.Request` directly as above.
+
+### Paper search
+
+```python
+import json
+from helpers import http_get
+
+results = json.loads(http_get(
+    "https://api.semanticscholar.org/graph/v1/paper/search"
+    "?query=large+language+model"
+    "&fields=paperId,externalIds,title,year,citationCount"
+    "&limit=5"
+))
+total = results['total']   # e.g. 3473582 for "large language model"
+for p in results['data']:
+    arxiv_id = p['externalIds'].get('ArXiv', 'no-arxiv')
+    print(arxiv_id, p['year'], p['citationCount'], p['title'][:50])
+# next page: use offset=5, offset=10, etc.
+```
+
+### Available fields (pass as comma-separated `fields=` query param)
+
+| Field | Type | Notes |
+|---|---|---|
+| `paperId` | str | Semantic Scholar internal ID |
+| `externalIds` | dict | Keys: `ArXiv`, `DOI`, `DBLP`, `MAG`, `ACL`, `CorpusId` |
+| `title` | str | |
+| `abstract` | str | |
+| `year` | int | Publication year |
+| `publicationDate` | str | `YYYY-MM-DD` |
+| `venue` | str | Conference/journal name |
+| `citationCount` | int | Total citations |
+| `influentialCitationCount` | int | Citations deemed highly influential |
+| `authors` | list | Each: `{authorId, name}` |
+| `references` | list | List of paper objects (needs own `fields`) |
+| `citations` | list | Citing papers (needs own `fields`) |
+| `openAccessPdf` | dict | `{url, status, license}` |
+
+---
+
+## Downloading PDFs
+
+Direct PDF download — no auth, no redirect for versionless URLs (returns 200 + PDF body directly).
+
+```python
+import urllib.request
+
+def download_pdf(arxiv_id, dest_path, version=None):
+    """
+    arxiv_id: bare ID like '1706.03762' or versioned '1706.03762v7'
+    version:  if given, appended as 'v{version}' — ignored if arxiv_id already has version
+    dest_path: where to save, e.g. '/tmp/paper.pdf'
+    """
+    if 'v' not in arxiv_id.split('.')[-1] and version:
+        arxiv_id = f"{arxiv_id}v{version}"
+    url = f"https://arxiv.org/pdf/{arxiv_id}"
+    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=60) as r:
+        with open(dest_path, 'wb') as f:
+            f.write(r.read())
+    print(f"Saved {r.headers.get('content-length', '?')} bytes to {dest_path}")
+
+download_pdf('1706.03762', '/tmp/attention.pdf')
+# Confirmed: saves 2215244 bytes, filename hint in header: '1706.03762v7.pdf'
+# Versionless URL resolves to latest version server-side (no redirect, 200 direct)
+```
+
+---
+
+## Gotchas
+
+- **OAI-PMH endpoint moved.** `https://export.arxiv.org/oai2` 301-redirects to `https://oaipmh.arxiv.org/oai`. Use the new URL. `helpers.http_get` (which uses `urllib`) does NOT follow redirects — you'll get an empty string or error. Either use `urllib.request.urlopen` with `follow_redirects` logic, or just use the canonical URL directly.
+
+- **OAI-PMH rate limit: 5 seconds between pages.** The protocol requires a `Retry-After` interval. The server embeds an `expirationDate` on the resumptionToken. Violating the rate limit causes the token to be invalidated and the harvest fails silently. Always `time.sleep(5)` between pages.
+
+- **Resumption token is opaque but URL-encoded.** The token looks like `verb%3DListRecords%26...%26skip%3D247`. Pass it verbatim as `&resumptionToken=<token>` — do not URL-encode it again.
+
+- **`datestamp` in OAI-PMH is last-modified date, not submission date.** A paper submitted in 2008 can appear in a 2024 harvest window if it was revised then. The `<created>` and `<updated>` fields inside `<arXiv>` metadata are the actual submission/revision dates.
+
+- **Deleted records have no `<metadata>` element.** The `<header>` will carry `status="deleted"`. Always check `meta is None` after `find('.//arXiv:arXiv', ...)`.
+
+- **Author structure differs between OAI-PMH formats.** In `arXiv` metadata, authors are structured: `<author><keyname>Vaswani</keyname><forenames>Ashish</forenames></author>`. In `arXivRaw`, they're a flat comma-separated string: `Ashish Vaswani, Noam Shazeer, ...`. In the Atom API, it's `<name>Ashish Vaswani</name>` (first-last order). Pick the source that matches your downstream use.
+
+- **Semantic Scholar 429 under unauthenticated bursts.** The unauthenticated limit is ~1 req/s. Rapid parallel calls return `{"code": "429"}`. Add `time.sleep(1)` between single lookups or use the batch POST endpoint (up to 500 IDs, single request) to stay under the limit. The batch endpoint itself counts as 1 request.
+
+- **Semantic Scholar `externalIds` may lack `ArXiv` key.** Not all papers have an arXiv preprint. When enriching an arXiv list with S2 data, always use `.get('ArXiv')` not `['ArXiv']`.
+
+- **Atom API rate limit: 1 request per 3 seconds for sustained crawls.** The API returns HTTP 429 `"Rate exceeded."` on rapid-fire requests. The OAI-PMH endpoint is designed for bulk and is more tolerant, but still requires the 5s sleep between resumption pages.
+
+- **OAI-PMH `set` param uses colon-separated hierarchy, not dot.** The Atom API uses `cat:cs.LG`; OAI-PMH uses `set=cs:cs:LG`. Using `set=cs.LG` returns zero results.
+
+- **`http_get` in helpers.py does NOT follow HTTP redirects.** If you must use it with the old OAI URL, you'll get an empty body. Either update the URL to the canonical one or use `urllib.request.urlopen` with a redirect handler.
+
+---
+
+## How this complements the existing arxiv skill
+
+| Task | Use |
+|---|---|
+| Search by keyword, author, or category | `arxiv` skill — Atom API |
+| Fetch 1–2000 specific papers by ID | `arxiv` skill — `id_list` batch |
+| Harvest all papers in a subject over a date range | **this skill** — OAI-PMH |
+| Get citation counts / influential citations | **this skill** — Semantic Scholar |
+| Get per-version history and submitter name | **this skill** — OAI-PMH `arXivRaw` |
+| Download a PDF | either skill (same URL structure) |