catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/amazon/product-search.md

@@ -0,0 +1,198 @@
+# Amazon — Product Search & Data Extraction
+
+Field-tested against amazon.com on 2025-04-18 using a logged-in Chrome session.
+No CAPTCHA or bot detection was triggered during any test run.
+
+## Navigation
+
+### Direct search URL (fastest, always use this)
+```python
+goto_url("https://www.amazon.com/s?k=mechanical+keyboard")
+wait_for_load()
+wait(2)  # dynamic content needs ~2s after readyState=complete
+```
+
+### Search box typing (use when you need category filtering)
+```python
+goto_url("https://www.amazon.com")
+wait_for_load()
+wait(1)
+js("document.querySelector('#twotabsearchtextbox').focus()")
+js("document.querySelector('#twotabsearchtextbox').click()")
+wait(0.3)
+type_text("wireless mouse")
+wait(0.3)
+press_key("Enter")
+wait_for_load()
+wait(2)
+```
+
+### Direct product page
+```python
+# URL pattern: /dp/{ASIN}  or  /dp/{ASIN}?th=1 (Amazon may redirect to add ?th=1)
+goto_url("https://www.amazon.com/dp/B08Z6X4NK3")
+wait_for_load()
+wait(2)
+```
+
+## Session Gotcha
+
+**Always use `new_tab()` when opening Amazon for the first time in a harness session.**
+`goto_url()` can silently fail to navigate if the current tab resists the navigation
+(observed when the daemon attached to a different real tab). The safe pattern:
+
+```python
+tid = new_tab("https://www.amazon.com/s?k=mechanical+keyboard")
+wait_for_load()
+wait(2)
+```
+
+After that, `goto_url()` works fine within the same Amazon session.
+
+## Search Results Extraction
+
+### Container selector
+`[data-component-type="s-search-result"]` — confirmed working, yields ~22 results per page.
+
+### Full extraction (field-tested)
+```python
+results = js("""
+  Array.from(document.querySelectorAll('[data-component-type="s-search-result"]')).map(el => ({
+    asin: el.getAttribute('data-asin'),
+    title: el.querySelector('h2 span')?.innerText?.trim(),
+    price: el.querySelector('.a-price .a-offscreen')?.innerText,
+    list_price: el.querySelector('.a-text-price .a-offscreen')?.innerText,
+    rating: el.querySelector('[aria-label*="out of 5 stars"]')?.getAttribute('aria-label')?.split(' ')[0],
+    reviews: el.querySelector('[aria-label*="ratings"]')?.getAttribute('aria-label'),
+    is_sponsored: !!el.querySelector('.puis-sponsored-label-text'),
+    url: el.querySelector('h2 a')?.href
+  }))
+""")
+```
+
+### Field notes
+- **`asin`**: `data-asin` attribute on the container div — always present, matches the `/dp/{ASIN}` URL.
+- **`title`**: `h2 span` works consistently. `h2 a.a-link-normal span` also works.
+- **`price`**: `.a-price .a-offscreen` returns the formatted string e.g. `"$69.99"`. Use this, not `.a-price-whole`.
+- **`list_price`**: `.a-text-price .a-offscreen` — only present when item is on sale (was/now pricing).
+- **`rating`**: Use `aria-label` on `[aria-label*="out of 5 stars"]` — gives `"4.5 out of 5 stars, rating details"`, split on space for the number.
+- **`reviews`**: Use `[aria-label*="ratings"]` attribute — gives `"1,514 ratings"`. Do NOT use `.a-size-base.s-underline-text` — that element exists on sponsored results and shows "Xbox" (a cross-sell widget text).
+- **`is_sponsored`**: `.puis-sponsored-label-text` is present on sponsored listings; first 2-3 results are usually sponsored.
+- **`url`**: `h2 a` href — contains the full `/dp/{ASIN}/...` URL.
+
+## Product Detail Page Extraction
+
+### Confirmed selectors (field-tested on B08Z6X4NK3)
+```python
+detail = js("""
+  ({
+    title: document.querySelector('#productTitle')?.innerText?.trim(),
+    price: (function() {
+      var whole = document.querySelector('.a-price-whole')?.innerText?.replace(/[\\n.]/g,'');
+      var frac  = document.querySelector('.a-price-fraction')?.innerText;
+      return (whole && frac) ? '$' + whole + '.' + frac
+           : document.querySelector('.a-price .a-offscreen')?.innerText || null;
+    })(),
+    list_price: document.querySelector('.basisPrice .a-offscreen')?.innerText,
+    rating: document.querySelector('#acrPopover')?.getAttribute('title'),
+    review_count: document.querySelector('#acrCustomerReviewText')?.innerText,
+    availability: document.querySelector('#availability span')?.innerText?.trim(),
+    brand: document.querySelector('#bylineInfo')?.innerText?.trim(),
+    asin: document.querySelector('input[name="ASIN"]')?.value,
+    bullet_points: Array.from(document.querySelectorAll('#feature-bullets li span.a-list-item'))
+                       .map(e => e.innerText?.trim()).filter(t => t)
+  })
+""")
+```
+
+### Price field notes
+- `#priceblock_ourprice` and `#priceblock_dealprice` are **legacy** — they return `null` on modern product pages.
+- Construct price from `.a-price-whole` + `.a-price-fraction` (both stripped of `\n` and `.`).
+- As a fallback: first `.a-price .a-offscreen` on the page also works (confirmed `$69.99`).
+- `list_price` from `.basisPrice .a-offscreen` shows the crossed-out "was" price when a discount exists.
+
+## Best Sellers Page
+
+URL: `https://www.amazon.com/Best-Sellers-{Category}/zgbs/{slug}/`
+e.g. `https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics/`
+
+### DOM structure (2025)
+`.zg-item-immersion` **does not exist** — Amazon migrated to CSS modules. Use `[data-asin]` anchored on `[id="gridItemRoot"]`:
+
+```python
+goto_url("https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics/")
+wait_for_load()
+wait(2)
+
+items = js("""
+  Array.from(document.querySelectorAll('[data-asin]')).map(el => {
+    var container = el.closest('[id="gridItemRoot"]') || el;
+    return {
+      asin: el.getAttribute('data-asin'),
+      rank: container.querySelector('[class*="zg-bdg-text"]')?.innerText,
+      title: container.querySelector('img[alt]')?.getAttribute('alt'),
+      price: container.querySelector('.p13n-sc-price, .a-size-base.a-color-price')?.innerText,
+      url: 'https://www.amazon.com/dp/' + el.getAttribute('data-asin')
+    }
+  }).filter(r => r.rank)
+""")
+```
+
+Note: Title comes from the product image `alt` attribute — the text title elements use obfuscated CSS module class names that change between deployments.
+
+## Pagination
+
+```python
+# Get next page URL directly
+next_url = js("document.querySelector('.s-pagination-next')?.href")
+if next_url:
+    goto_url(next_url)
+    wait_for_load()
+    wait(2)
+
+# Or construct by page number
+goto_url("https://www.amazon.com/s?k=wireless+mouse&page=2")
+```
+
+## Result Count
+
+```python
+count_text = js("document.querySelector('[data-component-type=\"s-result-info-bar\"] h1')?.innerText?.trim()")
+# Returns e.g.: '1-16 of over 40,000 results for "wireless mouse"\nSort by:\n...'
+# Extract just the count: count_text.split('\n')[0]
+```
+
+## CAPTCHA Detection
+
+No CAPTCHA was encountered during testing with a logged-in Chrome session. To detect defensively:
+
+```python
+def check_captcha():
+    text = js("document.body.innerText.slice(0,500)") or ""
+    url  = page_info()["url"]
+    return (
+        "captcha" in text.lower()
+        or "enter the characters" in text.lower()
+        or "sorry, we just need to make sure" in text.lower()
+        or "captcha" in url.lower()
+        or "validateCaptcha" in url
+    )
+
+if check_captcha():
+    raise RuntimeError("Amazon CAPTCHA hit — stop and notify user")
+```
+
+Amazon may serve a CAPTCHA on fresh/anonymous sessions. Using the browser's existing logged-in session avoids this in practice.
+
+## Gotchas
+
+- **`goto_url()` silent failure**: On first visit, use `new_tab(url)` instead. After the tab is on Amazon, `goto_url()` works.
+- **`.zg-item-immersion` is gone**: Best Sellers page uses CSS module classes (obfuscated). Use `[data-asin]` + `img[alt]` for title.
+- **`.a-size-base.s-underline-text` is unreliable for review count**: On sponsored results it shows unrelated text (e.g. "Xbox"). Use `[aria-label*="ratings"]` instead.
+- **`#priceblock_ourprice` is legacy**: Returns `null` on modern pages. Construct from `.a-price-whole` + `.a-price-fraction`.
+- **Sponsored results appear first**: First 2-3 results are almost always `is_sponsored: true`. Filter them out with `!el.querySelector('.puis-sponsored-label-text')` when you need organic results.
+- **`data-asin` can be empty string on non-product rows**: Filter with `.filter(r => r.asin)`.
+- **Price split DOM**: `.a-price-whole` innerText includes a trailing `\n.` — strip it: `.replace(/[\n.]/g,'')`.
+- **ASIN from URL**: Use `/dp/([A-Z0-9]{10})/` regex on the product URL. `data-asin` on search results is always the canonical ASIN.
+- **`?th=1` redirect**: Amazon appends `?th=1` (and sometimes `?psc=1`) to product URLs after redirect. This is normal — `input[name="ASIN"]` always has the clean ASIN.
+- **Wait 2s after `wait_for_load()`**: Amazon search results load the listing cards asynchronously. `readyState=complete` fires before cards render. A hard 2s wait is required.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/archive-org/scraping.md

@@ -0,0 +1,341 @@
+# Internet Archive / Wayback Machine — Scraping & Data Extraction
+
+`https://archive.org` / `https://web.archive.org` — all public data, no auth required. Every workflow here is pure `http_get` — no browser needed.
+
+## Do this first
+
+**Use the CDX API for anything Wayback-related — it is the reliable workhorse. The Wayback Availability API (`/wayback/available`) is known to return empty `archived_snapshots` even for well-archived URLs and should not be used as a primary mechanism.**
+
+```python
+import json
+
+# Find snapshots of any URL — primary entry point for Wayback data
+r = http_get(
+    "https://web.archive.org/cdx/search/cdx"
+    "?url=iana.org&output=json&limit=5"
+    "&fl=timestamp,original,statuscode,mimetype,length",
+    timeout=40.0
+)
+rows = json.loads(r)
+headers = rows[0]   # ['timestamp', 'original', 'statuscode', 'mimetype', 'length']
+for row in rows[1:]:
+    ts, orig, status, mime, length = row
+    snap_url = f"https://web.archive.org/web/{ts}/{orig}"
+    print(f"{ts}  {status}  {snap_url}")
+```
+
+For item metadata (books, video, audio, software), go straight to:
+
+```python
+data = json.loads(http_get("https://archive.org/metadata/{identifier}", timeout=30.0))
+```
+
+## Common workflows
+
+### Find the nearest archived snapshot to a target date
+
+```python
+import json
+
+# CDX sort=closest returns the single snapshot nearest to the given timestamp
+r = http_get(
+    "https://web.archive.org/cdx/search/cdx"
+    "?url=iana.org&output=json&limit=1"
+    "&fl=timestamp,original,statuscode"
+    "&closest=20230601120000&sort=closest",
+    timeout=60.0   # CDX can be slow — always use timeout >= 40s
+)
+rows = json.loads(r)
+# rows[0] = header, rows[1] = closest snapshot
+ts, orig, status = rows[1]
+snap_url = f"https://web.archive.org/web/{ts}/{orig}"
+# Result: ts='20230601114925', orig='https://www.iana.org/', status='200'
+# snap_url: https://web.archive.org/web/20230601114925/https://www.iana.org/
+```
+
+Timestamp format is always 14-digit `YYYYMMDDHHMMSS`. Pass any prefix — `20230601` (day), `202306` (month), `2023` (year) — and CDX will match.
+
+### List all monthly snapshots for a URL (collapsed)
+
+```python
+import json
+
+r = http_get(
+    "https://web.archive.org/cdx/search/cdx"
+    "?url=iana.org&output=json"
+    "&collapse=timestamp:6"   # :6 = dedupe by YYYYMM (one per month)
+    "&from=20230101&to=20231231"
+    "&fl=timestamp,original",
+    timeout=60.0
+)
+rows = json.loads(r)
+# rows[0] = header ['timestamp', 'original']
+# rows[1:] = one row per month:
+# ['20230101103807', 'https://www.iana.org/']
+# ['20230201144829', 'https://www.iana.org/']
+# ...12 rows for 2023
+
+for ts, orig in rows[1:]:
+    print(f"{ts[:4]}-{ts[4:6]}  https://web.archive.org/web/{ts}/{orig}")
+```
+
+`collapse=timestamp:N` deduplicates by the first N digits of the timestamp:
+- `:4` = one per year, `:6` = one per month, `:8` = one per day
+
+### List snapshots for an entire domain (all pages)
+
+```python
+import json
+
+# matchType=domain captures all URLs under that domain
+r = http_get(
+    "https://web.archive.org/cdx/search/cdx"
+    "?url=iana.org&matchType=domain&output=json"
+    "&limit=10&fl=timestamp,original,statuscode"
+    "&collapse=timestamp:8",  # one capture per URL per day
+    timeout=60.0
+)
+rows = json.loads(r)
+for row in rows[1:]:
+    print(row)
+# ['19971210061738', 'http://www.iana.org:80/', '200']
+# ['19980211065537', 'http://www.iana.org:80/', '200']
+# ...
+```
+
+`matchType` options: `exact` (default), `prefix` (URL + subpaths), `host` (all subdomains), `domain` (host + all subdomains).
+
+### Filter snapshots by prefix path
+
+```python
+import json
+
+# All archived pages under /domains/ path
+r = http_get(
+    "https://web.archive.org/cdx/search/cdx"
+    "?url=iana.org/domains/&matchType=prefix&output=json"
+    "&limit=5&fl=timestamp,original,statuscode",
+    timeout=40.0
+)
+rows = json.loads(r)
+for row in rows[1:]:
+    print(row)
+# ['20080509121811', 'http://www.iana.org/domains/', '200']
+# ['20080704174537', 'http://iana.org/domains/', '200']
+```
+
+### Paginate CDX results with resumeKey
+
+```python
+import json
+from urllib.parse import quote
+
+def cdx_all_snapshots(url, fl="timestamp,original,statuscode", page_size=500):
+    """Iterate all CDX records for a URL, yielding rows (excluding header)."""
+    base = (
+        f"https://web.archive.org/cdx/search/cdx"
+        f"?url={quote(url, safe='')}&output=json"
+        f"&fl={fl}&limit={page_size}&showResumeKey=true"
+    )
+    resume_key = None
+    while True:
+        endpoint = base if resume_key is None else f"{base}&resumeKey={quote(resume_key)}"
+        rows = json.loads(http_get(endpoint, timeout=60.0))
+        # rows structure with showResumeKey=true:
+        # [header, row1, row2, ..., [], [resume_key_string]]
+        # The second-to-last row is [] (separator), last row is [resume_key]
+        has_resume = len(rows) >= 2 and rows[-1] != [] and rows[-2] == []
+        data_rows = rows[1:-2] if has_resume else rows[1:]
+        for row in data_rows:
+            yield row
+        if not has_resume:
+            break
+        resume_key = rows[-1][0]
+
+for row in cdx_all_snapshots("iana.org", fl="timestamp,original"):
+    ts, orig = row
+    # process...
+```
+
+### Retrieve the actual archived page
+
+```python
+# Direct snapshot URL: /web/{14-digit-timestamp}/{original-url}
+snap_url = "https://web.archive.org/web/19971210061738/http://www.iana.org:80/"
+content = http_get(snap_url, timeout=30.0)
+# Returns the archived HTML with Wayback toolbar injected at top
+# The toolbar is inside <!-- BEGIN WAYBACK TOOLBAR INSERT --> comments
+
+# The calendar view URL pattern (for browser navigation, not http_get):
+# https://web.archive.org/web/20230101000000*/python.org
+# The * tells Wayback to show the calendar — returns HTML, not raw page
+```
+
+### Item metadata (books, video, audio, software, collections)
+
+```python
+import json
+from urllib.parse import quote
+
+identifier = "HardWonWisdomTrailer"
+data = json.loads(http_get(f"https://archive.org/metadata/{identifier}", timeout=30.0))
+
+# Top-level keys:
+# alternate_locations, created, d1, d2, dir, files, files_count,
+# is_collection, item_last_updated, item_size, metadata, server, uniq, workable_servers
+
+meta = data['metadata']
+# Common metadata fields (not all present on every item):
+print(meta.get('identifier'))   # 'HardWonWisdomTrailer'
+print(meta.get('title'))        # 'Hard Won Wisdom Trailer'
+print(meta.get('mediatype'))    # 'movies' | 'texts' | 'audio' | 'software' | 'collection'
+print(meta.get('creator'))      # 'jakemauz'
+print(meta.get('date'))         # '2017-02-18'
+print(meta.get('description'))  # HTML string — strip tags if needed
+print(meta.get('subject'))      # str OR list of str depending on item
+print(meta.get('publicdate'))   # '2017-02-18 11:51:16'
+print(meta.get('collection'))   # parent collection identifier
+
+files = data['files']
+# Each file entry:
+# name, source ('original'|'derivative'|'metadata'), format, size (bytes as str),
+# md5, sha1, crc32, mtime
+# For video/audio: length (seconds as str), height, width
+# For derivative: original (name of source file)
+
+# Find the primary original file
+orig_files = [f for f in files if f.get('source') == 'original']
+# orig_files[0]: {'name': 'Hard-won wisdom trailer.mp4', 'source': 'original',
+#  'format': 'MPEG4', 'size': '7532153', 'length': '94.13',
+#  'height': '360', 'width': '640', 'md5': 'aaeebe0481...', ...}
+
+# Build download URL — two equivalent forms:
+server = data['server']      # 'ia601405.us.archive.org'
+dir_path = data['dir']       # '/2/items/HardWonWisdomTrailer'
+fname = orig_files[0]['name']
+from urllib.parse import quote as urlquote
+# Form 1: direct storage server (fastest)
+url1 = f"https://{server}{dir_path}/{urlquote(fname)}"
+# Form 2: standard redirect URL (always works, resolved by CDN)
+url2 = f"https://archive.org/download/{identifier}/{urlquote(fname)}"
+# Both confirmed status 200, Content-Type: video/mp4
+```
+
+### Search items (books, audio, video, software)
+
+```python
+import json
+
+# advancedsearch.php is the correct API — /search returns HTML
+r = http_get(
+    "https://archive.org/advancedsearch.php"
+    "?q=artificial+intelligence+AND+mediatype:texts"
+    "&fl[]=identifier&fl[]=title&fl[]=creator&fl[]=date&fl[]=downloads"
+    "&rows=5&sort[]=downloads+desc&output=json",
+    timeout=30.0
+)
+data = json.loads(r)
+# data['responseHeader']['status'] = 0 (success)
+# data['responseHeader']['QTime'] = query time ms
+# data['response']['numFound'] = 25911 (total matches)
+# data['response']['start'] = 0 (offset)
+# data['response']['docs'] = list of item dicts
+
+resp = data['response']
+print(f"Total: {resp['numFound']}, showing: {len(resp['docs'])}")
+for doc in resp['docs']:
+    print(f"  {doc['identifier']}  {doc.get('title', '')[:50]}")
+    # doc fields are only present if they have values — always use .get()
+```
+
+Pagination: use `start=` offset (not `page=`). Max `rows=` is not documented but 100 works reliably.
+
+### Search with all supported parameters
+
+```python
+import json
+
+r = http_get(
+    "https://archive.org/advancedsearch.php"
+    "?q=machine+learning+AND+mediatype:texts"  # Lucene query syntax
+    "&fl[]=identifier&fl[]=title&fl[]=date&fl[]=year"
+    "&fl[]=creator&fl[]=subject&fl[]=description&fl[]=downloads"
+    "&rows=3"
+    "&start=0"               # pagination offset
+    "&sort[]=date+desc"      # sort field + direction
+    "&output=json",
+    timeout=30.0
+)
+data = json.loads(r)
+# Confirmed fields in fl[]:
+# identifier, title, date, year, creator, subject, description,
+# downloads, mediatype, collection, language, avg_rating, num_reviews
+
+# mediatype values: texts, audio, movies, software, image, etree, data, collection, account
+# Sort fields: date, downloads, avg_rating, num_reviews, publicdate, addeddate
+```
+
+## API reference
+
+| Endpoint | What it returns | Auth |
+|---|---|---|
+| `web.archive.org/cdx/search/cdx?url=...&output=json` | Snapshot index: all captures of a URL | None |
+| `archive.org/wayback/available?url=...` | Nearest snapshot (DEGRADED — see gotchas) | None |
+| `archive.org/metadata/{identifier}` | Item metadata + files list | None |
+| `archive.org/advancedsearch.php?q=...&output=json` | Full-text + metadata search | None |
+| `archive.org/download/{identifier}/{filename}` | Direct file download | None |
+| `web.archive.org/web/{timestamp}/{url}` | Archived page HTML | None |
+
+## CDX field reference
+
+The CDX API returns a JSON array of arrays. The first row is always the header when `output=json`.
+
+| Field | Description | Example |
+|---|---|---|
+| `urlkey` | SURT-format URL (reversed domain, path in parens) | `org,iana)/` |
+| `timestamp` | Capture time, 14-digit `YYYYMMDDHHMMSS` | `19971210061738` |
+| `original` | Original crawled URL (exact, including port) | `http://www.iana.org:80/` |
+| `mimetype` | Content-Type of the archived response | `text/html` |
+| `statuscode` | HTTP status at crawl time | `200` |
+| `digest` | SHA-1 of response body, base32-encoded | `I4YBMQ6PHPWE2TD6TIXNWHZB6MXRNTSR` |
+| `length` | Content length in bytes (as string) | `1418` |
+
+Default `fl=` when omitted: `urlkey,timestamp,original,mimetype,statuscode,digest,length` (all 7 fields in that order).
+
+## Rate limits
+
+No auth, no API key. In practice:
+- CDX API: **intermittently slow** — individual queries time out at 20s and succeed at 40–60s. Always use `timeout=40.0` minimum. 3 rapid sequential CDX calls in ~10s completed; 10 rapid calls produced 3 timeouts.
+- Metadata API: Fast and reliable — 5 sequential calls completed in 3.0s with no errors.
+- Search API: Fast — typically responds in 30–65ms (`QTime` in response header).
+- No documented per-second or per-day limits. Archive.org's policy is to be respectful: add `time.sleep(1)` between CDX calls in loops.
+
+## Gotchas
+
+- **CDX times out — always set `timeout=40.0` or higher.** The default 20s is often too short for CDX. Metadata and search APIs are fine at 20–30s. CDX slowness is backend-side and unpredictable; add retry logic for production use.
+
+- **Wayback Availability API is unreliable.** `GET /wayback/available?url=iana.org` returns `{"url": "iana.org", "archived_snapshots": {}}` even for URLs confirmed archived via CDX. Tested 2026-04-18 across many URLs and timestamp combinations — consistently empty. Use `CDX ?sort=closest&limit=1` instead (confirmed working).
+
+- **CDX first row is always the header when `output=json`.** `rows[0]` is `['timestamp', 'original', ...]`, not a data row. Always slice `rows[1:]` for data. When `showResumeKey=true`, the last two rows are `[]` (separator) and `['<resume_key_string>']`.
+
+- **CDX `fl=` must match exactly what you iterate.** If you request `&fl=timestamp,original` you get 2-element rows; forgetting a field breaks destructuring. When in doubt, omit `fl=` entirely and get all 7 fields.
+
+- **`output=json` is required — there is no default JSON mode.** Omitting `output=json` returns space-separated text. `output=text` also works and is slightly faster for simple queries.
+
+- **`timestamp` is a string, not an integer.** Even in JSON, CDX returns all fields as strings: `'1418'` not `1418`, `'200'` not `200`. Cast explicitly: `int(row[4])`, `int(row[6])`.
+
+- **The `original` field preserves port numbers.** Old crawls captured `http://www.iana.org:80/` — the `:80` is part of the URL. When building a playback URL, use `original` verbatim: `f"https://web.archive.org/web/{ts}/{orig}"` works correctly with the port included.
+
+- **Metadata `{}` means the item doesn't exist or is private.** `http_get("https://archive.org/metadata/nonexistent")` returns `'{}'` (2-byte response) with HTTP 200. Always check `if not data` or `if not data.get('metadata')` before accessing fields.
+
+- **Metadata `subject` can be a string or a list.** When a single subject tag is set, the API returns `"subject": "short film"`. When multiple, it returns `"subject": ["short film", "spoken word"]`. Normalize with: `subjects = [meta['subject']] if isinstance(meta.get('subject'), str) else meta.get('subject', [])`.
+
+- **File `size` and `length` are strings, not numbers.** `files[0]['size']` is `'7532153'` (bytes). `files[0]['length']` is `'94.13'` (seconds for video/audio). Cast with `int()` and `float()` respectively.
+
+- **Use `archive.org/download/` not the raw storage server URL for reliability.** The raw URL (`ia601405.us.archive.org/2/items/...`) is faster but server-specific. `archive.org/download/{id}/{file}` redirects to the correct storage node and remains stable as items migrate.
+
+- **`/search?output=json` returns HTML, not JSON.** The `/search` endpoint is a React SPA — it ignores `output=json`. Always use `advancedsearch.php` for programmatic access.
+
+- **`collapse=timestamp:6` gives one row per month, but it keeps the FIRST capture of that month.** If you want the last, you'd need to reverse and re-collapse, or fetch all and filter client-side. The `collapse` parameter de-duplicates by truncating the timestamp to N digits and keeping the first matching row.
+
+- **CDX `from=` / `to=` accept partial timestamps.** `from=20230101` means `20230101000000`. `to=20231231` means `20231231000000` (exclusive). To include all of 2023, use `to=20240101`.