catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/shopify-admin/embedded-apps.md

@@ -0,0 +1,72 @@
+# Shopify embedded apps run in iframes
+
+Every Shopify app surfaced in the admin (first-party like Knowledge Base, third-party like Okendo) renders inside a sandboxed iframe. Your top-level `document` queries find the Shopify chrome (sidebar, header, search bar) but **none of the app's UI**.
+
+## How to target the iframe
+
+```python
+from helpers import iframe_target, js, type_text
+
+# 1. Find the iframe by URL substring
+tid = iframe_target("qa-pairs-app")  # Knowledge Base App
+
+# 2. Run JS inside the iframe by passing target_id
+result = js("""
+(() => {
+  const button = Array.from(document.querySelectorAll('button')).find(b => b.textContent.trim() === 'Add FAQ');
+  if (button) { button.click(); return {clicked: true}; }
+  return {clicked: false};
+})()
+""", target_id=tid)
+```
+
+## Finding the URL substring
+
+The iframe's URL contains the app slug. Run:
+
+```python
+import json
+for t in cdp("Target.getTargets")["targetInfos"]:
+    if t["type"] == "iframe" and "shopify" in t.get("url", "").lower():
+        print(t["url"])
+```
+
+Then pick a substring unique to your target app.
+
+## Known Shopify app iframe slugs
+
+| App | iframe URL substring |
+|---|---|
+| Shopify Knowledge Base (qa-pairs-app) | `qa-pairs-app` |
+| Shopify Online Store editor | `online-store-web.shopifyapps.com` |
+| Shopify Hydrogen Storefront | `hydrogen-storefronts` (or similar — verify) |
+
+Add to this table when you discover new ones.
+
+## Why iframes
+
+Shopify uses App Bridge to embed third-party apps with isolation. Your top-level page CAN'T directly access app DOM for security reasons — you need iframe targeting (which the harness does via CDP `Target.attachToTarget`).
+
+## Coordinate clicks vs JS clicks
+
+Coordinate clicks (`click(x, y)`) pass through iframes at the compositor level — they work. But JS clicks scoped to the iframe target are more reliable for routine button taps because:
+
+- Element text content is stable across UI redesigns
+- DPR scaling on retina is automatic
+- React event handlers are guaranteed to fire (vs. CDP mouse events which sometimes hit a transparent layer above the button)
+
+## Gotcha — multiple iframes from same app
+
+The Online Store editor renders the storefront preview AND the editor toolbar in two separate iframes. Pick the right one by URL substring; don't assume the first match is correct.
+
+```python
+# WRONG — picks first match
+tid = iframe_target("online-store-web")
+
+# RIGHT — disambiguate
+for t in cdp("Target.getTargets")["targetInfos"]:
+    url = t.get("url", "")
+    if "online-store-web" in url and "editor" in url:
+        tid = t["targetId"]
+        break
+```

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/shopify-admin/knowledge-base.md

@@ -0,0 +1,109 @@
+# Shopify Knowledge Base App — automating FAQ entries
+
+The Knowledge Base App (Shopify Winter '26 Edition) lets merchants control how AI agents (ChatGPT, Perplexity, Claude, Copilot, Gemini) answer questions about their brand. Each entry is a Question / Answer pair. The app currently has no public API and is English-only as of Winter '26 — browser automation is the canonical path.
+
+## URL pattern
+
+```
+https://admin.shopify.com/store/<store-handle>/apps/shopify-knowledge-base/app
+```
+
+Sub-routes:
+- `/app` — overview (FAQ list, top unanswered questions, query log)
+- `/app/new` — Add FAQ form
+- `/app/pairs/<id>` — entry detail / edit
+
+## Iframe slug
+
+The app runs at iframe URL containing `qa-pairs-app`:
+
+```python
+tid = iframe_target("qa-pairs-app")
+```
+
+## Adding a single FAQ
+
+See `polaris-inputs.md` for the full canonical pattern. Quick version:
+
+```python
+def add_faq(question, answer):
+    tid = iframe_target("qa-pairs-app")
+    # focus question input via JS, type via CDP, focus answer, type, click Save
+    # poll URL for /pairs/<id> success signal
+```
+
+## Batching multiple FAQs
+
+After saving an entry, the success page shows "FAQ created. Add another FAQ" link. Click it via JS to skip navigating back to overview:
+
+```python
+def click_add_another():
+    tid = iframe_target("qa-pairs-app")
+    js("""
+    (() => {
+      const link = Array.from(document.querySelectorAll('a, button'))
+        .find(x => x.textContent.trim() === 'Add another FAQ');
+      if (link) link.click();
+    })()
+    """, target_id=tid)
+```
+
+Loop:
+
+```python
+ENTRIES = [(q1, a1), (q2, a2), ...]
+for q, a in ENTRIES:
+    click_add_another()
+    time.sleep(1.5)  # wait for form to render
+    ok, info = add_faq(q, a)
+    print(f"{q[:40]} -> {ok} ({info})")
+    if not ok: break
+```
+
+## Brand voice — what to put in answers
+
+This is application-specific (depends on the merchant). For JING the rule was Aesop founder-letter tone — sentence case, no exclamation points, "JING" not "we", specific over generic.
+
+The Shopify guidance "Provide a brief answer in 1 or 2 sentences" is a soft hint. The textarea accepts longer text and AI agents prefer specific multi-sentence answers. Aim for 2-4 short sentences with concrete details.
+
+## What to put in the Knowledge Base
+
+Categories that materially shape AI agent answers about your brand:
+
+1. **Brand voice / DNA** — "What is your brand?" / "What's your tone?"
+2. **Specs** — exact materials, dimensions, weights, sizes (NOT marketing prose)
+3. **Comparisons** — "How does X compare to <competitor>?" with concrete differences
+4. **Policies** — returns, shipping, care, warranty, contact (in brand voice)
+5. **Origin** — founder, where made, why brand exists
+6. **Limitations** — what you DON'T do (V1 scope, US-only, etc.) — agents that hallucinate availability hurt conversion
+
+Skip: anything marketing-speak. The Knowledge Base is for **truth, in voice**, not pitch copy.
+
+## Top unanswered questions
+
+The overview shows up to 7 "Top unanswered questions" Shopify auto-detected from query logs. **Answer these first** — they're real shopper queries hitting your store right now. Once answered, the section empties.
+
+## Query log
+
+`/admin/apps/shopify-knowledge-base/app/queries` (or "Query log" in app sidebar) shows what shoppers actually asked AI agents about your brand. Read weekly. New patterns become new FAQ entries.
+
+## Verifying entries surface in AI
+
+After adding an entry, allow 24 hours for AI provider indexing, then test:
+
+- ChatGPT: "Tell me about <your brand>'s return policy" → check if your exact wording surfaces
+- Perplexity: same
+- Claude: "Compare <your brand> vs <competitor>" → see if your comparison framing appears
+
+If the answer doesn't surface, the entry might be too long, too vague, or contradicted by another source (your homepage, an outdated blog post). Tighten the answer.
+
+## Limits
+
+As of Winter '26 Edition:
+- English-only
+- No bulk import / CSV upload
+- No API for read or write
+- Each entry maximum ~500 words (soft cap; UI shows guidance "1 or 2 sentences")
+- No version history visible to the merchant
+
+Watch Shopify changelogs for API exposure — likely in Spring '26 or Summer '26 Edition. When it ships, switch to API-driven population.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/shopify-admin/polaris-inputs.md

@@ -0,0 +1,137 @@
+# Polaris React inputs require CDP-native keystrokes
+
+Shopify admin uses Polaris (their design system). Until January 2026 it was React-based. Polaris React text inputs and textareas are controlled components that **reject the standard "React-friendly" synthetic value setter pattern.**
+
+## The trap
+
+This pattern looks like it works — the field's `value` shows the right text:
+
+```js
+const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value').set;
+setter.call(inputEl, "my text");
+inputEl.dispatchEvent(new Event('input', { bubbles: true }));
+```
+
+But the **Save / Submit button stays disabled**. Polaris's onChange handler reads from React's internal state, which the synthetic event chain doesn't fully update.
+
+## What works
+
+CDP-native keystrokes via `Input.insertText`:
+
+```python
+from helpers import js, type_text
+
+# 1. Focus the input via JS — this works fine
+js("""
+(() => {
+  const input = Array.from(document.querySelectorAll('input[type="text"], input:not([type])'))
+    .find(x => { const r = x.getBoundingClientRect(); return r.width > 100 && r.height > 0; });
+  if (input) input.focus();
+})()
+""", target_id=tid)
+
+# 2. Type via CDP — fires Input.insertText which is the lowest-level
+#    text-entry signal. React's controlled-input subscriber catches this.
+type_text("My question text")
+```
+
+For textareas, same pattern with `document.querySelectorAll('textarea')`.
+
+## Full add-FAQ pattern (Knowledge Base App)
+
+```python
+import time
+from helpers import iframe_target, js, type_text, page_info, screenshot
+
+def add_faq(question: str, answer: str) -> tuple[bool, str]:
+    tid = iframe_target("qa-pairs-app")
+
+    # 1. Make sure the form is rendered
+    for _ in range(15):
+        ready = js("""
+        (() => {
+          const i = Array.from(document.querySelectorAll('input[type="text"], input:not([type])'))
+            .find(x => { const r = x.getBoundingClientRect(); return r.width > 100; });
+          const t = Array.from(document.querySelectorAll('textarea'))
+            .find(x => { const r = x.getBoundingClientRect(); return r.width > 100; });
+          if (i && t) { i.focus(); return true; }
+          return false;
+        })()
+        """, target_id=tid)
+        if ready: break
+        time.sleep(0.3)
+
+    # 2. Type question (input has focus from step 1)
+    type_text(question)
+    time.sleep(0.2)
+
+    # 3. Focus textarea, type answer
+    js("""
+    (() => {
+      const t = Array.from(document.querySelectorAll('textarea'))
+        .find(x => { const r = x.getBoundingClientRect(); return r.width > 100; });
+      if (t) t.focus();
+    })()
+    """, target_id=tid)
+    time.sleep(0.2)
+    type_text(answer)
+    time.sleep(0.4)
+
+    # 4. Click Save (now enabled because Polaris saw real keystrokes)
+    saved = js("""
+    (() => {
+      const btn = Array.from(document.querySelectorAll('button')).find(b => b.textContent.trim() === 'Save');
+      if (!btn || btn.disabled) return {clicked: false, disabled: btn?.disabled};
+      btn.click();
+      return {clicked: true};
+    })()
+    """, target_id=tid)
+    if not saved.get("clicked"):
+        return False, "save_button_disabled"
+
+    # 5. Poll URL for save success — Shopify redirects to /pairs/<id>
+    for _ in range(20):
+        time.sleep(0.3)
+        url = page_info().get("url", "")
+        if "/pairs/" in url and "/new" not in url:
+            return True, url.split("/pairs/")[-1]
+    return False, "save_timeout"
+```
+
+## Why this works
+
+Polaris React components subscribe to native `inputType` events (e.g., `insertText` from IME / accessibility tools / paste). The synthetic React-friendly setter fires `input` events but skips the lower-level `inputType` signal that Polaris validates against to enable Save buttons.
+
+CDP `Input.insertText` (which the harness's `type_text()` calls) emits the full native event chain, including `inputType: 'insertText'`, which React catches via its synthetic event system.
+
+## Polaris Web Components (post January 2026)
+
+The `polaris-react` repo was archived January 6, 2026. New Polaris is web-component-based. For new admin surfaces (Catalog Mapping, parts of Settings), the pattern shifts:
+
+```js
+// Web components expose value setter on the element itself
+const wc = document.querySelector('s-text-field');
+wc.value = 'my text';
+wc.dispatchEvent(new CustomEvent('input', { bubbles: true, detail: { value: 'my text' } }));
+```
+
+But until Shopify completes the migration (probably late 2026), **always test the React pattern first** — most legacy surfaces still use it.
+
+## How to know which pattern to use
+
+Screenshot the form first. Then JS-introspect:
+
+```js
+// Check if React-based (Polaris-* class names) or web-component-based (s-* tags)
+const hasReact = document.querySelector('[class*="Polaris-"]');
+const hasWC = document.querySelector('s-text-field, s-button, s-textarea');
+return { hasReact: !!hasReact, hasWC: !!hasWC };
+```
+
+If both, lean web component (the surface is mid-migration and the WC will be authoritative).
+
+## Avoid
+
+- Coordinate-based typing via `Input.dispatchKeyEvent` keypress-by-keypress — slower, more brittle, no real benefit over `Input.insertText`
+- `el.value = 'x'` without the setter prototype trick — won't even fill the visible field on Polaris React
+- `dispatchEvent(new Event('change', ...))` only — Polaris listens for `input`, not `change`, on text fields

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/soundcloud/scraping.md

@@ -0,0 +1,362 @@
+# SoundCloud — Data Extraction
+
+Field-tested against soundcloud.com on 2026-04-18.
+No authentication required for any approach documented here. All code uses `http_get` (pure HTTP, no browser).
+
+---
+
+## Approach 1 (Fastest): oEmbed API — No Auth, No Client ID
+
+`https://soundcloud.com/oembed?url=<resource_url>&format=json`
+
+Returns JSON in ~0.3s. Works for **tracks, playlists/sets, and user profiles**. No key required.
+
+```python
+from helpers import http_get
+import json
+
+def soundcloud_oembed(resource_url):
+    """Fetch oEmbed metadata for any public SoundCloud URL.
+
+    Works for:
+      - https://soundcloud.com/{user}/{track-slug}
+      - https://soundcloud.com/{user}/sets/{playlist-slug}
+      - https://soundcloud.com/{user}
+    """
+    url = f"https://soundcloud.com/oembed?url={resource_url}&format=json"
+    return json.loads(http_get(url))
+
+# Track
+track = soundcloud_oembed("https://soundcloud.com/forss/flickermood")
+# {
+#   "version": 1.0,
+#   "type": "rich",
+#   "provider_name": "SoundCloud",
+#   "provider_url": "https://soundcloud.com",
+#   "height": 400,
+#   "width": "100%",
+#   "title": "Flickermood by Forss",
+#   "description": "From the Soulhack album...",
+#   "thumbnail_url": "https://i1.sndcdn.com/artworks-000067273316-smsiqx-t500x500.jpg",
+#   "html": "<iframe width=\"100%\" height=\"400\" scrolling=\"no\" frameborder=\"no\" src=\"https://w.soundcloud.com/player/?visual=true&url=...\">",
+#   "author_name": "Forss",
+#   "author_url": "https://soundcloud.com/forss"
+# }
+
+# Playlist/set
+pl = soundcloud_oembed("https://soundcloud.com/forss/sets/soulhack")
+# title="Soulhack by Forss", description="My 2003 debut album...", height=450
+
+# User profile
+user = soundcloud_oembed("https://soundcloud.com/forss")
+# title="Forss", description="Artist & Founder SoundCloud", height=450
+```
+
+### oEmbed fields
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `title` | str | "{Track Title} by {Artist}" for tracks, "{Name}" for users |
+| `author_name` | str | Artist/user display name |
+| `author_url` | str | Profile URL |
+| `thumbnail_url` | str | Artwork at 500×500px (t500x500) |
+| `description` | str | Track/profile description (may contain HTML entities) |
+| `html` | str | Embed iframe for the SoundCloud player widget |
+| `height` | int | 400 for tracks, 450 for playlists and users |
+| `width` | str | Always `"100%"` |
+
+---
+
+## Approach 2: Page Hydration (`__sc_hydration`) — Rich Metadata, No Client ID
+
+Every SoundCloud page embeds a JSON array in a `<script>` tag as `window.__sc_hydration`. This contains full API-grade metadata with no key required.
+
+```python
+from helpers import http_get
+import json, re
+
+def extract_hydration(page_url):
+    """Extract __sc_hydration JSON from any SoundCloud page."""
+    html = http_get(page_url)
+    match = re.search(r'window\.__sc_hydration\s*=\s*(\[.*?\]);\s*<', html, re.DOTALL)
+    if not match:
+        return []
+    return json.loads(match.group(1))
+
+def get_hydration_by_type(page_url, hydratable):
+    """Get the 'data' dict for a specific hydratable type."""
+    for obj in extract_hydration(page_url):
+        if obj.get('hydratable') == hydratable:
+            return obj.get('data')
+    return None
+
+# Track page — hydration key is 'sound'
+track = get_hydration_by_type("https://soundcloud.com/forss/flickermood", "sound")
+# track['id']             = 293
+# track['title']          = "Flickermood"
+# track['playback_count'] = 962685
+# track['likes_count']    = 2592
+# track['duration']       = 213886  (milliseconds)
+# track['genre']          = "Electronic"
+# track['created_at']     = "2007-09-22T14:45:46Z"
+# track['artwork_url']    = "https://i1.sndcdn.com/artworks-000067273316-smsiqx-large.jpg"
+# track['waveform_url']   = "https://wave.sndcdn.com/cWHNerOLlkUq_m.json"
+# track['streamable']     = True
+# track['downloadable']   = True
+# track['license']        = "all-rights-reserved"
+# track['tag_list']       = "downtempo"
+# track['urn']            = "soundcloud:tracks:293"
+# track['media']          = {'transcodings': [...]}  (HLS/progressive stream URLs — need auth)
+# track['user']           = {full user object nested}
+
+# User page — hydration key is 'user'
+user = get_hydration_by_type("https://soundcloud.com/forss", "user")
+# user['id']               = 183
+# user['username']         = "Forss"
+# user['full_name']        = "Eric Quidenus-Wahlforss"
+# user['followers_count']  = 132203
+# user['track_count']      = 26
+# user['verified']         = True
+# user['city']             = "Berlin"
+# user['country_code']     = "DE"
+# user['description']      = "Artist & Founder SoundCloud"
+# user['creator_subscription'] = {'product': {'id': 'creator-pro-unlimited'}}
+# user['badges']           = {'pro_unlimited': True, 'verified': True}
+
+# Playlist/set page — hydration key is 'playlist'
+playlist = get_hydration_by_type("https://soundcloud.com/forss/sets/soulhack", "playlist")
+# playlist['id']           = 18
+# playlist['title']        = "Soulhack"
+# playlist['track_count']  = 11
+# playlist['tracks']       = [full track objects list]
+# playlist['is_album']     = True/False
+# playlist['genre']        = "Electronic"
+```
+
+### All hydration keys on a typical page
+
+| `hydratable` | Content |
+|---|---|
+| `sound` | Full track object (on track pages) |
+| `playlist` | Full playlist + all tracks (on set pages) |
+| `user` | Full user object (on any page with a profile) |
+| `apiClient` | `{'id': '<client_id>', 'isExpiring': False}` — the client_id |
+| `geoip` | Viewer country/city/coordinates |
+| `features` | Feature flags dict |
+| `anonymousId` | Session tracking ID (not useful) |
+
+---
+
+## Approach 3: API v2 — Full Query Power (Requires Client ID)
+
+The `client_id` lives in every page's `__sc_hydration` under the `apiClient` key. It is **stable across all pages and sessions** — extract once and reuse.
+
+```python
+from helpers import http_get
+import json, re
+
+def get_client_id(page_url="https://soundcloud.com"):
+    """Extract client_id from any SoundCloud page's __sc_hydration."""
+    html = http_get(page_url)
+    match = re.search(r'window\.__sc_hydration\s*=\s*(\[.*?\]);\s*<', html, re.DOTALL)
+    if not match:
+        raise ValueError("No hydration found")
+    for obj in json.loads(match.group(1)):
+        if obj.get('hydratable') == 'apiClient':
+            return obj['data']['id']
+    raise ValueError("apiClient not found in hydration")
+
+CLIENT_ID = get_client_id()  # "efg2kjLJnAJpInbN6P3hsHzispI1SKQH" (example — extract fresh)
+
+def sc_api(path, **params):
+    """Call api-v2.soundcloud.com. Returns parsed JSON."""
+    params['client_id'] = CLIENT_ID
+    qs = "&".join(f"{k}={v}" for k, v in params.items())
+    return json.loads(http_get(f"https://api-v2.soundcloud.com/{path}?{qs}"))
+```
+
+### Resolve any URL to a resource
+
+```python
+# Resolve a permalink URL to get its resource with full metadata
+track = sc_api("resolve", url="https://soundcloud.com/forss/flickermood")
+# Returns: {'kind': 'track', 'id': 293, 'title': 'Flickermood', ...}
+
+user = sc_api("resolve", url="https://soundcloud.com/forss")
+# Returns: {'kind': 'user', 'id': 183, 'username': 'Forss', ...}
+```
+
+### Track lookup
+
+```python
+# Single track by numeric ID
+track = sc_api("tracks/293")
+
+# Bulk track lookup (comma-separated IDs — returns list)
+tracks = sc_api("tracks", ids="293,290,48031525")
+# Returns a JSON array directly (not wrapped in 'collection')
+for t in tracks:
+    print(t['id'], t['title'], t['playback_count'])
+```
+
+### Search
+
+```python
+# Tracks
+results = sc_api("search/tracks", q="jazz", limit=20)
+# results['collection']    = list of track objects
+# results['total_results'] = 5293248
+# results['next_href']     = pagination URL (see below)
+
+# Users
+results = sc_api("search/users", q="jazz", limit=10)
+
+# Playlists/sets
+results = sc_api("search/playlists", q="jazz", limit=10)
+
+# Paginate with next_href
+def paginate(first_response):
+    """Yield all pages of a collection response."""
+    yield from first_response.get('collection', [])
+    next_href = first_response.get('next_href')
+    while next_href:
+        page = json.loads(http_get(f"{next_href}&client_id={CLIENT_ID}"))
+        yield from page.get('collection', [])
+        next_href = page.get('next_href')
+```
+
+### Trending charts
+
+```python
+# Trending tracks across all genres
+trending = sc_api("charts", kind="trending",
+                  genre="soundcloud:genres:all-music", limit=20)
+for item in trending['collection']:
+    t = item['track']
+    print(f"{t['title']} — score={item['score']:.4f}")
+
+# Genre options: soundcloud:genres:all-music, soundcloud:genres:electronic,
+#                soundcloud:genres:hiphoprap, soundcloud:genres:ambient, etc.
+```
+
+### User resources
+
+```python
+user_id = 183  # numeric ID from resolve or hydration
+
+# User's tracks
+tracks = sc_api(f"users/{user_id}/tracks", limit=20)
+# tracks['collection'] = list of track objects
+
+# User's playlists
+playlists = sc_api(f"users/{user_id}/playlists", limit=10)
+
+# User's likes
+likes = sc_api(f"users/{user_id}/likes", limit=10)
+
+# Related tracks for a track
+related = sc_api("tracks/293/related", limit=10)
+# related['collection'] = list of track objects
+```
+
+### Waveform data
+
+```python
+# Waveform URL comes from track['waveform_url']
+waveform_url = "https://wave.sndcdn.com/cWHNerOLlkUq_m.json"
+waveform = json.loads(http_get(waveform_url))
+# {
+#   'width': 1800,   # number of sample points
+#   'height': 140,   # max amplitude value
+#   'samples': [11, 86, 91, 80, ...]  # 1800 amplitude values
+# }
+```
+
+---
+
+## Full track fields from `__sc_hydration` / API v2
+
+```
+id               int     Numeric track ID (e.g. 293)
+urn              str     "soundcloud:tracks:293"
+title            str     Track title
+description      str     May contain HTML entities/tags
+genre            str     Genre string
+tag_list         str     Space-separated tags
+created_at       str     ISO 8601 UTC
+last_modified    str     ISO 8601 UTC
+release_date     str     ISO 8601 UTC (original release)
+display_date     str     ISO 8601 UTC (shown to users)
+duration         int     Milliseconds
+full_duration    int     Milliseconds (untruncated)
+playback_count   int
+likes_count      int
+reposts_count    int
+comment_count    int
+download_count   int
+artwork_url      str     e.g. .../artworks-...-large.jpg (replace 'large' with 't500x500' for 500px)
+waveform_url     str     https://wave.sndcdn.com/....json
+permalink        str     Slug (e.g. "flickermood")
+permalink_url    str     Full canonical URL
+streamable       bool
+downloadable     bool
+license          str     e.g. "all-rights-reserved", "cc-by"
+sharing          str     "public" or "private"
+state            str     "finished" | "processing" | "failed"
+monetization_model str   "AD_SUPPORTED" | "SUB_HIGH_TIER" | "NOT_APPLICABLE"
+embeddable_by    str     "all" | "me" | "none"
+user             dict    Nested user object (id, username, avatar_url, verified, ...)
+user_id          int     Owner numeric ID
+publisher_metadata dict  {artist, publisher, isrc, contains_music, ...}
+media            dict    {'transcodings': [...]}  — stream URLs (require OAuth, not usable without login)
+label_name       str     Record label
+purchase_url     str     External buy link
+station_urn      str     "soundcloud:system-playlists:track-stations:{id}"
+```
+
+---
+
+## Gotchas
+
+**client_id is required for api-v2.soundcloud.com** — requests without it return HTTP 401. Always extract from `__sc_hydration['apiClient']['id']`.
+
+**client_id source: hydration, not JS bundles** — the JS bundles on `a-v2.sndcdn.com` do NOT contain the `client_id` pattern. The only reliable source is the `apiClient` object in the page hydration. It is stable across all pages (same value from homepage, track pages, user pages) and does not appear to rotate on short timescales.
+
+**Artwork URL sizes** — hydration/API returns `...-large.jpg` (100×100). Replace the size suffix to get larger images:
+- `-large.jpg` → 100×100
+- `-t300x300.jpg` → 300×300
+- `-t500x500.jpg` → 500×500 (oEmbed returns this size)
+
+**Regex must use `re.DOTALL`** — the `__sc_hydration` JSON spans multiple lines. Without `re.DOTALL`, the `.` in the regex won't match newlines.
+
+**Stream URLs (media.transcodings) are gated** — the HLS/progressive audio stream URLs in `track['media']['transcodings']` require an OAuth token even to fetch a stream manifest. They cannot be played without a logged-in session.
+
+**Bulk track lookup returns a list, not collection** — `GET /tracks?ids=...` returns a JSON array directly. Do NOT look for `.get('collection')`.
+
+**Search `total_results` can be huge** — results like 5M+ are normal for broad queries. Use `next_href` for pagination; do not calculate offsets manually.
+
+**oEmbed description contains HTML** — SoundCloud descriptions may include `&nbsp;` and anchor tags. Decode with `html.unescape()` if you need plain text.
+
+**HTTP 400 on some endpoints** — `/tracks/{id}/comments` returns 400 without OAuth headers. Timed comments are not accessible without login.
+
+**No browser required** — all documented approaches work with plain `http_get`. SoundCloud does not require JavaScript rendering for metadata extraction.
+
+**Rate limits** — 20 rapid sequential API v2 requests completed without errors in testing. SoundCloud does not publish official rate limits; stay under ~50 req/s for sustained scraping. oEmbed is more lenient than api-v2.
+
+---
+
+## Quick Reference
+
+| Goal | Approach | Auth |
+|------|----------|------|
+| Track title/author/thumbnail from URL | oEmbed | None |
+| Full track metadata + play counts | `__sc_hydration` `sound` key | None |
+| Full user profile + stats | `__sc_hydration` `user` key | None |
+| Full playlist with all tracks | `__sc_hydration` `playlist` key | None |
+| Search tracks/users/playlists | API v2 `/search/*` | client_id |
+| Trending charts | API v2 `/charts` | client_id |
+| Bulk track lookup by IDs | API v2 `/tracks?ids=` | client_id |
+| User's track list | API v2 `/users/{id}/tracks` | client_id |
+| Resolve permalink to resource | API v2 `/resolve?url=` | client_id |
+| Waveform amplitude data | Direct fetch of `waveform_url` | None |
+| Audio stream playback | OAuth login required | Login |