@pencil-agent/nano-pencil 2.0.0-beta.8 → 2.0.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/tradingview/scraping.md

@@ -1,309 +1,309 @@
-# TradingView — Scraping & Data Extraction
-
-`https://www.tradingview.com` — charting platform with multiple internal REST APIs. Stock/crypto/forex screener and symbol search work without auth. Use `http_get` or raw `urllib` for all workflows except JS-rendered chart pages.
-
-## Do this first
-
-**Use the scanner API for bulk screener data — one POST, no browser, full column control.**
-
-```python
-import json, urllib.request
-
-def tv_scan(payload, market="america"):
-    data = json.dumps(payload).encode()
-    req = urllib.request.Request(
-        f"https://scanner.tradingview.com/{market}/scan",
-        data=data,
-        headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
-    )
-    with urllib.request.urlopen(req, timeout=20) as r:
-        return json.loads(r.read())
-```
-
-**No auth, no Referer, no cookies required for the scanner.** Responses arrive in ~200ms.
-
-## Common workflows
-
-### Top stocks by market cap (screener)
-
-```python
-import json, urllib.request
-
-payload = {
-    "filter": [],
-    "options": {"lang": "en"},
-    "columns": ["name", "close", "change", "volume", "market_cap_basic"],
-    "sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
-    "range": [0, 10]   # [start, end] — half-open, so this returns rows 0–9
-}
-
-data = json.dumps(payload).encode()
-req = urllib.request.Request(
-    "https://scanner.tradingview.com/america/scan",
-    data=data,
-    headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
-)
-with urllib.request.urlopen(req, timeout=20) as r:
-    resp = json.loads(r.read())
-
-# resp["totalCount"] = 19549 (all US-listed instruments)
-# resp["data"] is a list of {"s": "NASDAQ:NVDA", "d": [col0, col1, ...]}
-# "d" values align positionally with "columns" in the payload
-
-cols = payload["columns"]
-for item in resp["data"]:
-    row = dict(zip(cols, item["d"]))
-    symbol = item["s"]   # e.g. "NASDAQ:AAPL"
-    print(symbol, row["close"], row["change"], row["market_cap_basic"])
-# NASDAQ:NVDA  201.68  1.68  4900823822021.0
-# NASDAQ:AAPL  270.23  2.59  3967284528489.0
-# ...
-```
-
-**Critical**: `"d"` is a plain positional array — index 0 = columns[0], index 1 = columns[1], etc. There are no keys in the row data itself.
-
-### Pagination
-
-```python
-# Page 1: range [0, 20]
-# Page 2: range [20, 40]
-payload["range"] = [20, 40]
-```
-
-### Filtering stocks
-
-```python
-payload = {
-    "filter": [
-        {"left": "market_cap_basic", "operation": "greater", "right": 10_000_000_000},
-        {"left": "volume",           "operation": "greater", "right": 5_000_000},
-        {"left": "change",           "operation": "in_range", "right": [2, 10]},
-        {"left": "exchange",         "operation": "equal",   "right": "NASDAQ"},
-        {"left": "sector",           "operation": "equal",   "right": "Electronic Technology"},
-    ],
-    "columns": ["name", "close", "change", "volume", "market_cap_basic",
-                "description", "sector", "industry"],
-    "sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
-    "range": [0, 20]
-}
-```
-
-Valid filter operations: `greater`, `less`, `equal`, `in_range` (right = [min, max]), `match` (substring on `name`).
-
-Sector names use TradingView taxonomy (not GICS). Confirmed working values:
-- `"Electronic Technology"` — NVDA, AAPL, TSM
-- `"Technology Services"` — MSFT, GOOGL, META
-- `"Finance"`, `"Health Technology"`, `"Consumer Non-Durables"`
-
-### Full list of tested valid column names
-
-```python
-# Price & volume
-"name"                     # ticker (e.g. "AAPL")
-"description"              # full name ("Apple Inc.")
-"close"                    # last price
-"open", "high", "low"
-"volume"
-"change"                   # % change today
-"change_abs"               # absolute price change
-"change|1M"                # 1-month % change (also: |6M, |1Y)
-"High.1M", "High.6M"       # period high
-"High.All", "Low.All"      # all-time high/low
-"price_52_week_high"       # confirmed works
-"price_52_week_low"        # confirmed works
-"premarket_change"         # pre-market %
-"postmarket_change"        # after-hours %
-"gap"                      # overnight gap %
-"change_from_open_abs"     # intraday move from open
-"average_volume_10d_calc"  # 10-day avg volume
-"relative_volume_10d_calc" # relative volume vs 10-day avg
-"relative_volume_intraday|5"  # intraday relative vol (5m bars)
-
-# Fundamentals
-"market_cap_basic"          # market cap in USD
-"earnings_per_share_diluted_ttm"  # EPS TTM
-"price_earnings_ttm"        # P/E TTM
-"P/E"                       # P/E (snapshot)
-"dividends_yield"           # dividend yield %
-"beta_1_year"               # beta
-"float_shares_outstanding"  # float shares
-
-# Technical ratings & indicators
-"Recommend.All"   # composite rating: -1 (strong sell) to +1 (strong buy)
-"RSI"             # RSI 14
-"MACD.macd"       # MACD line
-
-# Classification
-"sector", "industry", "country", "exchange"
-"type"    # "stock", "fund", "dr" (depository receipt), etc.
-
-# NOTE: "52_week_high" / "52_week_low" are INVALID — use "price_52_week_high" / "price_52_week_low"
-# NOTE: "EPS_diluted_net" is INVALID — use "earnings_per_share_diluted_ttm"
-```
-
-Bad columns return HTTP 400 with `{"error": "Unknown field \"X\""}`.
-
-### Other scanner markets
-
-```python
-# market argument options (confirmed working):
-# "america"  — US equities (19,549 instruments)
-# "crypto"   — crypto across exchanges (56,455 instruments)
-# "forex"    — FX pairs (6,401 instruments)
-# "futures"  — futures (53,947 instruments)
-
-# Crypto example
-payload = {
-    "filter": [],
-    "columns": ["name", "close", "change", "volume", "market_cap_calc"],
-    "sort": {"sortBy": "market_cap_calc", "sortOrder": "desc"},
-    "range": [0, 10]
-}
-resp = tv_scan(payload, market="crypto")
-# Returns BTC, ETH, etc. across Binance, Bybit, OKX...
-```
-
-### Symbol search (requires Origin header)
-
-```python
-import json, urllib.request
-
-def symbol_search(query, exchange="", type_filter="", limit=50):
-    url = (
-        f"https://symbol-search.tradingview.com/symbol_search/v3/"
-        f"?text={query}&hl=1&exchange={exchange}&lang=en"
-        f"&search_type={type_filter or 'undefined'}&domain=production"
-    )
-    req = urllib.request.Request(url, headers={
-        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
-        "Origin": "https://www.tradingview.com",   # REQUIRED — 403 without this
-    })
-    with urllib.request.urlopen(req, timeout=15) as r:
-        return json.loads(r.read())
-
-result = symbol_search("AAPL")
-# result["symbols_remaining"] = 137
-# result["symbols"] = list of up to 50 matches
-# result["symbols"][0] keys:
-#   symbol, description, type, exchange, country, currency_code,
-#   cusip, isin, cik_code, logoid, provider_id, source_id,
-#   is_primary_listing, typespecs
-```
-
-**Gotcha**: `symbol-search.tradingview.com` requires `Origin: https://www.tradingview.com`. Referer alone is not enough. The scanner API does NOT need Origin or Referer.
-
-Filter by exchange and type:
-
-```python
-# Exact match on NASDAQ:AAPL
-result = symbol_search("AAPL", exchange="NASDAQ", type_filter="stock")
-# Returns 1 result — exact symbol only when exchange is specified
-```
-
-### News headlines for a symbol
-
-```python
-import json, urllib.request
-
-def get_news(symbol, limit=20):
-    # symbol format: "NASDAQ:AAPL", "NYSE:TSLA"
-    url = (
-        f"https://news-headlines.tradingview.com/v2/view/headlines/symbol"
-        f"?symbol={symbol}&client=web&streaming=false&lang=en&limit={limit}"
-    )
-    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
-    with urllib.request.urlopen(req, timeout=15) as r:
-        data = json.loads(r.read())
-    return data["items"]  # list of news items
-
-items = get_news("NASDAQ:AAPL", limit=10)
-# item keys: id, title, provider, sourceLogoId, published (unix ts),
-#            source, urgency, link, permission, relatedSymbols, storyPath
-# example:
-# items[0]["title"]     = "Apple Clears Major Legal Hurdle..."
-# items[0]["published"] = 1776472317  (unix timestamp)
-# items[0]["link"]      = "https://stocktwits.com/..."
-# items[0]["relatedSymbols"] = [{"symbol": "NASDAQ:AAPL", "logoid": "apple"}]
-```
-
-No auth or special headers needed. Returns up to 200 items per request.
-
-### Published trading ideas feed
-
-```python
-import json, urllib.request
-
-def get_ideas(sort="trending", page=1, symbol=None):
-    # Valid sort values (others return 400):
-    # "trending", "recent", "latest_popular", "week_popular",
-    # "suggested", "recent_extended", "picked_time"
-    url = f"https://www.tradingview.com/api/v1/ideas/?lang=en&sort={sort}&page={page}"
-    if symbol:
-        url += f"&symbol={symbol}"  # e.g. "NASDAQ:AAPL"
-    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
-    with urllib.request.urlopen(req, timeout=15) as r:
-        return json.loads(r.read())
-
-data = get_ideas("trending")
-# data["count"]      = 1000 (always 1000 — soft cap)
-# data["page_size"]  = 20
-# data["page_count"] = 50
-# data["next"]       = "https://www.tradingview.com/api/v1/ideas/?page=2"
-# data["results"]    = list of idea objects
-
-idea = data["results"][0]
-# idea keys: id, name, description, created_at, chart_url, views_count,
-#            likes_count, comments_count, is_video, is_education, is_hot,
-#            symbol (dict with name/exchange/type/interval/direction),
-#            user (dict with username/is_pro/badges), image (big/middle URLs)
-# idea["symbol"]["direction"]: 1=long, 2=short, 0=neutral
-
-# Filter by symbol:
-aapl_ideas = get_ideas(symbol="NASDAQ:AAPL")
-```
-
-## API summary table
-
-| Endpoint | Auth | Headers needed | Speed |
-|---|---|---|---|
-| `scanner.tradingview.com/{market}/scan` | None | None | ~200ms |
-| `symbol-search.tradingview.com/symbol_search/v3/` | None | `Origin: https://www.tradingview.com` | ~150ms |
-| `symbol-search.tradingview.com/symbol_search/` (v1) | None | `Origin: https://www.tradingview.com` | ~100ms |
-| `news-headlines.tradingview.com/v2/view/headlines/symbol` | None | None | ~400ms |
-| `www.tradingview.com/api/v1/ideas/` | None | None | ~300ms |
-| `data.tradingview.com/quotes/` | None | None | **Dead** — connection refused |
-| `economic-calendar.tradingview.com/events` | Yes | — | HTTP 403 |
-
-## Gotchas
-
-**Scanner `range` is half-open**: `[0, 10]` returns rows 0–9 (10 rows total). `[10, 20]` for the next page.
-
-**Column order is critical**: The `"d"` array in each result row is positional — it exactly mirrors your `"columns"` array. Always zip them: `dict(zip(columns, item["d"]))`.
-
-**`data.tradingview.com/quotes/` is dead**: The URL `https://data.tradingview.com/quotes/?symbols=NASDAQ:AAPL` closes the connection without a response. Use the scanner API instead for real-time quotes.
-
-**Scanner needs no Referer**: `scanner.tradingview.com` works with just `User-Agent`. The symbol-search subdomain checks `Origin` (CORS enforcement on the server side).
-
-**Symbol search highlights**: The v3 endpoint wraps matched text in `<em>` tags (e.g. `"<em>AAPL</em>"`). Strip them: `re.sub(r'</?em>', '', symbol["symbol"])`.
-
-**Ideas sort validation**: Only specific values work. `"sort=popular"` returns 400. Use `"trending"`, `"recent"`, `"latest_popular"`, `"week_popular"`, `"suggested"`.
-
-**Ideas count cap**: The API always reports `count=1000` regardless of actual corpus size. With `page_size=20`, max pages is 50.
-
-**Scanner server is AWS CloudFront** (`X-Amz-Cf-Pop` header) with a custom `Server: tv` — no Cloudflare. No anti-bot on the scanner subdomain. Main `www.tradingview.com` is a React SPA with `window.initData = {}` (empty — no embedded data). All data is loaded via API calls after hydration.
-
-**Rate limits**: No 429s observed in testing. 5 concurrent scanner calls complete in ~1s. Symbol search returns `symbols_remaining` in the response (counts against some quota — varies 90–180 across calls but never blocks). Observed no blocking after 15 rapid calls in a row.
-
-**Sector names**: Use TradingView's own taxonomy, not GICS. "Technology" does not exist — use `"Electronic Technology"` (hardware/semis) or `"Technology Services"` (software/internet).
-
-## When to use the browser
-
-The charting UI (`/chart/`), symbol detail pages (`/symbols/NASDAQ-AAPL/`), and the ideas page (`/ideas/`) are React SPAs — their visible data comes from the APIs above, not embedded HTML. Use browser + JS extraction only if you need visual chart screenshots or data from auth-gated pages (watchlists, portfolio, paper trading).
-
-```python
-# Only if you need a chart screenshot:
-goto_url("https://www.tradingview.com/chart/?symbol=NASDAQ:AAPL")
-wait_for_load()
-wait(3)   # chart renders asynchronously after readyState
-capture_screenshot("/tmp/aapl_chart.png", full=False)
-```
+# TradingView — Scraping & Data Extraction
+
+`https://www.tradingview.com` — charting platform with multiple internal REST APIs. Stock/crypto/forex screener and symbol search work without auth. Use `http_get` or raw `urllib` for all workflows except JS-rendered chart pages.
+
+## Do this first
+
+**Use the scanner API for bulk screener data — one POST, no browser, full column control.**
+
+```python
+import json, urllib.request
+
+def tv_scan(payload, market="america"):
+    data = json.dumps(payload).encode()
+    req = urllib.request.Request(
+        f"https://scanner.tradingview.com/{market}/scan",
+        data=data,
+        headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
+    )
+    with urllib.request.urlopen(req, timeout=20) as r:
+        return json.loads(r.read())
+```
+
+**No auth, no Referer, no cookies required for the scanner.** Responses arrive in ~200ms.
+
+## Common workflows
+
+### Top stocks by market cap (screener)
+
+```python
+import json, urllib.request
+
+payload = {
+    "filter": [],
+    "options": {"lang": "en"},
+    "columns": ["name", "close", "change", "volume", "market_cap_basic"],
+    "sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
+    "range": [0, 10]   # [start, end] — half-open, so this returns rows 0–9
+}
+
+data = json.dumps(payload).encode()
+req = urllib.request.Request(
+    "https://scanner.tradingview.com/america/scan",
+    data=data,
+    headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
+)
+with urllib.request.urlopen(req, timeout=20) as r:
+    resp = json.loads(r.read())
+
+# resp["totalCount"] = 19549 (all US-listed instruments)
+# resp["data"] is a list of {"s": "NASDAQ:NVDA", "d": [col0, col1, ...]}
+# "d" values align positionally with "columns" in the payload
+
+cols = payload["columns"]
+for item in resp["data"]:
+    row = dict(zip(cols, item["d"]))
+    symbol = item["s"]   # e.g. "NASDAQ:AAPL"
+    print(symbol, row["close"], row["change"], row["market_cap_basic"])
+# NASDAQ:NVDA  201.68  1.68  4900823822021.0
+# NASDAQ:AAPL  270.23  2.59  3967284528489.0
+# ...
+```
+
+**Critical**: `"d"` is a plain positional array — index 0 = columns[0], index 1 = columns[1], etc. There are no keys in the row data itself.
+
+### Pagination
+
+```python
+# Page 1: range [0, 20]
+# Page 2: range [20, 40]
+payload["range"] = [20, 40]
+```
+
+### Filtering stocks
+
+```python
+payload = {
+    "filter": [
+        {"left": "market_cap_basic", "operation": "greater", "right": 10_000_000_000},
+        {"left": "volume",           "operation": "greater", "right": 5_000_000},
+        {"left": "change",           "operation": "in_range", "right": [2, 10]},
+        {"left": "exchange",         "operation": "equal",   "right": "NASDAQ"},
+        {"left": "sector",           "operation": "equal",   "right": "Electronic Technology"},
+    ],
+    "columns": ["name", "close", "change", "volume", "market_cap_basic",
+                "description", "sector", "industry"],
+    "sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
+    "range": [0, 20]
+}
+```
+
+Valid filter operations: `greater`, `less`, `equal`, `in_range` (right = [min, max]), `match` (substring on `name`).
+
+Sector names use TradingView taxonomy (not GICS). Confirmed working values:
+- `"Electronic Technology"` — NVDA, AAPL, TSM
+- `"Technology Services"` — MSFT, GOOGL, META
+- `"Finance"`, `"Health Technology"`, `"Consumer Non-Durables"`
+
+### Full list of tested valid column names
+
+```python
+# Price & volume
+"name"                     # ticker (e.g. "AAPL")
+"description"              # full name ("Apple Inc.")
+"close"                    # last price
+"open", "high", "low"
+"volume"
+"change"                   # % change today
+"change_abs"               # absolute price change
+"change|1M"                # 1-month % change (also: |6M, |1Y)
+"High.1M", "High.6M"       # period high
+"High.All", "Low.All"      # all-time high/low
+"price_52_week_high"       # confirmed works
+"price_52_week_low"        # confirmed works
+"premarket_change"         # pre-market %
+"postmarket_change"        # after-hours %
+"gap"                      # overnight gap %
+"change_from_open_abs"     # intraday move from open
+"average_volume_10d_calc"  # 10-day avg volume
+"relative_volume_10d_calc" # relative volume vs 10-day avg
+"relative_volume_intraday|5"  # intraday relative vol (5m bars)
+
+# Fundamentals
+"market_cap_basic"          # market cap in USD
+"earnings_per_share_diluted_ttm"  # EPS TTM
+"price_earnings_ttm"        # P/E TTM
+"P/E"                       # P/E (snapshot)
+"dividends_yield"           # dividend yield %
+"beta_1_year"               # beta
+"float_shares_outstanding"  # float shares
+
+# Technical ratings & indicators
+"Recommend.All"   # composite rating: -1 (strong sell) to +1 (strong buy)
+"RSI"             # RSI 14
+"MACD.macd"       # MACD line
+
+# Classification
+"sector", "industry", "country", "exchange"
+"type"    # "stock", "fund", "dr" (depository receipt), etc.
+
+# NOTE: "52_week_high" / "52_week_low" are INVALID — use "price_52_week_high" / "price_52_week_low"
+# NOTE: "EPS_diluted_net" is INVALID — use "earnings_per_share_diluted_ttm"
+```
+
+Bad columns return HTTP 400 with `{"error": "Unknown field \"X\""}`.
+
+### Other scanner markets
+
+```python
+# market argument options (confirmed working):
+# "america"  — US equities (19,549 instruments)
+# "crypto"   — crypto across exchanges (56,455 instruments)
+# "forex"    — FX pairs (6,401 instruments)
+# "futures"  — futures (53,947 instruments)
+
+# Crypto example
+payload = {
+    "filter": [],
+    "columns": ["name", "close", "change", "volume", "market_cap_calc"],
+    "sort": {"sortBy": "market_cap_calc", "sortOrder": "desc"},
+    "range": [0, 10]
+}
+resp = tv_scan(payload, market="crypto")
+# Returns BTC, ETH, etc. across Binance, Bybit, OKX...
+```
+
+### Symbol search (requires Origin header)
+
+```python
+import json, urllib.request
+
+def symbol_search(query, exchange="", type_filter="", limit=50):
+    url = (
+        f"https://symbol-search.tradingview.com/symbol_search/v3/"
+        f"?text={query}&hl=1&exchange={exchange}&lang=en"
+        f"&search_type={type_filter or 'undefined'}&domain=production"
+    )
+    req = urllib.request.Request(url, headers={
+        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
+        "Origin": "https://www.tradingview.com",   # REQUIRED — 403 without this
+    })
+    with urllib.request.urlopen(req, timeout=15) as r:
+        return json.loads(r.read())
+
+result = symbol_search("AAPL")
+# result["symbols_remaining"] = 137
+# result["symbols"] = list of up to 50 matches
+# result["symbols"][0] keys:
+#   symbol, description, type, exchange, country, currency_code,
+#   cusip, isin, cik_code, logoid, provider_id, source_id,
+#   is_primary_listing, typespecs
+```
+
+**Gotcha**: `symbol-search.tradingview.com` requires `Origin: https://www.tradingview.com`. Referer alone is not enough. The scanner API does NOT need Origin or Referer.
+
+Filter by exchange and type:
+
+```python
+# Exact match on NASDAQ:AAPL
+result = symbol_search("AAPL", exchange="NASDAQ", type_filter="stock")
+# Returns 1 result — exact symbol only when exchange is specified
+```
+
+### News headlines for a symbol
+
+```python
+import json, urllib.request
+
+def get_news(symbol, limit=20):
+    # symbol format: "NASDAQ:AAPL", "NYSE:TSLA"
+    url = (
+        f"https://news-headlines.tradingview.com/v2/view/headlines/symbol"
+        f"?symbol={symbol}&client=web&streaming=false&lang=en&limit={limit}"
+    )
+    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=15) as r:
+        data = json.loads(r.read())
+    return data["items"]  # list of news items
+
+items = get_news("NASDAQ:AAPL", limit=10)
+# item keys: id, title, provider, sourceLogoId, published (unix ts),
+#            source, urgency, link, permission, relatedSymbols, storyPath
+# example:
+# items[0]["title"]     = "Apple Clears Major Legal Hurdle..."
+# items[0]["published"] = 1776472317  (unix timestamp)
+# items[0]["link"]      = "https://stocktwits.com/..."
+# items[0]["relatedSymbols"] = [{"symbol": "NASDAQ:AAPL", "logoid": "apple"}]
+```
+
+No auth or special headers needed. Returns up to 200 items per request.
+
+### Published trading ideas feed
+
+```python
+import json, urllib.request
+
+def get_ideas(sort="trending", page=1, symbol=None):
+    # Valid sort values (others return 400):
+    # "trending", "recent", "latest_popular", "week_popular",
+    # "suggested", "recent_extended", "picked_time"
+    url = f"https://www.tradingview.com/api/v1/ideas/?lang=en&sort={sort}&page={page}"
+    if symbol:
+        url += f"&symbol={symbol}"  # e.g. "NASDAQ:AAPL"
+    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=15) as r:
+        return json.loads(r.read())
+
+data = get_ideas("trending")
+# data["count"]      = 1000 (always 1000 — soft cap)
+# data["page_size"]  = 20
+# data["page_count"] = 50
+# data["next"]       = "https://www.tradingview.com/api/v1/ideas/?page=2"
+# data["results"]    = list of idea objects
+
+idea = data["results"][0]
+# idea keys: id, name, description, created_at, chart_url, views_count,
+#            likes_count, comments_count, is_video, is_education, is_hot,
+#            symbol (dict with name/exchange/type/interval/direction),
+#            user (dict with username/is_pro/badges), image (big/middle URLs)
+# idea["symbol"]["direction"]: 1=long, 2=short, 0=neutral
+
+# Filter by symbol:
+aapl_ideas = get_ideas(symbol="NASDAQ:AAPL")
+```
+
+## API summary table
+
+| Endpoint | Auth | Headers needed | Speed |
+|---|---|---|---|
+| `scanner.tradingview.com/{market}/scan` | None | None | ~200ms |
+| `symbol-search.tradingview.com/symbol_search/v3/` | None | `Origin: https://www.tradingview.com` | ~150ms |
+| `symbol-search.tradingview.com/symbol_search/` (v1) | None | `Origin: https://www.tradingview.com` | ~100ms |
+| `news-headlines.tradingview.com/v2/view/headlines/symbol` | None | None | ~400ms |
+| `www.tradingview.com/api/v1/ideas/` | None | None | ~300ms |
+| `data.tradingview.com/quotes/` | None | None | **Dead** — connection refused |
+| `economic-calendar.tradingview.com/events` | Yes | — | HTTP 403 |
+
+## Gotchas
+
+**Scanner `range` is half-open**: `[0, 10]` returns rows 0–9 (10 rows total). `[10, 20]` for the next page.
+
+**Column order is critical**: The `"d"` array in each result row is positional — it exactly mirrors your `"columns"` array. Always zip them: `dict(zip(columns, item["d"]))`.
+
+**`data.tradingview.com/quotes/` is dead**: The URL `https://data.tradingview.com/quotes/?symbols=NASDAQ:AAPL` closes the connection without a response. Use the scanner API instead for real-time quotes.
+
+**Scanner needs no Referer**: `scanner.tradingview.com` works with just `User-Agent`. The symbol-search subdomain checks `Origin` (CORS enforcement on the server side).
+
+**Symbol search highlights**: The v3 endpoint wraps matched text in `<em>` tags (e.g. `"<em>AAPL</em>"`). Strip them: `re.sub(r'</?em>', '', symbol["symbol"])`.
+
+**Ideas sort validation**: Only specific values work. `"sort=popular"` returns 400. Use `"trending"`, `"recent"`, `"latest_popular"`, `"week_popular"`, `"suggested"`.
+
+**Ideas count cap**: The API always reports `count=1000` regardless of actual corpus size. With `page_size=20`, max pages is 50.
+
+**Scanner server is AWS CloudFront** (`X-Amz-Cf-Pop` header) with a custom `Server: tv` — no Cloudflare. No anti-bot on the scanner subdomain. Main `www.tradingview.com` is a React SPA with `window.initData = {}` (empty — no embedded data). All data is loaded via API calls after hydration.
+
+**Rate limits**: No 429s observed in testing. 5 concurrent scanner calls complete in ~1s. Symbol search returns `symbols_remaining` in the response (counts against some quota — varies 90–180 across calls but never blocks). Observed no blocking after 15 rapid calls in a row.
+
+**Sector names**: Use TradingView's own taxonomy, not GICS. "Technology" does not exist — use `"Electronic Technology"` (hardware/semis) or `"Technology Services"` (software/internet).
+
+## When to use the browser
+
+The charting UI (`/chart/`), symbol detail pages (`/symbols/NASDAQ-AAPL/`), and the ideas page (`/ideas/`) are React SPAs — their visible data comes from the APIs above, not embedded HTML. Use browser + JS extraction only if you need visual chart screenshots or data from auth-gated pages (watchlists, portfolio, paper trading).
+
+```python
+# Only if you need a chart screenshot:
+goto_url("https://www.tradingview.com/chart/?symbol=NASDAQ:AAPL")
+wait_for_load()
+wait(3)   # chart renders asynchronously after readyState
+capture_screenshot("/tmp/aapl_chart.png", full=False)
+```