@pencil-agent/nano-pencil 2.0.0-beta.8 → 2.0.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/macrotrends/scraping.md

@@ -1,537 +1,537 @@
-# Macrotrends — Data Extraction
-
-`https://www.macrotrends.net` — long-term historical financial and economic charts. Three access patterns depending on page type; all work with plain `http_get`, no browser required.
-
-All results validated against live site on 2026-04-18.
-
-## Do this first: pick your access pattern
-
-| Goal | Pattern | Latency | Variable |
-|------|---------|---------|----------|
-| Stock OHLCV price history | Direct iframe PHP | ~190ms | `dataDaily` |
-| Stock market cap (daily) | Direct iframe PHP | ~200ms | `chartData` |
-| Stock fundamentals (PE, revenue, margins) | Direct iframe PHP | ~140ms | `chartData` |
-| S&P 500 / composite index charts | `chart_iframe_comp.php` | ~90ms | `originalData` |
-| Economic indicators (rates, yields, CPI) | `/economic-data/` JSON API | ~150ms | `data[]` array |
-| Gold, commodity prices | Either path (both work) | ~150ms | `data[]` or `originalData` |
-
-**Never use the browser for Macrotrends read-only tasks.** All endpoints are accessible via `http_get` with the default `Mozilla/5.0` UA. For pages that occasionally 403, switch to a Chrome UA (see gotchas).
-
----
-
-## Pattern 1: Stock price history (OHLCV)
-
-Construct the iframe URL directly — no need to fetch the main page first.
-
-```python
-import json, re
-from helpers import http_get
-
-def get_stock_ohlcv(ticker: str, years_back: int = None) -> list[dict]:
-    """
-    Returns daily OHLCV records for any US stock.
-
-    ticker:     uppercase ticker symbol, e.g. 'AAPL', 'MSFT', 'TSLA', 'NVDA'
-    years_back: number of years of history (1=~250 records, 15=~3772 records).
-                Omit (None) to get ALL available history (AAPL goes back to 1980).
-    """
-    url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t={ticker}"
-    if years_back:
-        url += f"&yb={years_back}"
-
-    html = http_get(url)
-    m = re.search(r'var\s+dataDaily\s*=\s*\[', html)
-    if not m:
-        raise ValueError(f"No dataDaily found for ticker {ticker!r}")
-
-    si = html.index('[', m.start())
-    bc = 0
-    for j, ch in enumerate(html[si:], si):
-        if ch == '[':   bc += 1
-        elif ch == ']':
-            bc -= 1
-            if bc == 0: ei = j; break
-    return json.loads(html[si:ei+1])
-
-# Usage
-records = get_stock_ohlcv('AAPL', years_back=15)
-# [{'d': '2011-04-18', 'o': '9.771', 'h': '9.9547', 'l': '9.593', 'c': '9.9433', 'v': '18.275'}, ...]
-
-latest = records[-1]
-# {'d': '2026-04-17', 'o': '266.96', 'h': '272.3', 'l': '266.72', 'c': '270.23',
-#  'v': '55.211', 'ma50': '260.554', 'ma200': '251.828'}
-
-print(f"{latest['d']}: close=${latest['c']} vol={latest['v']}M shares")
-```
-
-### dataDaily field reference
-
-| Field | Meaning | Type |
-|-------|---------|------|
-| `d` | Date (YYYY-MM-DD) | str |
-| `o` | Open price (adjusted for splits) | str/float |
-| `h` | High | str/float |
-| `l` | Low | str/float |
-| `c` | Close | str/float |
-| `v` | Volume in **millions of shares** | str/float |
-| `ma50` | 50-day moving average | str/float (appears on recent records only) |
-| `ma200` | 200-day moving average | str/float (appears on recent records only) |
-
-**Note:** All price values are strings — cast with `float()`. Volume is millions: `55.211` = 55.2M shares traded.
-
-### Confirmed tickers (2026-04-18)
-
-All tested with direct iframe URL, no page fetch needed:
-
-```python
-# All work: AAPL, MSFT, TSLA, NVDA, GOOGL, AMZN, META, NFLX, etc.
-# 3772 records for yb=15 (goes back to 2011-04-18)
-# AAPL full history: 11428 records back to 1980-12-12
-```
-
----
-
-## Pattern 2: Stock fundamentals (PE ratio, revenue, market cap, margins)
-
-Different PHP files depending on metric. Construct directly.
-
-### Market cap (daily, in billions USD)
-
-```python
-import json, re
-from helpers import http_get
-
-def get_market_cap(ticker: str, years_back: int = 15) -> list[dict]:
-    url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/market_cap.php?t={ticker}&yb={years_back}"
-    html = http_get(url)
-    m = re.search(r'var\s+chartData\s*=\s*\[', html)
-    si = html.index('[', m.start())
-    bc = 0
-    for j, ch in enumerate(html[si:], si):
-        if ch == '[':   bc += 1
-        elif ch == ']':
-            bc -= 1
-            if bc == 0: ei = j; break
-    return json.loads(html[si:ei+1])
-
-data = get_market_cap('AAPL')
-# [{'date': '2026-04-15', 'v1': 3929.35}, {'date': '2026-04-16', 'v1': 3884.67}, ...]
-# v1 = market cap in billions USD
-```
-
-### PE ratio, revenue, current ratio (quarterly/annual fundamentals)
-
-```python
-import json, re
-from helpers import http_get
-
-def get_fundamental(ticker: str, metric_type: str, statement: str,
-                    freq: str = 'Q', years_back: int = 15) -> list[dict]:
-    """
-    freq: 'Q' = quarterly, 'A' = annual
-    """
-    url = (
-        f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
-        f"fundamental_iframe.php?t={ticker}&type={metric_type}&statement={statement}"
-        f"&freq={freq}&sub=&yb={years_back}"
-    )
-    html = http_get(url)
-    m = re.search(r'var\s+chartData\s*=\s*\[', html)
-    si = html.index('[', m.start())
-    bc = 0
-    for j, ch in enumerate(html[si:], si):
-        if ch == '[':   bc += 1
-        elif ch == ']':
-            bc -= 1
-            if bc == 0: ei = j; break
-    return json.loads(html[si:ei+1])
-
-# PE ratio
-pe = get_fundamental('AAPL', 'pe-ratio', 'price-ratios')
-# [{'date': '2025-09-30', 'v1': 254.146, 'v2': 7.46, 'v3': 34.07}, ...]
-# v1 = stock price, v2 = quarterly EPS, v3 = PE ratio
-
-# Revenue
-rev = get_fundamental('AAPL', 'revenue', 'income-statement')
-# [{'date': '2025-12-31', 'v1': 435.617, 'v2': 143.756, 'v3': 15.65}, ...]
-# v1 = TTM revenue ($B), v2 = quarterly revenue ($B), v3 = YoY growth %
-
-# Total assets
-assets = get_fundamental('AAPL', 'total-assets', 'balance-sheet')
-
-# Current ratio
-ratio = get_fundamental('AAPL', 'current-ratio', 'ratios')
-```
-
-### Profit margins
-
-```python
-def get_profit_margins(ticker: str, years_back: int = 15) -> list[dict]:
-    url = (
-        f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
-        f"fundamental_metric.php?t={ticker}&chart=profit-margin&sub=&yb={years_back}"
-    )
-    html = http_get(url)
-    m = re.search(r'var\s+chartData\s*=\s*\[', html)
-    si = html.index('[', m.start())
-    bc = 0
-    for j, ch in enumerate(html[si:], si):
-        if ch == '[':   bc += 1
-        elif ch == ']':
-            bc -= 1
-            if bc == 0: ei = j; break
-    return json.loads(html[si:ei+1])
-
-margins = get_profit_margins('AAPL')
-# [{'date': '2025-12-31', 'v1': 47.33, 'v2': 32.38, 'v3': 27.04}, ...]
-# v1 = gross margin %, v2 = operating margin %, v3 = net margin %
-```
-
-### Dividend yield
-
-```python
-def get_dividend_yield(ticker: str, years_back: int = 15) -> list[dict]:
-    url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/dividend_yield.php?t={ticker}&yb={years_back}"
-    html = http_get(url)
-    m = re.search(r'var\s+chartData\s*=\s*\[', html)
-    si = html.index('[', m.start())
-    bc = 0
-    for j, ch in enumerate(html[si:], si):
-        if ch == '[':   bc += 1
-        elif ch == ']':
-            bc -= 1
-            if bc == 0: ei = j; break
-    return json.loads(html[si:ei+1])
-
-dy = get_dividend_yield('AAPL')
-# [{'date': '2026-04-17', 'c': 270.23, 'ttm_d': 1.03848, 'ttm_dy': 0.3843}, ...]
-# c = stock price, ttm_d = TTM dividend ($), ttm_dy = TTM yield (%)
-```
-
-### Stock metric URL reference
-
-| Metric | PHP file | Extra params |
-|--------|----------|-------------|
-| Stock price OHLCV | `stock_price_history.php` | — |
-| Market cap (daily) | `market_cap.php` | — |
-| Dividend yield | `dividend_yield.php` | — |
-| Stock splits (price history) | `stock_splits.php` | — |
-| PE ratio | `fundamental_iframe.php` | `type=pe-ratio&statement=price-ratios` |
-| Revenue | `fundamental_iframe.php` | `type=revenue&statement=income-statement` |
-| Total assets | `fundamental_iframe.php` | `type=total-assets&statement=balance-sheet` |
-| Current ratio | `fundamental_iframe.php` | `type=current-ratio&statement=ratios` |
-| Profit margins | `fundamental_metric.php` | `chart=profit-margin` |
-
-Base URL prefix: `https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/`
-
-All take `?t={TICKER}&yb={N}` (or `&sub=&yb={N}` for the fundamental ones).
-
----
-
-## Pattern 3: Index and composite charts (S&P 500, Shiller PE, etc.)
-
-These pages embed chart data via `chart_iframe_comp.php`. The variable is `originalData`.
-
-```python
-import json, re
-from helpers import http_get
-
-def extract_index_chart(page_id: int, url_slug: str) -> list[dict]:
-    """
-    page_id:  the numeric ID from the page URL, e.g. 2577
-    url_slug: last segment of the page URL, e.g. 'sp500-pe-ratio-price-to-earnings-chart'
-    """
-    url = f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={page_id}&url={url_slug}"
-    html = http_get(url)
-    m = re.search(r'var\s+originalData\s*=\s*\[', html)
-    if not m:
-        raise ValueError("originalData not found — this page may use a different pattern")
-    si = html.index('[', m.start())
-    bc = 0
-    for j, ch in enumerate(html[si:], si):
-        if ch == '[':   bc += 1
-        elif ch == ']':
-            bc -= 1
-            if bc == 0: ei = j; break
-    return json.loads(html[si:ei+1])
-
-# S&P 500 PE ratio (1180 monthly records, 1927-2026)
-pe_data = extract_index_chart(2577, 'sp500-pe-ratio-price-to-earnings-chart')
-# [{'date': '1927-12-01', 'close': '15.9099'}, ..., {'date': '2026-03-01', 'close': '27.8925'}]
-# 'close' is the PE ratio value
-
-# Gold prices (1336 monthly records, 1915-2026)
-gold_data = extract_index_chart(1333, 'historical-gold-prices-100-year-chart')
-# [{'id': 'GOLDAMGBD228NLBM', 'date': '1915-01-01', 'close': '629.36', 'close1': '19.250'}, ...]
-# 'close' = inflation-adjusted price, 'close1' = nominal USD price
-
-print(f"Latest S&P PE: {pe_data[-1]}")   # {'date': '2026-03-01', 'close': '27.8925'}
-print(f"Latest gold:   {gold_data[-1]}") # {'id': ..., 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
-```
-
-### Detecting which pattern a page uses
-
-```python
-def get_page_pattern(page_url: str) -> str:
-    html = http_get(page_url)
-    if 'chart_iframe_comp.php' in html:
-        return 'index_chart'           # use extract_index_chart()
-    elif 'generateChart' in html and 'highchartsURL' in html:
-        return 'economic_api'          # use get_economic_data()
-    elif '/production/stocks/desktop/PRODUCTION/' in html:
-        return 'stock_iframe'          # use get_stock_ohlcv() etc.
-    return 'unknown'
-```
-
-### To get the ID and slug from a page
-
-```python
-import re
-from helpers import http_get
-
-page_url = "https://www.macrotrends.net/2577/sp500-pe-ratio-price-to-earnings-chart"
-html = http_get(page_url)
-
-# Option A: parse from the iframe src in the HTML
-m = re.search(r'chart_iframe_comp\.php\?id=(\d+)&url=([^"&]+)', html)
-if m:
-    page_id, url_slug = int(m.group(1)), m.group(2)
-
-# Option B: derive from the page URL (works when slug matches)
-import urllib.parse
-parts = page_url.rstrip('/').split('/')
-page_id  = int(parts[-2])   # 2577
-url_slug = parts[-1]         # 'sp500-pe-ratio-price-to-earnings-chart'
-```
-
----
-
-## Pattern 4: Economic indicator API
-
-Pages that use `generateChart()` in their JS load data from `/economic-data/{pageID}/{freq}`.
-This endpoint requires a `Referer` header matching the page URL.
-
-```python
-import json, datetime, gzip, urllib.request
-from helpers import http_get
-
-def get_economic_data(page_id: int, referer_url: str, freq: str = 'D') -> dict:
-    """
-    page_id:     numeric ID from the page URL (e.g. 2015 for Fed Funds Rate)
-    referer_url: the full page URL — required as Referer header
-    freq:        'D' = daily, 'M' = monthly (not all support both)
-    
-    Returns {'data': [[ts_ms, value], ...], 'metadata': {...}}
-    """
-    url = f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
-    headers = {
-        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
-        "Accept": "application/json, */*",
-        "Accept-Encoding": "gzip",
-        "Referer": referer_url,
-    }
-    with urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=20) as r:
-        raw = r.read()
-        if r.headers.get("Content-Encoding") == "gzip":
-            raw = gzip.decompress(raw)
-        result = json.loads(raw)
-    if result is None:
-        raise ValueError(f"pageID={page_id} does not support freq={freq!r}")
-    return result
-
-# Fed Funds Rate (daily, 25319 records)
-ffr = get_economic_data(2015, "https://www.macrotrends.net/2015/fed-funds-rate-historical-chart", freq='D')
-print(ffr['metadata']['name'])  # 'Fed Funds Interest Rate'
-print(ffr['metadata']['label']) # '%'
-
-# Convert timestamps to dates
-for ts_ms, value in ffr['data'][-3:]:
-    dt = datetime.datetime.fromtimestamp(ts_ms / 1000, datetime.UTC)
-    print(f"{dt.strftime('%Y-%m-%d')}: {value}%")
-# 2026-04-13: 3.64%
-# 2026-04-14: 3.64%
-# 2026-04-15: 3.64%
-
-# 10-Year Treasury yield (daily, 16074 records)
-t10 = get_economic_data(2016, "https://www.macrotrends.net/2016/10-year-treasury-bond-rate-yield-chart", freq='D')
-# Last: 2026-04-15: 4.29%
-
-# Gold prices (monthly, 1336 records, 1915-present) — template=5
-gold = get_economic_data(1333, "https://www.macrotrends.net/1333/historical-gold-prices-100-year-chart", freq='M')
-# metadata: {'name': 'Gold Prices', 'currency': '$', 'label': ''}
-
-# US Unemployment Rate (monthly, 938 records)
-unemp = get_economic_data(1316, "https://www.macrotrends.net/1316/us-national-unemployment-rate", freq='M')
-# metadata: {'name': 'U.S. Unemployment Rate', 'label': '%'}
-
-# Debt-to-GDP ratio (monthly, 712 records)
-debt_gdp = get_economic_data(1381, "https://www.macrotrends.net/1381/debt-to-gdp-ratio-historical-chart", freq='M')
-```
-
-### metadata fields
-
-```python
-{
-    'name':            'Fed Funds Interest Rate',  # chart title
-    'tableHeaderName': 'Fed Funds Interest Rate',
-    'currency':        '',            # '$' for dollar-denominated series
-    'label':           '%',          # units label
-    'chartType':       'line',
-    'mobileChartType': 'line',
-    'lineWidth':       2,
-    'positiveColor':   '#2caffe',
-    'negativeColor':   '',
-    'decimals':        '',
-    'chartScale':      'linear',
-    'seriesUnits':     ''
-}
-```
-
-### Available frequency codes
-
-| Code | Meaning | Notes |
-|------|---------|-------|
-| `D` | Daily | Most series support this |
-| `M` | Monthly | Returns `null` if not available |
-| `Q` | Quarterly | Usually `null` — use `M` instead |
-| `A` | Annual | Usually `null` — use `M` instead |
-| `DEFAULT` | Default (usually monthly) | Same data as `M` for most series |
-| `INDEXMONTHLY` | Monthly index close | Some commodity/index series |
-| `INDEXDAILY` | Daily index | Some series |
-| `DAILYEXCHANGERATE` | Daily FX rate | Currency pairs |
-| `10YD` | 10-year daily | Specialized series |
-
-Try `D` first, fall back to `M` if you get `null`.
-
-### Known economic page IDs
-
-| ID | URL slug | Description |
-|----|----------|-------------|
-| 1316 | us-national-unemployment-rate | U.S. Unemployment Rate (monthly, back to 1948) |
-| 1333 | historical-gold-prices-100-year-chart | Gold Prices (monthly, back to 1915) |
-| 1381 | debt-to-gdp-ratio-historical-chart | U.S. Debt to GDP Ratio |
-| 2015 | fed-funds-rate-historical-chart | Fed Funds Interest Rate (daily, back to 1954) |
-| 2016 | 10-year-treasury-bond-rate-yield-chart | 10-Year Treasury Yield (daily, back to 1962) |
-| 2577 | sp500-pe-ratio-price-to-earnings-chart | S&P 500 PE Ratio (uses `chart_iframe_comp.php`) |
-
----
-
-## Generic extraction helper
-
-One function that handles all three embedded-JS patterns:
-
-```python
-import json, re
-from helpers import http_get
-
-def extract_chart_var(html: str, var_name: str) -> list:
-    """Extract a JS array variable from Macrotrends iframe HTML."""
-    m = re.search(rf'var\s+{re.escape(var_name)}\s*=\s*\[', html)
-    if not m:
-        return []
-    si = html.index('[', m.start())
-    bc = 0
-    for j, ch in enumerate(html[si:], si):
-        if ch == '[':   bc += 1
-        elif ch == ']':
-            bc -= 1
-            if bc == 0:
-                return json.loads(html[si:j+1])
-    return []
-
-# Works for dataDaily, chartData, or originalData:
-html = http_get("https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t=AAPL&yb=15")
-daily = extract_chart_var(html, 'dataDaily')
-
-html2 = http_get("https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=2577&url=sp500-pe-ratio-price-to-earnings-chart")
-pe_data = extract_chart_var(html2, 'originalData')
-```
-
----
-
-## URL construction guide
-
-### Stock pages
-
-```python
-STOCK_BASE = "https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
-
-# Price history OHLCV
-f"{STOCK_BASE}stock_price_history.php?t={ticker}"                        # all history
-f"{STOCK_BASE}stock_price_history.php?t={ticker}&yb={years}"             # last N years
-
-# Market cap
-f"{STOCK_BASE}market_cap.php?t={ticker}&yb={years}"
-
-# Fundamentals
-f"{STOCK_BASE}fundamental_iframe.php?t={ticker}&type={type}&statement={stmt}&freq={freq}&sub=&yb={years}"
-# type/statement combos: pe-ratio/price-ratios, revenue/income-statement,
-#                        total-assets/balance-sheet, current-ratio/ratios
-
-# Metrics
-f"{STOCK_BASE}fundamental_metric.php?t={ticker}&chart={metric}&sub=&yb={years}"
-# metrics: profit-margin
-
-# Dividend yield
-f"{STOCK_BASE}dividend_yield.php?t={ticker}&yb={years}"
-```
-
-### Economic / index pages
-
-```python
-# From numeric ID + URL slug (read from page source or page URL)
-f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={id}&url={slug}"
-
-# Economic indicator JSON API (requires Referer header)
-f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
-```
-
----
-
-## Rate limits and anti-bot
-
-- **No rate limiting observed** at any tested volume. 10 rapid requests to the same stock iframe completed in 1.8s with no throttling, CAPTCHA, or 429 errors.
-- **Default UA works** (`Mozilla/5.0`) for most endpoints. The iframe PHP files never 403'd.
-- **Chrome UA needed** for some main HTML pages (not data endpoints): use when fetching `/stocks/charts/...` or `/2015/...` wrapper pages if you get 403. Switch to:
-  ```python
-  headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
-  ```
-- **Referer required** for `/economic-data/{id}/{freq}` — send the page URL as `Referer`. Without it, the request is allowed but you get a 403 on some pages.
-- **No cookies, sessions, or auth tokens** needed for any endpoint.
-
----
-
-## Gotchas
-
-**Main page URL ≠ data page:** Some URLs redirect to different content. `/1316/us-national-debt-by-year` redirects to `/1316/us-national-unemployment-rate`. Always check the final URL with `r.url` if the returned data looks wrong. Use the final URL as the Referer.
-
-**yb parameter controls history depth:**
-- `yb=1` → ~250 records (last year)
-- `yb=15` → ~3772 records (last 15 years)
-- omit → full history (AAPL: 11428 records to 1980; default for most queries)
-
-**Two iframe patterns for economic pages:** Pages at `macrotrends.net/NNNN/slug` use either `chart_iframe_comp.php` (→ `originalData`) or `generateChart` + `/economic-data/` API. Check the main page HTML to detect which:
-```python
-if 'chart_iframe_comp.php' in html:   # use extract_index_chart()
-elif 'highchartsURL' in html:          # use get_economic_data()
-```
-
-**Gold data has two price columns:**
-```python
-{'id': 'GOLDAMGBD228NLBM', 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
-# 'close'  = inflation-adjusted price (base year adjusts over time)
-# 'close1' = nominal USD price (the raw market price)
-```
-
-**Economic API frequency codes:** Only `D` and `M` consistently return data across most series. `A` and `Q` return `null` for most economic indicators. Always try `D` first.
-
-**chartData fields vary by metric:**
-- `market_cap.php` → `{'date', 'v1'}` (v1 = market cap in $B)
-- `fundamental_iframe.php` type=pe-ratio → `{'date', 'v1', 'v2', 'v3'}` (stock price, EPS, PE)
-- `fundamental_iframe.php` type=revenue → `{'date', 'v1', 'v2', 'v3'}` (TTM revenue, quarterly revenue, YoY%)
-- `fundamental_metric.php` chart=profit-margin → `{'date', 'v1', 'v2', 'v3'}` (gross%, operating%, net%)
-- `dividend_yield.php` → `{'date', 'c', 'ttm_d', 'ttm_dy'}` (price, dividend, yield%)
-
-**Bracket matching required for large arrays:** The `var dataDaily = [...]` in stock iframes is ~450KB with 3772 OHLCV records. The `re.DOTALL` greedy approach works but is slow; bracket-counting (`bc` pattern above) is O(n) and fast.
-
-**No public API for ticker lookup:** To find the company slug for a URL, check the search endpoint: `https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ticker_search_list.php?v=YYYYMMDD` — but the stock price iframe only needs the ticker symbol (`?t=AAPL`), not the slug.
+# Macrotrends — Data Extraction
+
+`https://www.macrotrends.net` — long-term historical financial and economic charts. Three access patterns depending on page type; all work with plain `http_get`, no browser required.
+
+All results validated against live site on 2026-04-18.
+
+## Do this first: pick your access pattern
+
+| Goal | Pattern | Latency | Variable |
+|------|---------|---------|----------|
+| Stock OHLCV price history | Direct iframe PHP | ~190ms | `dataDaily` |
+| Stock market cap (daily) | Direct iframe PHP | ~200ms | `chartData` |
+| Stock fundamentals (PE, revenue, margins) | Direct iframe PHP | ~140ms | `chartData` |
+| S&P 500 / composite index charts | `chart_iframe_comp.php` | ~90ms | `originalData` |
+| Economic indicators (rates, yields, CPI) | `/economic-data/` JSON API | ~150ms | `data[]` array |
+| Gold, commodity prices | Either path (both work) | ~150ms | `data[]` or `originalData` |
+
+**Never use the browser for Macrotrends read-only tasks.** All endpoints are accessible via `http_get` with the default `Mozilla/5.0` UA. For pages that occasionally 403, switch to a Chrome UA (see gotchas).
+
+---
+
+## Pattern 1: Stock price history (OHLCV)
+
+Construct the iframe URL directly — no need to fetch the main page first.
+
+```python
+import json, re
+from helpers import http_get
+
+def get_stock_ohlcv(ticker: str, years_back: int = None) -> list[dict]:
+    """
+    Returns daily OHLCV records for any US stock.
+
+    ticker:     uppercase ticker symbol, e.g. 'AAPL', 'MSFT', 'TSLA', 'NVDA'
+    years_back: number of years of history (1=~250 records, 15=~3772 records).
+                Omit (None) to get ALL available history (AAPL goes back to 1980).
+    """
+    url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t={ticker}"
+    if years_back:
+        url += f"&yb={years_back}"
+
+    html = http_get(url)
+    m = re.search(r'var\s+dataDaily\s*=\s*\[', html)
+    if not m:
+        raise ValueError(f"No dataDaily found for ticker {ticker!r}")
+
+    si = html.index('[', m.start())
+    bc = 0
+    for j, ch in enumerate(html[si:], si):
+        if ch == '[':   bc += 1
+        elif ch == ']':
+            bc -= 1
+            if bc == 0: ei = j; break
+    return json.loads(html[si:ei+1])
+
+# Usage
+records = get_stock_ohlcv('AAPL', years_back=15)
+# [{'d': '2011-04-18', 'o': '9.771', 'h': '9.9547', 'l': '9.593', 'c': '9.9433', 'v': '18.275'}, ...]
+
+latest = records[-1]
+# {'d': '2026-04-17', 'o': '266.96', 'h': '272.3', 'l': '266.72', 'c': '270.23',
+#  'v': '55.211', 'ma50': '260.554', 'ma200': '251.828'}
+
+print(f"{latest['d']}: close=${latest['c']} vol={latest['v']}M shares")
+```
+
+### dataDaily field reference
+
+| Field | Meaning | Type |
+|-------|---------|------|
+| `d` | Date (YYYY-MM-DD) | str |
+| `o` | Open price (adjusted for splits) | str/float |
+| `h` | High | str/float |
+| `l` | Low | str/float |
+| `c` | Close | str/float |
+| `v` | Volume in **millions of shares** | str/float |
+| `ma50` | 50-day moving average | str/float (appears on recent records only) |
+| `ma200` | 200-day moving average | str/float (appears on recent records only) |
+
+**Note:** All price values are strings — cast with `float()`. Volume is millions: `55.211` = 55.2M shares traded.
+
+### Confirmed tickers (2026-04-18)
+
+All tested with direct iframe URL, no page fetch needed:
+
+```python
+# All work: AAPL, MSFT, TSLA, NVDA, GOOGL, AMZN, META, NFLX, etc.
+# 3772 records for yb=15 (goes back to 2011-04-18)
+# AAPL full history: 11428 records back to 1980-12-12
+```
+
+---
+
+## Pattern 2: Stock fundamentals (PE ratio, revenue, market cap, margins)
+
+Different PHP files depending on metric. Construct directly.
+
+### Market cap (daily, in billions USD)
+
+```python
+import json, re
+from helpers import http_get
+
+def get_market_cap(ticker: str, years_back: int = 15) -> list[dict]:
+    url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/market_cap.php?t={ticker}&yb={years_back}"
+    html = http_get(url)
+    m = re.search(r'var\s+chartData\s*=\s*\[', html)
+    si = html.index('[', m.start())
+    bc = 0
+    for j, ch in enumerate(html[si:], si):
+        if ch == '[':   bc += 1
+        elif ch == ']':
+            bc -= 1
+            if bc == 0: ei = j; break
+    return json.loads(html[si:ei+1])
+
+data = get_market_cap('AAPL')
+# [{'date': '2026-04-15', 'v1': 3929.35}, {'date': '2026-04-16', 'v1': 3884.67}, ...]
+# v1 = market cap in billions USD
+```
+
+### PE ratio, revenue, current ratio (quarterly/annual fundamentals)
+
+```python
+import json, re
+from helpers import http_get
+
+def get_fundamental(ticker: str, metric_type: str, statement: str,
+                    freq: str = 'Q', years_back: int = 15) -> list[dict]:
+    """
+    freq: 'Q' = quarterly, 'A' = annual
+    """
+    url = (
+        f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
+        f"fundamental_iframe.php?t={ticker}&type={metric_type}&statement={statement}"
+        f"&freq={freq}&sub=&yb={years_back}"
+    )
+    html = http_get(url)
+    m = re.search(r'var\s+chartData\s*=\s*\[', html)
+    si = html.index('[', m.start())
+    bc = 0
+    for j, ch in enumerate(html[si:], si):
+        if ch == '[':   bc += 1
+        elif ch == ']':
+            bc -= 1
+            if bc == 0: ei = j; break
+    return json.loads(html[si:ei+1])
+
+# PE ratio
+pe = get_fundamental('AAPL', 'pe-ratio', 'price-ratios')
+# [{'date': '2025-09-30', 'v1': 254.146, 'v2': 7.46, 'v3': 34.07}, ...]
+# v1 = stock price, v2 = quarterly EPS, v3 = PE ratio
+
+# Revenue
+rev = get_fundamental('AAPL', 'revenue', 'income-statement')
+# [{'date': '2025-12-31', 'v1': 435.617, 'v2': 143.756, 'v3': 15.65}, ...]
+# v1 = TTM revenue ($B), v2 = quarterly revenue ($B), v3 = YoY growth %
+
+# Total assets
+assets = get_fundamental('AAPL', 'total-assets', 'balance-sheet')
+
+# Current ratio
+ratio = get_fundamental('AAPL', 'current-ratio', 'ratios')
+```
+
+### Profit margins
+
+```python
+def get_profit_margins(ticker: str, years_back: int = 15) -> list[dict]:
+    url = (
+        f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
+        f"fundamental_metric.php?t={ticker}&chart=profit-margin&sub=&yb={years_back}"
+    )
+    html = http_get(url)
+    m = re.search(r'var\s+chartData\s*=\s*\[', html)
+    si = html.index('[', m.start())
+    bc = 0
+    for j, ch in enumerate(html[si:], si):
+        if ch == '[':   bc += 1
+        elif ch == ']':
+            bc -= 1
+            if bc == 0: ei = j; break
+    return json.loads(html[si:ei+1])
+
+margins = get_profit_margins('AAPL')
+# [{'date': '2025-12-31', 'v1': 47.33, 'v2': 32.38, 'v3': 27.04}, ...]
+# v1 = gross margin %, v2 = operating margin %, v3 = net margin %
+```
+
+### Dividend yield
+
+```python
+def get_dividend_yield(ticker: str, years_back: int = 15) -> list[dict]:
+    url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/dividend_yield.php?t={ticker}&yb={years_back}"
+    html = http_get(url)
+    m = re.search(r'var\s+chartData\s*=\s*\[', html)
+    si = html.index('[', m.start())
+    bc = 0
+    for j, ch in enumerate(html[si:], si):
+        if ch == '[':   bc += 1
+        elif ch == ']':
+            bc -= 1
+            if bc == 0: ei = j; break
+    return json.loads(html[si:ei+1])
+
+dy = get_dividend_yield('AAPL')
+# [{'date': '2026-04-17', 'c': 270.23, 'ttm_d': 1.03848, 'ttm_dy': 0.3843}, ...]
+# c = stock price, ttm_d = TTM dividend ($), ttm_dy = TTM yield (%)
+```
+
+### Stock metric URL reference
+
+| Metric | PHP file | Extra params |
+|--------|----------|-------------|
+| Stock price OHLCV | `stock_price_history.php` | — |
+| Market cap (daily) | `market_cap.php` | — |
+| Dividend yield | `dividend_yield.php` | — |
+| Stock splits (price history) | `stock_splits.php` | — |
+| PE ratio | `fundamental_iframe.php` | `type=pe-ratio&statement=price-ratios` |
+| Revenue | `fundamental_iframe.php` | `type=revenue&statement=income-statement` |
+| Total assets | `fundamental_iframe.php` | `type=total-assets&statement=balance-sheet` |
+| Current ratio | `fundamental_iframe.php` | `type=current-ratio&statement=ratios` |
+| Profit margins | `fundamental_metric.php` | `chart=profit-margin` |
+
+Base URL prefix: `https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/`
+
+All take `?t={TICKER}&yb={N}` (or `&sub=&yb={N}` for the fundamental ones).
+
+---
+
+## Pattern 3: Index and composite charts (S&P 500, Shiller PE, etc.)
+
+These pages embed chart data via `chart_iframe_comp.php`. The variable is `originalData`.
+
+```python
+import json, re
+from helpers import http_get
+
+def extract_index_chart(page_id: int, url_slug: str) -> list[dict]:
+    """
+    page_id:  the numeric ID from the page URL, e.g. 2577
+    url_slug: last segment of the page URL, e.g. 'sp500-pe-ratio-price-to-earnings-chart'
+    """
+    url = f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={page_id}&url={url_slug}"
+    html = http_get(url)
+    m = re.search(r'var\s+originalData\s*=\s*\[', html)
+    if not m:
+        raise ValueError("originalData not found — this page may use a different pattern")
+    si = html.index('[', m.start())
+    bc = 0
+    for j, ch in enumerate(html[si:], si):
+        if ch == '[':   bc += 1
+        elif ch == ']':
+            bc -= 1
+            if bc == 0: ei = j; break
+    return json.loads(html[si:ei+1])
+
+# S&P 500 PE ratio (1180 monthly records, 1927-2026)
+pe_data = extract_index_chart(2577, 'sp500-pe-ratio-price-to-earnings-chart')
+# [{'date': '1927-12-01', 'close': '15.9099'}, ..., {'date': '2026-03-01', 'close': '27.8925'}]
+# 'close' is the PE ratio value
+
+# Gold prices (1336 monthly records, 1915-2026)
+gold_data = extract_index_chart(1333, 'historical-gold-prices-100-year-chart')
+# [{'id': 'GOLDAMGBD228NLBM', 'date': '1915-01-01', 'close': '629.36', 'close1': '19.250'}, ...]
+# 'close' = inflation-adjusted price, 'close1' = nominal USD price
+
+print(f"Latest S&P PE: {pe_data[-1]}")   # {'date': '2026-03-01', 'close': '27.8925'}
+print(f"Latest gold:   {gold_data[-1]}") # {'id': ..., 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
+```
+
+### Detecting which pattern a page uses
+
+```python
+def get_page_pattern(page_url: str) -> str:
+    html = http_get(page_url)
+    if 'chart_iframe_comp.php' in html:
+        return 'index_chart'           # use extract_index_chart()
+    elif 'generateChart' in html and 'highchartsURL' in html:
+        return 'economic_api'          # use get_economic_data()
+    elif '/production/stocks/desktop/PRODUCTION/' in html:
+        return 'stock_iframe'          # use get_stock_ohlcv() etc.
+    return 'unknown'
+```
+
+### To get the ID and slug from a page
+
+```python
+import re
+from helpers import http_get
+
+page_url = "https://www.macrotrends.net/2577/sp500-pe-ratio-price-to-earnings-chart"
+html = http_get(page_url)
+
+# Option A: parse from the iframe src in the HTML
+m = re.search(r'chart_iframe_comp\.php\?id=(\d+)&url=([^"&]+)', html)
+if m:
+    page_id, url_slug = int(m.group(1)), m.group(2)
+
+# Option B: derive from the page URL (works when slug matches)
+import urllib.parse
+parts = page_url.rstrip('/').split('/')
+page_id  = int(parts[-2])   # 2577
+url_slug = parts[-1]         # 'sp500-pe-ratio-price-to-earnings-chart'
+```
+
+---
+
+## Pattern 4: Economic indicator API
+
+Pages that use `generateChart()` in their JS load data from `/economic-data/{pageID}/{freq}`.
+This endpoint requires a `Referer` header matching the page URL.
+
+```python
+import json, datetime, gzip, urllib.request
+from helpers import http_get
+
+def get_economic_data(page_id: int, referer_url: str, freq: str = 'D') -> dict:
+    """
+    page_id:     numeric ID from the page URL (e.g. 2015 for Fed Funds Rate)
+    referer_url: the full page URL — required as Referer header
+    freq:        'D' = daily, 'M' = monthly (not all support both)
+    
+    Returns {'data': [[ts_ms, value], ...], 'metadata': {...}}
+    """
+    url = f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
+    headers = {
+        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+        "Accept": "application/json, */*",
+        "Accept-Encoding": "gzip",
+        "Referer": referer_url,
+    }
+    with urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=20) as r:
+        raw = r.read()
+        if r.headers.get("Content-Encoding") == "gzip":
+            raw = gzip.decompress(raw)
+        result = json.loads(raw)
+    if result is None:
+        raise ValueError(f"pageID={page_id} does not support freq={freq!r}")
+    return result
+
+# Fed Funds Rate (daily, 25319 records)
+ffr = get_economic_data(2015, "https://www.macrotrends.net/2015/fed-funds-rate-historical-chart", freq='D')
+print(ffr['metadata']['name'])  # 'Fed Funds Interest Rate'
+print(ffr['metadata']['label']) # '%'
+
+# Convert timestamps to dates
+for ts_ms, value in ffr['data'][-3:]:
+    dt = datetime.datetime.fromtimestamp(ts_ms / 1000, datetime.UTC)
+    print(f"{dt.strftime('%Y-%m-%d')}: {value}%")
+# 2026-04-13: 3.64%
+# 2026-04-14: 3.64%
+# 2026-04-15: 3.64%
+
+# 10-Year Treasury yield (daily, 16074 records)
+t10 = get_economic_data(2016, "https://www.macrotrends.net/2016/10-year-treasury-bond-rate-yield-chart", freq='D')
+# Last: 2026-04-15: 4.29%
+
+# Gold prices (monthly, 1336 records, 1915-present) — template=5
+gold = get_economic_data(1333, "https://www.macrotrends.net/1333/historical-gold-prices-100-year-chart", freq='M')
+# metadata: {'name': 'Gold Prices', 'currency': '$', 'label': ''}
+
+# US Unemployment Rate (monthly, 938 records)
+unemp = get_economic_data(1316, "https://www.macrotrends.net/1316/us-national-unemployment-rate", freq='M')
+# metadata: {'name': 'U.S. Unemployment Rate', 'label': '%'}
+
+# Debt-to-GDP ratio (monthly, 712 records)
+debt_gdp = get_economic_data(1381, "https://www.macrotrends.net/1381/debt-to-gdp-ratio-historical-chart", freq='M')
+```
+
+### metadata fields
+
+```python
+{
+    'name':            'Fed Funds Interest Rate',  # chart title
+    'tableHeaderName': 'Fed Funds Interest Rate',
+    'currency':        '',            # '$' for dollar-denominated series
+    'label':           '%',          # units label
+    'chartType':       'line',
+    'mobileChartType': 'line',
+    'lineWidth':       2,
+    'positiveColor':   '#2caffe',
+    'negativeColor':   '',
+    'decimals':        '',
+    'chartScale':      'linear',
+    'seriesUnits':     ''
+}
+```
+
+### Available frequency codes
+
+| Code | Meaning | Notes |
+|------|---------|-------|
+| `D` | Daily | Most series support this |
+| `M` | Monthly | Returns `null` if not available |
+| `Q` | Quarterly | Usually `null` — use `M` instead |
+| `A` | Annual | Usually `null` — use `M` instead |
+| `DEFAULT` | Default (usually monthly) | Same data as `M` for most series |
+| `INDEXMONTHLY` | Monthly index close | Some commodity/index series |
+| `INDEXDAILY` | Daily index | Some series |
+| `DAILYEXCHANGERATE` | Daily FX rate | Currency pairs |
+| `10YD` | 10-year daily | Specialized series |
+
+Try `D` first, fall back to `M` if you get `null`.
+
+### Known economic page IDs
+
+| ID | URL slug | Description |
+|----|----------|-------------|
+| 1316 | us-national-unemployment-rate | U.S. Unemployment Rate (monthly, back to 1948) |
+| 1333 | historical-gold-prices-100-year-chart | Gold Prices (monthly, back to 1915) |
+| 1381 | debt-to-gdp-ratio-historical-chart | U.S. Debt to GDP Ratio |
+| 2015 | fed-funds-rate-historical-chart | Fed Funds Interest Rate (daily, back to 1954) |
+| 2016 | 10-year-treasury-bond-rate-yield-chart | 10-Year Treasury Yield (daily, back to 1962) |
+| 2577 | sp500-pe-ratio-price-to-earnings-chart | S&P 500 PE Ratio (uses `chart_iframe_comp.php`) |
+
+---
+
+## Generic extraction helper
+
+One function that handles all three embedded-JS patterns:
+
+```python
+import json, re
+from helpers import http_get
+
+def extract_chart_var(html: str, var_name: str) -> list:
+    """Extract a JS array variable from Macrotrends iframe HTML."""
+    m = re.search(rf'var\s+{re.escape(var_name)}\s*=\s*\[', html)
+    if not m:
+        return []
+    si = html.index('[', m.start())
+    bc = 0
+    for j, ch in enumerate(html[si:], si):
+        if ch == '[':   bc += 1
+        elif ch == ']':
+            bc -= 1
+            if bc == 0:
+                return json.loads(html[si:j+1])
+    return []
+
+# Works for dataDaily, chartData, or originalData:
+html = http_get("https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t=AAPL&yb=15")
+daily = extract_chart_var(html, 'dataDaily')
+
+html2 = http_get("https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=2577&url=sp500-pe-ratio-price-to-earnings-chart")
+pe_data = extract_chart_var(html2, 'originalData')
+```
+
+---
+
+## URL construction guide
+
+### Stock pages
+
+```python
+STOCK_BASE = "https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
+
+# Price history OHLCV
+f"{STOCK_BASE}stock_price_history.php?t={ticker}"                        # all history
+f"{STOCK_BASE}stock_price_history.php?t={ticker}&yb={years}"             # last N years
+
+# Market cap
+f"{STOCK_BASE}market_cap.php?t={ticker}&yb={years}"
+
+# Fundamentals
+f"{STOCK_BASE}fundamental_iframe.php?t={ticker}&type={type}&statement={stmt}&freq={freq}&sub=&yb={years}"
+# type/statement combos: pe-ratio/price-ratios, revenue/income-statement,
+#                        total-assets/balance-sheet, current-ratio/ratios
+
+# Metrics
+f"{STOCK_BASE}fundamental_metric.php?t={ticker}&chart={metric}&sub=&yb={years}"
+# metrics: profit-margin
+
+# Dividend yield
+f"{STOCK_BASE}dividend_yield.php?t={ticker}&yb={years}"
+```
+
+### Economic / index pages
+
+```python
+# From numeric ID + URL slug (read from page source or page URL)
+f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={id}&url={slug}"
+
+# Economic indicator JSON API (requires Referer header)
+f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
+```
+
+---
+
+## Rate limits and anti-bot
+
+- **No rate limiting observed** at any tested volume. 10 rapid requests to the same stock iframe completed in 1.8s with no throttling, CAPTCHA, or 429 errors.
+- **Default UA works** (`Mozilla/5.0`) for most endpoints. The iframe PHP files never 403'd.
+- **Chrome UA needed** for some main HTML pages (not data endpoints): use when fetching `/stocks/charts/...` or `/2015/...` wrapper pages if you get 403. Switch to:
+  ```python
+  headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
+  ```
+- **Referer required** for `/economic-data/{id}/{freq}` — send the page URL as `Referer`. Without it, the request is allowed but you get a 403 on some pages.
+- **No cookies, sessions, or auth tokens** needed for any endpoint.
+
+---
+
+## Gotchas
+
+**Main page URL ≠ data page:** Some URLs redirect to different content. `/1316/us-national-debt-by-year` redirects to `/1316/us-national-unemployment-rate`. Always check the final URL with `r.url` if the returned data looks wrong. Use the final URL as the Referer.
+
+**yb parameter controls history depth:**
+- `yb=1` → ~250 records (last year)
+- `yb=15` → ~3772 records (last 15 years)
+- omit → full history (AAPL: 11428 records to 1980; default for most queries)
+
+**Two iframe patterns for economic pages:** Pages at `macrotrends.net/NNNN/slug` use either `chart_iframe_comp.php` (→ `originalData`) or `generateChart` + `/economic-data/` API. Check the main page HTML to detect which:
+```python
+if 'chart_iframe_comp.php' in html:   # use extract_index_chart()
+elif 'highchartsURL' in html:          # use get_economic_data()
+```
+
+**Gold data has two price columns:**
+```python
+{'id': 'GOLDAMGBD228NLBM', 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
+# 'close'  = inflation-adjusted price (base year adjusts over time)
+# 'close1' = nominal USD price (the raw market price)
+```
+
+**Economic API frequency codes:** Only `D` and `M` consistently return data across most series. `A` and `Q` return `null` for most economic indicators. Always try `D` first.
+
+**chartData fields vary by metric:**
+- `market_cap.php` → `{'date', 'v1'}` (v1 = market cap in $B)
+- `fundamental_iframe.php` type=pe-ratio → `{'date', 'v1', 'v2', 'v3'}` (stock price, EPS, PE)
+- `fundamental_iframe.php` type=revenue → `{'date', 'v1', 'v2', 'v3'}` (TTM revenue, quarterly revenue, YoY%)
+- `fundamental_metric.php` chart=profit-margin → `{'date', 'v1', 'v2', 'v3'}` (gross%, operating%, net%)
+- `dividend_yield.php` → `{'date', 'c', 'ttm_d', 'ttm_dy'}` (price, dividend, yield%)
+
+**Bracket matching required for large arrays:** The `var dataDaily = [...]` in stock iframes is ~450KB with 3772 OHLCV records. The `re.DOTALL` greedy approach works but is slow; bracket-counting (`bc` pattern above) is O(n) and fast.
+
+**No public API for ticker lookup:** To find the company slug for a URL, check the search endpoint: `https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ticker_search_list.php?v=YYYYMMDD` — but the stock price iframe only needs the ticker symbol (`?t=AAPL`), not the slug.