catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/tradingview/scraping.md

@@ -0,0 +1,309 @@
+# TradingView — Scraping & Data Extraction
+
+`https://www.tradingview.com` — charting platform with multiple internal REST APIs. Stock/crypto/forex screener and symbol search work without auth. Use `http_get` or raw `urllib` for all workflows except JS-rendered chart pages.
+
+## Do this first
+
+**Use the scanner API for bulk screener data — one POST, no browser, full column control.**
+
+```python
+import json, urllib.request
+
+def tv_scan(payload, market="america"):
+    data = json.dumps(payload).encode()
+    req = urllib.request.Request(
+        f"https://scanner.tradingview.com/{market}/scan",
+        data=data,
+        headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
+    )
+    with urllib.request.urlopen(req, timeout=20) as r:
+        return json.loads(r.read())
+```
+
+**No auth, no Referer, no cookies required for the scanner.** Responses arrive in ~200ms.
+
+## Common workflows
+
+### Top stocks by market cap (screener)
+
+```python
+import json, urllib.request
+
+payload = {
+    "filter": [],
+    "options": {"lang": "en"},
+    "columns": ["name", "close", "change", "volume", "market_cap_basic"],
+    "sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
+    "range": [0, 10]   # [start, end] — half-open, so this returns rows 0–9
+}
+
+data = json.dumps(payload).encode()
+req = urllib.request.Request(
+    "https://scanner.tradingview.com/america/scan",
+    data=data,
+    headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
+)
+with urllib.request.urlopen(req, timeout=20) as r:
+    resp = json.loads(r.read())
+
+# resp["totalCount"] = 19549 (all US-listed instruments)
+# resp["data"] is a list of {"s": "NASDAQ:NVDA", "d": [col0, col1, ...]}
+# "d" values align positionally with "columns" in the payload
+
+cols = payload["columns"]
+for item in resp["data"]:
+    row = dict(zip(cols, item["d"]))
+    symbol = item["s"]   # e.g. "NASDAQ:AAPL"
+    print(symbol, row["close"], row["change"], row["market_cap_basic"])
+# NASDAQ:NVDA  201.68  1.68  4900823822021.0
+# NASDAQ:AAPL  270.23  2.59  3967284528489.0
+# ...
+```
+
+**Critical**: `"d"` is a plain positional array — index 0 = columns[0], index 1 = columns[1], etc. There are no keys in the row data itself.
+
+### Pagination
+
+```python
+# Page 1: range [0, 20]
+# Page 2: range [20, 40]
+payload["range"] = [20, 40]
+```
+
+### Filtering stocks
+
+```python
+payload = {
+    "filter": [
+        {"left": "market_cap_basic", "operation": "greater", "right": 10_000_000_000},
+        {"left": "volume",           "operation": "greater", "right": 5_000_000},
+        {"left": "change",           "operation": "in_range", "right": [2, 10]},
+        {"left": "exchange",         "operation": "equal",   "right": "NASDAQ"},
+        {"left": "sector",           "operation": "equal",   "right": "Electronic Technology"},
+    ],
+    "columns": ["name", "close", "change", "volume", "market_cap_basic",
+                "description", "sector", "industry"],
+    "sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
+    "range": [0, 20]
+}
+```
+
+Valid filter operations: `greater`, `less`, `equal`, `in_range` (right = [min, max]), `match` (substring on `name`).
+
+Sector names use TradingView taxonomy (not GICS). Confirmed working values:
+- `"Electronic Technology"` — NVDA, AAPL, TSM
+- `"Technology Services"` — MSFT, GOOGL, META
+- `"Finance"`, `"Health Technology"`, `"Consumer Non-Durables"`
+
+### Full list of tested valid column names
+
+```python
+# Price & volume
+"name"                     # ticker (e.g. "AAPL")
+"description"              # full name ("Apple Inc.")
+"close"                    # last price
+"open", "high", "low"
+"volume"
+"change"                   # % change today
+"change_abs"               # absolute price change
+"change|1M"                # 1-month % change (also: |6M, |1Y)
+"High.1M", "High.6M"       # period high
+"High.All", "Low.All"      # all-time high/low
+"price_52_week_high"       # confirmed works
+"price_52_week_low"        # confirmed works
+"premarket_change"         # pre-market %
+"postmarket_change"        # after-hours %
+"gap"                      # overnight gap %
+"change_from_open_abs"     # intraday move from open
+"average_volume_10d_calc"  # 10-day avg volume
+"relative_volume_10d_calc" # relative volume vs 10-day avg
+"relative_volume_intraday|5"  # intraday relative vol (5m bars)
+
+# Fundamentals
+"market_cap_basic"          # market cap in USD
+"earnings_per_share_diluted_ttm"  # EPS TTM
+"price_earnings_ttm"        # P/E TTM
+"P/E"                       # P/E (snapshot)
+"dividends_yield"           # dividend yield %
+"beta_1_year"               # beta
+"float_shares_outstanding"  # float shares
+
+# Technical ratings & indicators
+"Recommend.All"   # composite rating: -1 (strong sell) to +1 (strong buy)
+"RSI"             # RSI 14
+"MACD.macd"       # MACD line
+
+# Classification
+"sector", "industry", "country", "exchange"
+"type"    # "stock", "fund", "dr" (depository receipt), etc.
+
+# NOTE: "52_week_high" / "52_week_low" are INVALID — use "price_52_week_high" / "price_52_week_low"
+# NOTE: "EPS_diluted_net" is INVALID — use "earnings_per_share_diluted_ttm"
+```
+
+Bad columns return HTTP 400 with `{"error": "Unknown field \"X\""}`.
+
+### Other scanner markets
+
+```python
+# market argument options (confirmed working):
+# "america"  — US equities (19,549 instruments)
+# "crypto"   — crypto across exchanges (56,455 instruments)
+# "forex"    — FX pairs (6,401 instruments)
+# "futures"  — futures (53,947 instruments)
+
+# Crypto example
+payload = {
+    "filter": [],
+    "columns": ["name", "close", "change", "volume", "market_cap_calc"],
+    "sort": {"sortBy": "market_cap_calc", "sortOrder": "desc"},
+    "range": [0, 10]
+}
+resp = tv_scan(payload, market="crypto")
+# Returns BTC, ETH, etc. across Binance, Bybit, OKX...
+```
+
+### Symbol search (requires Origin header)
+
+```python
+import json, urllib.request
+
+def symbol_search(query, exchange="", type_filter="", limit=50):
+    url = (
+        f"https://symbol-search.tradingview.com/symbol_search/v3/"
+        f"?text={query}&hl=1&exchange={exchange}&lang=en"
+        f"&search_type={type_filter or 'undefined'}&domain=production"
+    )
+    req = urllib.request.Request(url, headers={
+        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
+        "Origin": "https://www.tradingview.com",   # REQUIRED — 403 without this
+    })
+    with urllib.request.urlopen(req, timeout=15) as r:
+        return json.loads(r.read())
+
+result = symbol_search("AAPL")
+# result["symbols_remaining"] = 137
+# result["symbols"] = list of up to 50 matches
+# result["symbols"][0] keys:
+#   symbol, description, type, exchange, country, currency_code,
+#   cusip, isin, cik_code, logoid, provider_id, source_id,
+#   is_primary_listing, typespecs
+```
+
+**Gotcha**: `symbol-search.tradingview.com` requires `Origin: https://www.tradingview.com`. Referer alone is not enough. The scanner API does NOT need Origin or Referer.
+
+Filter by exchange and type:
+
+```python
+# Exact match on NASDAQ:AAPL
+result = symbol_search("AAPL", exchange="NASDAQ", type_filter="stock")
+# Returns 1 result — exact symbol only when exchange is specified
+```
+
+### News headlines for a symbol
+
+```python
+import json, urllib.request
+
+def get_news(symbol, limit=20):
+    # symbol format: "NASDAQ:AAPL", "NYSE:TSLA"
+    url = (
+        f"https://news-headlines.tradingview.com/v2/view/headlines/symbol"
+        f"?symbol={symbol}&client=web&streaming=false&lang=en&limit={limit}"
+    )
+    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=15) as r:
+        data = json.loads(r.read())
+    return data["items"]  # list of news items
+
+items = get_news("NASDAQ:AAPL", limit=10)
+# item keys: id, title, provider, sourceLogoId, published (unix ts),
+#            source, urgency, link, permission, relatedSymbols, storyPath
+# example:
+# items[0]["title"]     = "Apple Clears Major Legal Hurdle..."
+# items[0]["published"] = 1776472317  (unix timestamp)
+# items[0]["link"]      = "https://stocktwits.com/..."
+# items[0]["relatedSymbols"] = [{"symbol": "NASDAQ:AAPL", "logoid": "apple"}]
+```
+
+No auth or special headers needed. Returns up to 200 items per request.
+
+### Published trading ideas feed
+
+```python
+import json, urllib.request
+
+def get_ideas(sort="trending", page=1, symbol=None):
+    # Valid sort values (others return 400):
+    # "trending", "recent", "latest_popular", "week_popular",
+    # "suggested", "recent_extended", "picked_time"
+    url = f"https://www.tradingview.com/api/v1/ideas/?lang=en&sort={sort}&page={page}"
+    if symbol:
+        url += f"&symbol={symbol}"  # e.g. "NASDAQ:AAPL"
+    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=15) as r:
+        return json.loads(r.read())
+
+data = get_ideas("trending")
+# data["count"]      = 1000 (always 1000 — soft cap)
+# data["page_size"]  = 20
+# data["page_count"] = 50
+# data["next"]       = "https://www.tradingview.com/api/v1/ideas/?page=2"
+# data["results"]    = list of idea objects
+
+idea = data["results"][0]
+# idea keys: id, name, description, created_at, chart_url, views_count,
+#            likes_count, comments_count, is_video, is_education, is_hot,
+#            symbol (dict with name/exchange/type/interval/direction),
+#            user (dict with username/is_pro/badges), image (big/middle URLs)
+# idea["symbol"]["direction"]: 1=long, 2=short, 0=neutral
+
+# Filter by symbol:
+aapl_ideas = get_ideas(symbol="NASDAQ:AAPL")
+```
+
+## API summary table
+
+| Endpoint | Auth | Headers needed | Speed |
+|---|---|---|---|
+| `scanner.tradingview.com/{market}/scan` | None | None | ~200ms |
+| `symbol-search.tradingview.com/symbol_search/v3/` | None | `Origin: https://www.tradingview.com` | ~150ms |
+| `symbol-search.tradingview.com/symbol_search/` (v1) | None | `Origin: https://www.tradingview.com` | ~100ms |
+| `news-headlines.tradingview.com/v2/view/headlines/symbol` | None | None | ~400ms |
+| `www.tradingview.com/api/v1/ideas/` | None | None | ~300ms |
+| `data.tradingview.com/quotes/` | None | None | **Dead** — connection refused |
+| `economic-calendar.tradingview.com/events` | Yes | — | HTTP 403 |
+
+## Gotchas
+
+**Scanner `range` is half-open**: `[0, 10]` returns rows 0–9 (10 rows total). `[10, 20]` for the next page.
+
+**Column order is critical**: The `"d"` array in each result row is positional — it exactly mirrors your `"columns"` array. Always zip them: `dict(zip(columns, item["d"]))`.
+
+**`data.tradingview.com/quotes/` is dead**: The URL `https://data.tradingview.com/quotes/?symbols=NASDAQ:AAPL` closes the connection without a response. Use the scanner API instead for real-time quotes.
+
+**Scanner needs no Referer**: `scanner.tradingview.com` works with just `User-Agent`. The symbol-search subdomain checks `Origin` (CORS enforcement on the server side).
+
+**Symbol search highlights**: The v3 endpoint wraps matched text in `<em>` tags (e.g. `"<em>AAPL</em>"`). Strip them: `re.sub(r'</?em>', '', symbol["symbol"])`.
+
+**Ideas sort validation**: Only specific values work. `"sort=popular"` returns 400. Use `"trending"`, `"recent"`, `"latest_popular"`, `"week_popular"`, `"suggested"`.
+
+**Ideas count cap**: The API always reports `count=1000` regardless of actual corpus size. With `page_size=20`, max pages is 50.
+
+**Scanner server is AWS CloudFront** (`X-Amz-Cf-Pop` header) with a custom `Server: tv` — no Cloudflare. No anti-bot on the scanner subdomain. Main `www.tradingview.com` is a React SPA with `window.initData = {}` (empty — no embedded data). All data is loaded via API calls after hydration.
+
+**Rate limits**: No 429s observed in testing. 5 concurrent scanner calls complete in ~1s. Symbol search returns `symbols_remaining` in the response (counts against some quota — varies 90–180 across calls but never blocks). Observed no blocking after 15 rapid calls in a row.
+
+**Sector names**: Use TradingView's own taxonomy, not GICS. "Technology" does not exist — use `"Electronic Technology"` (hardware/semis) or `"Technology Services"` (software/internet).
+
+## When to use the browser
+
+The charting UI (`/chart/`), symbol detail pages (`/symbols/NASDAQ-AAPL/`), and the ideas page (`/ideas/`) are React SPAs — their visible data comes from the APIs above, not embedded HTML. Use browser + JS extraction only if you need visual chart screenshots or data from auth-gated pages (watchlists, portfolio, paper trading).
+
+```python
+# Only if you need a chart screenshot:
+goto_url("https://www.tradingview.com/chart/?symbol=NASDAQ:AAPL")
+wait_for_load()
+wait(3)   # chart renders asynchronously after readyState
+capture_screenshot("/tmp/aapl_chart.png", full=False)
+```

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/trello/boards-and-lists.md

@@ -0,0 +1,88 @@
+# Trello — boards and lists
+
+Read-only pattern for Trello: land on a board URL, scrape lists and their
+cards. Works for signed-in agents that inherit Tom's Chrome session —
+no login dance, no captcha.
+
+## URL patterns
+
+- `https://trello.com/` — marketing home for anonymous; when signed in,
+  redirects to `https://trello.com/u/<username>/boards` (the dashboard).
+  Note: `https://trello.com/boards` (no `/u/<username>`) returns an
+  error page for signed-in users. Always route via `/u/<username>/boards`.
+- Board URL: `https://trello.com/b/<8-char-boardId>/<slug>`. The `slug`
+  is cosmetic — the boardId is the canonical identifier.
+- Card URL: `https://trello.com/c/<8-char-cardId>/<n>-<slug>`, where
+  `<n>` is the card's 1-based position within its list at creation time.
+
+## Stable selectors
+
+- List: `[data-testid="list"]` — or `[data-list-id]` for the id. Returns
+  exactly the lists on the board, no sidebar noise.
+- List header: within a list, `[data-testid="list-header-name"]`.
+  Falling back to `h2` inside the list also works, but the board's
+  outer chrome has h2s too (sidebar plans etc.) — stay scoped to the
+  list element.
+- Card: `a[href*="/c/"]` — scoped to each list element. The `href` is
+  the canonical link; the visible text is the card name.
+- Card name within the anchor: `[data-testid="card-name"]`. Anchor
+  `innerText` is a reasonable fallback.
+
+## Site structure
+
+The board page is a single-page React app. All lists render in one pass
+into horizontally scrolling columns. Cards within a list are vertical.
+Count via `document.querySelectorAll('[data-testid="list"]').length` —
+matches the visible column count.
+
+## Framework / interaction quirks
+
+- `goto_url('https://trello.com/')` redirects asynchronously to the user's
+  boards dashboard. After `wait_for_load()`, the URL in `page_info()`
+  will be `.../u/<username>/boards`. Don't hard-code the username;
+  read it back from `page_info()['url']` if you need it.
+- On initial load the board's lists can take ~1-2 s to render after
+  `wait_for_load()` returns. A brief `time.sleep(2)` before scraping is
+  reliable; alternatively, wait for `[data-testid="list"]` to be
+  present with a count > 0.
+
+## Waits
+
+- `wait_for_load()` on the board URL returns before all lists/cards are
+  in the DOM. The list containers appear first (empty), then cards
+  populate. Wait for a non-zero card count inside a list before
+  declaring "done".
+
+## Traps
+
+- A board user-link scrape (`a[href^="/b/"]`) on the dashboard returns
+  each board twice (recent + starred/all). Deduplicate by `href`.
+- Sidebar h2 headings include plan names ("Standard", "Premium") —
+  don't treat them as list names. Scope header reads to elements
+  inside `[data-testid="list"]`.
+
+## Read-only, one-hit scrape (JS-as-extract, fast path)
+
+```js
+(() => Array.from(document.querySelectorAll('[data-testid="list"]'))
+  .map(list => ({
+    name: list.querySelector('[data-testid="list-header-name"]')?.innerText?.trim(),
+    cards: Array.from(list.querySelectorAll('a[href*="/c/"]'))
+      .map(a => ({
+        href: a.getAttribute('href'),
+        title: (a.querySelector('[data-testid="card-name"]')
+                ?? a).innerText.trim()
+      }))
+      .filter(c => c.title)
+  })))()
+```
+
+Wrap in `js(...)` + `json.loads()` and you have structured board data
+in one round trip.
+
+## What NOT to capture here
+
+Private API endpoints (Trello exposes `https://trello.com/1/...` REST
+endpoints, but those need a token). Keep this file DOM-only. If you
+discover public API shapes worth documenting, put them in a
+separate `api.md` under this skill folder.