catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/world-bank/scraping.md

@@ -0,0 +1,356 @@
+# World Bank Open Data — Scraping & Data Extraction
+
+`https://api.worldbank.org/v2` — free REST API for global development indicators. No API key, no auth, no browser needed. All data via `http_get`.
+
+## Do this first
+
+**Every response is a 2-element JSON array: `[metadata, data]`.** The metadata element is always at index 0 (pagination info); the data array is at index 1. This is the single biggest gotcha — `json.loads(raw)` gives you a list, not a dict.
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get("https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD?format=json")
+d = json.loads(raw)
+meta = d[0]   # {"page": 1, "pages": 2, "per_page": 50, "total": 66, ...}
+rows = d[1]   # list of data records
+```
+
+Always append `?format=json` — default response is XML.
+
+## Common workflows
+
+### Single country, single indicator
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get("https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD?format=json")
+d = json.loads(raw)
+meta, rows = d[0], d[1]
+
+for r in rows:
+    if r["value"] is not None:   # recent years often have null values
+        print(r["date"], r["value"])
+# Confirmed output (2026-04-18):
+# 2024 28750956130731.2
+# 2023 27292170793214.4
+# 2022 25604848907611.0
+# ...
+```
+
+### Most recent N values (`mrv` param)
+
+`mrv` (most recent values) skips null years and returns the N most recent non-provisional points.
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get(
+    "https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD"
+    "?format=json&mrv=5"
+)
+d = json.loads(raw)
+for r in d[1]:
+    print(r["date"], r["value"])
+# Confirmed output (2026-04-18):
+# 2024 28750956130731.2
+# 2023 27292170793214.4
+# 2022 25604848907611.0
+# 2021 23315080560000.0
+# 2020 21060473613000.0
+```
+
+### Multiple countries, date range
+
+Semicolon-delimit country codes in the URL path. Use `date=YYYY:YYYY` for a range.
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get(
+    "https://api.worldbank.org/v2/country/US;CN;GB/indicator/SP.POP.TOTL"
+    "?format=json&date=2000:2023&per_page=100"
+)
+d = json.loads(raw)
+meta, rows = d[0], d[1]
+print(f"Total records: {meta['total']}, pages: {meta['pages']}")
+
+for r in rows:
+    print(r["country"]["value"], r["date"], r["value"])
+# Confirmed: returns 8 records per page (50 default), date range honored exactly
+# Countries: ['China', 'United States', 'United Kingdom']
+# Dates: ['2000', '2001', ..., '2023']
+```
+
+### All countries, latest value only
+
+Use `mrv=1` with `per_page=1000` to get all 266 countries in a single call.
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get(
+    "https://api.worldbank.org/v2/country/all/indicator/NY.GDP.PCAP.CD"
+    "?format=json&mrv=1&per_page=1000"
+)
+d = json.loads(raw)
+meta, rows = d[0], d[1]
+print(f"Countries returned: {len(rows)}")  # 266 (includes aggregates)
+
+# Filter out regional aggregates — they have no iso2Code or have aggregate ids
+countries_only = [r for r in rows if len(r["country"]["id"]) == 2]
+for r in sorted(countries_only, key=lambda x: -(x["value"] or 0))[:5]:
+    print(r["country"]["value"], r["date"], f"${r['value']:,.0f}")
+# Confirmed output (2026-04-18):
+# Luxembourg 2024 $135,605
+# Norway 2024 $105,056
+# ...
+```
+
+### Full pagination (fetch all pages)
+
+```python
+from helpers import http_get
+import json
+
+def fetch_all_pages(base_url):
+    """Fetch all pages of a World Bank API endpoint."""
+    all_rows = []
+    page = 1
+    while True:
+        url = f"{base_url}&page={page}" if "?" in base_url else f"{base_url}?page={page}"
+        d = json.loads(http_get(url))
+        meta, rows = d[0], d[1]
+        all_rows.extend(rows)
+        if page >= meta["pages"]:
+            break
+        page += 1
+    return all_rows
+
+# Example: all US GDP data (66 years, 2 pages)
+rows = fetch_all_pages(
+    "https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD"
+    "?format=json&per_page=50"
+)
+print(f"Total rows: {len(rows)}")  # 66
+non_null = [(r["date"], r["value"]) for r in rows if r["value"] is not None]
+print(f"Non-null: {len(non_null)}, range: {non_null[-1][0]}–{non_null[0][0]}")
+```
+
+### Indicators list (discover available indicators)
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get("https://api.worldbank.org/v2/indicator?format=json&per_page=50")
+d = json.loads(raw)
+meta = d[0]
+print(f"Total indicators: {meta['total']}, pages: {meta['pages']}")
+# Confirmed: 29,511 indicators across 591 pages
+
+for ind in d[1][:3]:
+    print(ind["id"], "-", ind["name"])
+# 1.0.HCount.1.90usd - Poverty Headcount ($1.90 a day)
+# ...
+```
+
+### Indicators by topic
+
+```python
+from helpers import http_get
+import json
+
+# Topic 3 = Economy & Growth
+raw = http_get("https://api.worldbank.org/v2/topic/3/indicator?format=json&per_page=50")
+d = json.loads(raw)
+print(f"Economy & Growth indicators: {d[0]['total']}")  # 306
+
+for ind in d[1][:5]:
+    print(ind["id"], "-", ind["name"])
+```
+
+### Country metadata
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get("https://api.worldbank.org/v2/country/US?format=json")
+d = json.loads(raw)
+c = d[1][0]
+print(c["name"], c["capitalCity"], c["region"]["value"], c["incomeLevel"]["value"])
+# United States  Washington D.C.  North America  High income
+
+# Filter countries by income level
+raw = http_get("https://api.worldbank.org/v2/country?format=json&incomeLevel=LIC&per_page=300")
+d = json.loads(raw)
+print(f"Low-income countries: {d[0]['total']}")  # 25
+```
+
+### Topics list
+
+```python
+from helpers import http_get
+import json
+
+raw = http_get("https://api.worldbank.org/v2/topics?format=json")
+d = json.loads(raw)
+for t in d[1]:
+    print(t["id"], t["value"])
+# 1  Agriculture & Rural Development
+# 2  Aid Effectiveness
+# 3  Economy & Growth
+# ... (21 topics total)
+```
+
+### Parallel fetch for multiple indicators (ThreadPoolExecutor)
+
+```python
+from helpers import http_get
+from concurrent.futures import ThreadPoolExecutor
+import json
+
+INDICATORS = {
+    "NY.GDP.MKTP.CD": "GDP (current US$)",
+    "SP.POP.TOTL": "Population",
+    "NY.GDP.PCAP.CD": "GDP per capita",
+}
+
+def fetch_indicator(ind_id):
+    url = (
+        f"https://api.worldbank.org/v2/country/US/indicator/{ind_id}"
+        f"?format=json&mrv=5"
+    )
+    d = json.loads(http_get(url))
+    return ind_id, d[1]
+
+with ThreadPoolExecutor(max_workers=3) as ex:
+    results = dict(ex.map(lambda i: fetch_indicator(i), INDICATORS))
+
+for ind_id, rows in results.items():
+    latest = next((r for r in rows if r["value"] is not None), None)
+    if latest:
+        print(f"{INDICATORS[ind_id]}: {latest['date']} = {latest['value']:,.2f}")
+```
+
+## URL reference
+
+### Base URL
+
+```
+https://api.worldbank.org/v2
+```
+
+HTTP redirects to HTTPS (302). Always use HTTPS directly.
+
+### Endpoint patterns
+
+| Endpoint | Description |
+|---|---|
+| `/country/{code}/indicator/{id}` | Single country + indicator time series |
+| `/country/{c1};{c2};{c3}/indicator/{id}` | Multi-country (semicolon-delimited) |
+| `/country/all/indicator/{id}` | All countries |
+| `/country/{code}` | Country metadata |
+| `/country` | All countries metadata (filterable) |
+| `/indicator` | All indicators list |
+| `/indicator/{id}` | Single indicator metadata |
+| `/topic/{id}/indicator` | Indicators for a topic |
+| `/topics` | All topics |
+
+### Query parameters
+
+| Parameter | Values | Notes |
+|---|---|---|
+| `format` | `json`, `xml` (default) | Always set `format=json` |
+| `per_page` | integer, default 50, max 1000 | Higher is faster for bulk |
+| `page` | integer, default 1 | For paginating results |
+| `date` | `2020`, `2000:2023` | Single year or colon-separated range |
+| `mrv` | integer | N most recent non-null values |
+| `gapfill` | `Y` | Forward-fill nulls when used with `mrv` |
+| `incomeLevel` | `LIC`, `MIC`, `HIC`, `LMC`, `UMC` | Filter countries by income |
+| `region` | `EAS`, `ECS`, `LAC`, `MEA`, `NAC`, `SAS`, `SSF` | Filter countries by region |
+
+### Common indicator IDs (confirmed working, 2026-04-18)
+
+| ID | Name |
+|---|---|
+| `NY.GDP.MKTP.CD` | GDP (current US$) |
+| `NY.GDP.PCAP.CD` | GDP per capita (current US$) |
+| `SP.POP.TOTL` | Population, total |
+| `SL.UEM.TOTL.ZS` | Unemployment (% of labor force) |
+| `FP.CPI.TOTL.ZG` | Inflation, consumer prices (%) |
+| `NE.EXP.GNFS.ZS` | Exports of goods and services (% of GDP) |
+| `SP.DYN.LE00.IN` | Life expectancy at birth |
+| `SE.ADT.LITR.ZS` | Literacy rate, adult total (%) |
+| `EG.USE.PCAP.KG.OE` | Energy use per capita (kg of oil equiv.) |
+
+Find more: `https://api.worldbank.org/v2/indicator?format=json&per_page=50&page=N`
+
+### Country codes (ISO2)
+
+Standard ISO 3166-1 alpha-2 codes: `US`, `CN`, `GB`, `DE`, `JP`, `IN`, `BR`, etc.
+Special: `all` for all countries.
+
+## Response structure
+
+Every endpoint returns a 2-element array:
+
+```json
+[
+  {
+    "page": 1,
+    "pages": 2,
+    "per_page": 50,
+    "total": 66,
+    "sourceid": "2",
+    "lastupdated": "2026-04-08"
+  },
+  [
+    {
+      "indicator": {"id": "NY.GDP.MKTP.CD", "value": "GDP (current US$)"},
+      "country": {"id": "US", "value": "United States"},
+      "countryiso3code": "USA",
+      "date": "2024",
+      "value": 28750956130731.2,
+      "unit": "",
+      "obs_status": "",
+      "decimal": 0
+    },
+    ...
+  ]
+]
+```
+
+Country metadata endpoint returns same 2-element shape but with country objects (not indicator rows) at index 1.
+
+## Gotchas
+
+- **Response is always a 2-element array, not a dict.** `d = json.loads(raw)` gives a list. `d[0]` is pagination metadata, `d[1]` is the data list. Accessing `d["page"]` raises `TypeError`. This is the most common mistake.
+
+- **`value` can be null.** Recent years (e.g. 2025) and data-sparse countries frequently have `null` values. Always check `if r["value"] is not None` before using. Use `mrv=N` to skip nulls automatically.
+
+- **Always append `?format=json`.** The default response format is XML. Without `format=json`, you get an XML string that fails `json.loads`.
+
+- **`all/indicator/{id}` includes regional aggregates.** The "all countries" endpoint returns 266 entries including aggregated regions like "Africa Eastern and Southern" (`id: "ZH"`). Filter to real countries with `len(r["country"]["id"]) == 2` (ISO2 codes are always 2 chars; aggregate codes are 2-3 chars but with non-standard values).
+
+- **Semicolons in URL path, not query string.** Multi-country requests use `country/US;CN;GB/indicator/...` not `?countries=US,CN,GB`. Commas do not work.
+
+- **HTTP 302 redirects HTTP to HTTPS.** Always use `https://` directly to avoid an extra round trip.
+
+- **`per_page` in metadata is sometimes a string, sometimes an integer.** The API returns `"per_page": "50"` (string) for some endpoints and `"per_page": 50` (int) for others. Don't compare with `==` without casting: `int(meta["per_page"])`.
+
+- **Invalid country codes return an error object, not a 2-element array.** A bad code gives `[{"message": [{"id": "120", "key": "Invalid value", ...}]}]` — a 1-element list with an error dict. Check `if isinstance(d[0], dict) and "message" in d[0]` before accessing `d[1]`.
+
+- **`mrv` + `gapfill=Y` forward-fills the latest value into future years.** If 2024 is the latest data point and `mrv=3`, `gapfill=Y` returns 2025 (the current year) with the 2024 value copied in. Useful for "current" lookups, but the filled date is misleading.
+
+- **No rate limit documented, but 3 req/s sustained is safe.** The API handles bursts (parallel ThreadPoolExecutor with `max_workers=3`) without issue. For crawling thousands of indicators, add `time.sleep(0.5)` between pages.
+
+- **`date` range returns records newest-first.** Results within a date range are sorted descending by year. If you need ascending order, sort after fetching: `sorted(rows, key=lambda r: r["date"])`.
+
+- **Indicator IDs are case-sensitive.** `ny.gdp.mktp.cd` returns an error; use the uppercase dot-separated form `NY.GDP.MKTP.CD`.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/xiaohongshu/scraping.md

@@ -0,0 +1,84 @@
+# Xiaohongshu — Search and Sort
+
+URL patterns:
+- Home / discovery: `https://www.xiaohongshu.com/explore`
+- Search results: `https://www.xiaohongshu.com/search_result?keyword=...`
+
+## Search flow
+
+- Prefer direct navigation to the desktop search results page over automating the home-page search box.
+- Reliable primary path: `https://www.xiaohongshu.com/search_result?keyword=<url-encoded keyword>&source=web_explore_feed`
+- This route loads the normal desktop results page and avoids home-page input flakiness.
+- The search results page can also appear with variants such as `type=51` or other `source` values after in-app navigation; do not treat those as suspicious if the rendered results are correct.
+- The top search box on `explore` can work, and searching from the home page has transitioned to `search_result` without a login wall in some sessions.
+- The page exposes duplicate search inputs in the DOM with the same placeholder `搜索小红书`.
+- The home-page search input can behave like a tightly controlled app field: direct DOM value assignment may be cleared immediately, and harness `type_text()` may fail to populate it even when the input is focused.
+- Treat the home-page input as best-effort only. Use it when a human-like interactive flow matters, but for automation default to constructing the `search_result` URL directly.
+
+## Sort behavior
+
+- On the current desktop results layout, `最新` is **not** a top-level tab beside `综合`.
+- Open the `筛选` control in the upper-right of the results header to access sort options.
+- Inside `筛选`, `排序依据` contains:
+  - `综合`
+  - `最新`
+  - `最多点赞`
+  - `最多评论`
+  - `最多收藏`
+- The `排序依据` row can render duplicate DOM nodes for the same pill text, including non-interactive clones.
+- Raw global text search for `最新` can hit the wrong node first. Scope to the `排序依据` section and then choose the visible interactive `.tags` node.
+- Prefer semantic filtering such as `aria-hidden != "true"` or section-scoped visible `.tags` selection over style-specific checks.
+- When `最新` is active, the `筛选` trigger changes to `已筛选`.
+- The rendered feed and the `已筛选` / active-pill UI are more reliable than `window.__INITIAL_STATE__.search.searchContext.sort` for confirming latest sort.
+
+## Stable cues
+
+- Search channel tabs near the top: `全部`, `图文`, `视频`, `用户`
+- Sort panel labels: `筛选`, `排序依据`, `最新`
+- Filter sections also visible in the panel: `笔记类型`, `发布时间`, `搜索范围`, `位置距离`
+
+## Interaction notes
+
+- DOM `.click()` opened the `筛选` panel reliably.
+- DOM `.click()` on the visible `最新` pill inside the open `排序依据` section reliably activated latest sort.
+- The reliable DOM pattern was:
+  - find the `排序依据` section / `.filters` block
+  - search within that block for `.tags`
+  - choose the one whose text is `最新` and which is the visible interactive node
+  - call `.click()` on that visible node
+- Example selector strategy:
+  - find `.filters` whose first label is `排序依据`
+  - inside it, pick `.tags` where `textContent.trim() === "最新"` and `el.getAttribute("aria-hidden") !== "true"`
+- `getClientRects().length > 0` alone may be insufficient to distinguish the working node from a duplicate.
+- A broad `document.querySelectorAll("*")` text match for `最新` is not reliable on this page because it may click the hidden duplicate instead of the visible control.
+- Coordinate click on the visible `最新` pill also worked and remains a valid fallback if DOM targeting gets confused by future UI changes.
+- After selecting `最新`, the grid briefly showed skeleton placeholders before the refreshed results appeared.
+- The search page stores the currently rendered note cards in `window.__INITIAL_STATE__.search.feeds._value` as an array of feed entries. For ordinary note cards, the useful fields were:
+  - `id`
+  - `xsecToken`
+  - `noteCard.displayTitle`
+  - `noteCard.user.nickname`
+- The feed array can contain non-note inserts such as hot-query modules. Filter for entries with `noteCard` before treating an item as a note result.
+
+## Post opening
+
+- Do **not** assume a raw results link like `https://www.xiaohongshu.com/explore/<id>` is directly openable.
+- Opening that raw `/explore/<id>` URL in a fresh tab can redirect to the web `404` / app-only gate even when the same post is openable from search results.
+- To open a post from search results, click the visible card image / card in-page first.
+- That click navigation can land on a tokenized URL like `https://www.xiaohongshu.com/explore/<id>?xsec_token=...&xsec_source=pc_search`, which is a more reliable note URL than the raw `/explore/<id>` form.
+- Once the tokenized URL is obtained from the click flow, it can be revisited in-session for extraction.
+- If the search results state is already loaded, you can reconstruct the tokenized note URL directly from a feed item without re-clicking:
+  - `https://www.xiaohongshu.com/explore/<id>?xsec_token=<xsecToken>&xsec_source=pc_search`
+
+## Post extraction
+
+- On tokenized post pages opened via `pc_search`, `document.body.innerText` can be a useful first-pass extraction source because it often includes the rendered note text, hashtags, timestamp, engagement counts, and visible comments.
+- Verify that the note content actually rendered before trusting `document.body.innerText`, because the page can also include substantial navigation, footer, and comment noise.
+- Prefer `document.body.innerText` as a fallback or initial probe before writing fragile per-element selectors for post content.
+
+## Gotchas
+
+- Do not assume `Enter` alone finished the workflow until you verify the URL changed to `search_result` or the result grid appeared.
+- Do not assume the visible `综合` tab controls all sorting; on this layout, time ordering is hidden inside `筛选`.
+- Do not assume the first DOM node whose text is `最新` is the clickable one; this panel duplicates pills and the hidden clone can absorb naive text-based targeting without changing state.
+- Do not assume a successfully opened post can be reproduced by stripping query params; preserve the `xsec_token` when reopening results-derived post URLs.