catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/rest-countries/scraping.md

@@ -0,0 +1,233 @@
+# REST Countries — Scraping & Data Extraction
+
+`https://restcountries.com` — open JSON API for country data. **Never use the browser.** All data is reachable via `http_get`. No auth required, no API key.
+
+## Do this first
+
+**Fetch all 250 countries in one call with a field filter — almost always the fastest approach.**
+
+```python
+import json
+from helpers import http_get
+
+data = http_get("https://restcountries.com/v3.1/all?fields=name,cca2,capital,population,area,region")
+countries = json.loads(data)
+# countries is a list of 250 dicts — confirmed 2026-04-18
+
+for c in countries:
+    name       = c["name"]["common"]          # "Germany"
+    official   = c["name"]["official"]        # "Federal Republic of Germany"
+    code       = c["cca2"]                    # "DE"
+    capital    = c["capital"][0] if c.get("capital") else None   # list — may be empty
+    population = c["population"]              # 83491249
+    area       = c["area"]                    # 357114.0 (km²)
+    region     = c["region"]                  # "Europe"
+    print(code, name, population)
+# Confirmed output (first result varies — API returns unsorted):
+# CI Ivory Coast 31719275
+```
+
+Use the `?fields=` query param to limit response size — essential when fetching all 250.
+
+## Common workflows
+
+### Lookup a single country by code (ISO 3166-1 alpha-2 or alpha-3)
+
+```python
+import json
+from helpers import http_get
+
+# Single code — returns a list (one element)
+data = http_get("https://restcountries.com/v3.1/alpha/DE")
+country = json.loads(data)[0]
+
+# But: /alpha/CODE?fields=... returns a plain dict, not a list — watch for this
+data2 = http_get("https://restcountries.com/v3.1/alpha/DE?fields=name,cca2,currencies,languages,flags")
+country2 = json.loads(data2)          # dict, NOT list
+
+name       = country2["name"]["common"]                             # "Germany"
+currencies = country2["currencies"]                                 # {"EUR": {"name": "euro", "symbol": "€"}}
+currency_codes = list(currencies.keys())                            # ["EUR"]
+currency_name  = currencies["EUR"]["name"]                          # "euro"
+languages  = country2["languages"]                                  # {"deu": "German"}
+lang_names = list(languages.values())                               # ["German"]
+flag_png   = country2["flags"]["png"]                               # "https://flagcdn.com/w320/de.png"
+flag_svg   = country2["flags"]["svg"]                               # "https://flagcdn.com/de.svg"
+flag_alt   = country2["flags"]["alt"]                               # description text
+
+print(name, currency_codes, lang_names)
+# Confirmed: Germany ['EUR'] ['German']
+```
+
+### Batch lookup — multiple codes in one call
+
+Use `/alpha?codes=` for fetching a known list of countries — always returns a list.
+
+```python
+import json
+from helpers import http_get
+
+codes = ["US", "GB", "FR", "DE", "JP", "CN", "IN", "BR", "AU", "CA"]
+data = http_get(f"https://restcountries.com/v3.1/alpha?codes={','.join(codes)}&fields=name,cca2,population")
+countries = json.loads(data)
+# Returns list, order NOT guaranteed to match input order
+for c in countries:
+    print(c["cca2"], c["name"]["common"], c["population"])
+# Confirmed: 10 results, returned in arbitrary order
+```
+
+### Search by name
+
+```python
+import json
+from helpers import http_get
+
+# Partial match (default) — may return multiple results
+data = http_get("https://restcountries.com/v3.1/name/united")
+results = json.loads(data)
+# Returns 7 countries: United States, UK, UAE, Tanzania, Mexico, ...
+
+# Exact match — use fullText=true with the full common or official name
+data2 = http_get("https://restcountries.com/v3.1/name/united%20kingdom?fullText=true")
+results2 = json.loads(data2)
+# Returns exactly 1 result: United Kingdom
+print(results2[0]["name"]["common"])  # United Kingdom
+```
+
+### Filter by region
+
+```python
+import json
+from helpers import http_get
+
+data = http_get("https://restcountries.com/v3.1/region/europe?fields=name,cca2,population")
+countries = json.loads(data)
+# 53 European countries — confirmed
+
+# Sort by population
+ranked = sorted(countries, key=lambda x: x["population"], reverse=True)
+for c in ranked[:5]:
+    print(c["cca2"], c["name"]["common"], f"{c['population']:,}")
+# Confirmed top 5: RU Russia, DE Germany, FR France, GB United Kingdom, IT Italy
+```
+
+Valid region values: `africa`, `americas`, `asia`, `europe`, `oceania`, `antarctic`
+
+### Filter by subregion
+
+```python
+import json
+from helpers import http_get
+
+data = http_get("https://restcountries.com/v3.1/subregion/Western%20Europe?fields=name,cca2")
+countries = json.loads(data)
+print([c["cca2"] for c in countries])
+# Confirmed: ['FR', 'NL', 'MC', 'DE', 'BE', 'LI', 'CH', 'LU']
+```
+
+### Filter by language
+
+```python
+import json
+from helpers import http_get
+
+data = http_get("https://restcountries.com/v3.1/lang/arabic")
+countries = json.loads(data)
+print(f"Arabic-speaking countries: {len(countries)}")
+# Confirmed: 25 countries
+
+# Language param is the language name (English), not the ISO 639-3 code
+# Works: arabic, french, spanish, english, portuguese, german, russian, chinese
+```
+
+### Filter by currency
+
+```python
+import json
+from helpers import http_get
+
+data = http_get("https://restcountries.com/v3.1/currency/EUR")
+countries = json.loads(data)
+print(f"EUR countries: {len(countries)}")  # Confirmed: 36
+names = [c["name"]["common"] for c in countries]
+print(names[:5])
+
+# Use ISO 4217 currency code (uppercase)
+```
+
+### Filter by capital city
+
+```python
+import json
+from helpers import http_get
+
+data = http_get("https://restcountries.com/v3.1/capital/berlin?fields=name,cca2,capital")
+result = json.loads(data)
+print(result[0]["name"]["common"], result[0]["capital"])
+# Confirmed: Germany ['Berlin']
+# Capital param is case-insensitive
+```
+
+### Full country detail — all fields
+
+```python
+import json
+from helpers import http_get
+
+data = http_get("https://restcountries.com/v3.1/alpha/US")
+c = json.loads(data)[0]
+
+# Available top-level keys (confirmed for US/DE):
+# name, tld, cca2, ccn3, cca3, cioc, independent, status, unMember,
+# currencies, idd, capital, altSpellings, region, subregion, languages,
+# translations, latlng, landlocked, borders, area, demonyms, flag (emoji),
+# maps, population, gini, fifa, car, timezones, continents, flags,
+# coatOfArms, startOfWeek, capitalInfo, postalCode
+
+print(c["idd"])          # {"root": "+1", "suffixes": ["201", "202", ...]}
+print(c["car"]["side"])  # "right" or "left"
+print(c["gini"])         # {"2018": 41.4}  — year keyed, may be absent
+print(c["timezones"])    # list of UTC offset strings
+print(c["borders"])      # list of cca3 codes for bordering countries
+print(c["latlng"])       # [lat, lng] of geographic center
+```
+
+## URL reference
+
+| Endpoint | Pattern | Notes |
+|---|---|---|
+| All countries | `/v3.1/all` | Always add `?fields=` |
+| By code | `/v3.1/alpha/{code}` | cca2 or cca3; single code → list (no fields) or dict (with fields) |
+| By codes | `/v3.1/alpha?codes=DE,FR,JP` | Always returns list |
+| By name | `/v3.1/name/{name}` | Partial; add `?fullText=true` for exact match |
+| By region | `/v3.1/region/{region}` | africa, americas, asia, europe, oceania, antarctic |
+| By subregion | `/v3.1/subregion/{subregion}` | URL-encode spaces as `%20` |
+| By language | `/v3.1/lang/{language}` | English language name |
+| By currency | `/v3.1/currency/{code}` | ISO 4217 (EUR, USD, GBP) |
+| By capital | `/v3.1/capital/{city}` | Case-insensitive |
+
+All endpoints accept `?fields=field1,field2,...` to limit response payload.
+
+## Gotchas
+
+- **`name` is a nested object, not a string.** Use `c["name"]["common"]` for the familiar English name, `c["name"]["official"]` for the full official name. `nativeName` is a dict keyed by ISO 639-3 language code.
+
+- **`/alpha/CODE` return type depends on whether `?fields=` is present.** Without `?fields=`, returns a list (one element). With `?fields=...`, returns a plain dict. Use `json.loads(data)[0]` for the no-fields case, `json.loads(data)` for the fields case. Using `/alpha?codes=CODE` always returns a list regardless.
+
+- **`currencies` is a dict keyed by ISO 4217 code, not a list.** `c["currencies"]["EUR"]["name"]` → `"euro"`, `c["currencies"]["EUR"]["symbol"]` → `"€"`. A country can have multiple currencies — iterate `currencies.items()`.
+
+- **`languages` is a dict keyed by ISO 639-3 code.** `c["languages"]["deu"]` → `"German"`. Use `list(c["languages"].values())` for a simple list of language names.
+
+- **`capital` is a list and may be empty.** Some territories (Antarctica, Bouvet Island, Macau, Heard Island) have no capital — `c.get("capital")` returns `[]`, not `None`. Guard with `c["capital"][0] if c.get("capital") else None`. South Africa has 3 capitals.
+
+- **`gini` is a dict keyed by year string, may be absent entirely.** `c["gini"]` → `{"2016": 31.9}`. Many small countries or territories have no gini data — always check `c.get("gini")`.
+
+- **`borders` uses cca3 codes, not cca2.** `c["borders"]` → `["AUT", "BEL", ...]`. Cross-reference with another `/alpha?codes=` call to resolve to names.
+
+- **`translations` covers ~45 languages.** Each entry: `c["translations"]["deu"]` → `{"official": "Bundesrepublik Deutschland", "common": "Deutschland"}`. Useful for multilingual apps.
+
+- **No rate limit headers, no documented rate limit.** In practice the API handles rapid sequential calls fine. For bulk crawling hundreds of per-country requests, add a short sleep (`time.sleep(0.5)`) between calls to be polite.
+
+- **404 returns JSON, not HTML.** `{"message": "Not Found", "status": 404}`. Wrap calls in try/except and check for this pattern when handling user-supplied country names or codes.
+
+- **`?fields=` is the key performance lever.** The full all-countries payload without field filtering is ~1.5 MB. With `?fields=name,cca2,population` it drops to ~50 KB. Always filter when you don't need all fields.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/sec-edgar/scraping.md

@@ -0,0 +1,361 @@
+# SEC EDGAR — Scraping & Data Extraction
+
+`https://www.sec.gov` / `https://data.sec.gov` / `https://efts.sec.gov` — all public data, no auth required. Every workflow here is pure `http_get` — no browser needed.
+
+## Do this first
+
+**SEC.gov requires a custom User-Agent on `www.sec.gov` and `data.sec.gov`. Always pass `headers=UA` or you get 403.**
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+# Format required: "CompanyName contact@email.com"
+# "Mozilla/5.0" (http_get default) works on efts.sec.gov and data.sec.gov
+# but FAILS on www.sec.gov (company_tickers.json, Archives/, etc.)
+```
+
+Start with `company_tickers.json` to resolve any ticker → CIK in one call, then branch to whichever endpoint you need.
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+tickers = json.loads(http_get("https://www.sec.gov/files/company_tickers.json", headers=UA))
+# 10,391 public companies, ~50KB, always fresh
+# Entry format: {"cik_str": 320193, "ticker": "AAPL", "title": "Apple Inc."}
+
+# Look up by ticker (exact, case-sensitive in the data)
+aapl = next(v for v in tickers.values() if v['ticker'] == 'AAPL')
+# {'cik_str': 320193, 'ticker': 'AAPL', 'title': 'Apple Inc.'}
+
+# CIK is an int here; pad to 10 digits for API URLs
+cik = str(aapl['cik_str']).zfill(10)  # "0000320193"
+```
+
+## Common workflows
+
+### Ticker / name → CIK lookup
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+tickers = json.loads(http_get("https://www.sec.gov/files/company_tickers.json", headers=UA))
+
+# By ticker
+tsla = next((v for v in tickers.values() if v['ticker'] == 'TSLA'), None)
+# {'cik_str': 1318605, 'ticker': 'TSLA', 'title': 'Tesla, Inc.'}
+
+# By partial name match
+apples = [v for v in tickers.values() if 'APPLE' in v['title'].upper()]
+# [{'cik_str': 320193, 'ticker': 'AAPL', 'title': 'Apple Inc.'}, ...]
+```
+
+### Company submissions (metadata + recent filings list)
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+cik = "0000320193"  # Apple - always zero-pad to 10 digits
+data = json.loads(http_get(f"https://data.sec.gov/submissions/CIK{cik}.json", headers=UA))
+
+print(data['name'])             # "Apple Inc."
+print(data['cik'])              # "0000320193"
+print(data['sic'])              # "3571"
+print(data['sicDescription'])   # "Electronic Computers"
+print(data['tickers'])          # ["AAPL"]
+print(data['exchanges'])        # ["Nasdaq"]
+
+# Most recent ~1,000 filings are in data['filings']['recent']
+recent = data['filings']['recent']
+# Fields per filing (parallel arrays, same index):
+# accessionNumber, filingDate, reportDate, form, primaryDocument,
+# primaryDocDescription, size, isXBRL, items, fileNumber
+
+# Filter for 10-K and 10-Q only
+filings_10k = [
+    (f, d, a, doc)
+    for f, d, a, doc in zip(
+        recent['form'], recent['filingDate'],
+        recent['accessionNumber'], recent['primaryDocument']
+    )
+    if f in ('10-K', '10-Q')
+]
+# Result: [('10-Q', '2026-01-30', '0000320193-26-000006', 'aapl-20251227.htm'), ...]
+```
+
+### Build direct filing document URL
+
+```python
+# Given accessionNumber and primaryDocument from submissions JSON:
+accn = "0000320193-25-000079"
+doc  = "aapl-20250927.htm"
+cik  = "320193"  # int part only (no leading zeros) for Archives path
+
+accn_nodash = accn.replace("-", "")
+url = f"https://www.sec.gov/Archives/edgar/data/{cik}/{accn_nodash}/{doc}"
+# https://www.sec.gov/Archives/edgar/data/320193/000032019325000079/aapl-20250927.htm
+
+# Full 10-K is 1.5MB of XBRL-tagged HTML — use http_get for text extraction
+content = http_get(url, headers=UA)  # UA required on www.sec.gov
+```
+
+### XBRL financial data — single company, one concept over time
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+cik_padded = "0000320193"
+
+# companyconcept: one metric, all reported values (quarterly + annual)
+data = json.loads(http_get(
+    f"https://data.sec.gov/api/xbrl/companyconcept/CIK{cik_padded}/us-gaap/Assets.json",
+    headers=UA
+))
+# data keys: cik, taxonomy, tag, label, description, entityName, units
+# data['units']['USD'] -> list of {end, val, accn, fy, fp, form, filed}
+
+entries = data['units']['USD']
+
+# Deduplicate: same period re-reported across multiple filings — keep latest
+def annual_series(entries):
+    seen = {}
+    for e in entries:
+        if e.get('form') == '10-K' and e.get('fp') == 'FY':
+            end = e['end']
+            if end not in seen or e['filed'] > seen[end]['filed']:
+                seen[end] = e
+    return [seen[k] for k in sorted(seen)]
+
+assets = annual_series(entries)
+for e in assets[-5:]:
+    print(f"{e['end']}  ${e['val']/1e9:.1f}B")
+# 2021-09-25  $351.0B
+# 2022-09-24  $352.8B
+# 2023-09-30  $352.6B
+# 2024-09-28  $365.0B
+# 2025-09-27  $359.2B
+```
+
+### XBRL financial data — all US-GAAP metrics for a company
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+
+# companyfacts: all reported XBRL concepts in one ~5MB call
+data = json.loads(http_get(
+    "https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json",
+    headers=UA
+))
+# data['entityName'] = "Apple Inc."
+# data['facts'] = {'us-gaap': {...503 concepts...}, 'dei': {...}}
+
+usgaap = data['facts']['us-gaap']
+print(len(usgaap))   # 503 concepts for Apple
+
+# Common concept names (companies vary — check what's available):
+# Revenue:     RevenueFromContractWithCustomerExcludingAssessedTax  (post-2018 standard)
+#              SalesRevenueNet                                       (older filings)
+#              Revenues                                              (some companies still use)
+# Net income:  NetIncomeLoss
+# Assets:      Assets
+# Cash:        CashAndCashEquivalentsAtCarryingValue
+# EPS:         EarningsPerShareBasic, EarningsPerShareDiluted
+
+# Find all revenue-related concepts this company reported:
+revenue_keys = [k for k in usgaap if 'Revenue' in k]
+
+# Extract annual revenue — handle company-specific concept name
+for concept in ['RevenueFromContractWithCustomerExcludingAssessedTax', 'SalesRevenueNet', 'Revenues']:
+    if concept in usgaap:
+        entries = usgaap[concept]['units'].get('USD', [])
+        annual = {}
+        for e in entries:
+            if e.get('form') == '10-K' and e.get('fp') == 'FY':
+                end = e['end']
+                if end not in annual or e['filed'] > annual[end]['filed']:
+                    annual[end] = e
+        if annual:
+            print(f"Using: {concept}")
+            for end in sorted(annual)[-3:]:
+                print(f"  {end}  ${annual[end]['val']/1e9:.1f}B")
+            break
+# Apple output:
+# Using: RevenueFromContractWithCustomerExcludingAssessedTax
+#   2023-09-30  $383.3B
+#   2024-09-28  $391.0B
+#   2025-09-27  $416.2B
+```
+
+### Cross-company financial comparison (XBRL frames)
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+
+# frames: one concept, one period, all companies that reported it
+# Period formats:
+#   CY2024           = calendar year 2024 (annual)
+#   CY2024Q4I        = Q4 2024 instantaneous (balance sheet items)
+#   CY2024Q4         = Q4 2024 duration (income statement items)
+
+# Top companies by annual revenue (2024)
+data = json.loads(http_get(
+    "https://data.sec.gov/api/xbrl/frames/us-gaap/RevenueFromContractWithCustomerExcludingAssessedTax/USD/CY2024.json",
+    headers=UA
+))
+companies = sorted(data['data'], key=lambda x: x['val'], reverse=True)
+# data['data'] entries: {accn, cik, entityName, loc, start, end, val}
+for c in companies[:5]:
+    print(f"{c['entityName']:<40}  ${c['val']/1e9:.0f}B")
+# Walmart Inc.                              $675B
+# AMAZON.COM, INC.                          $638B
+# Apple Inc.                                $391B
+# McKESSON CORPORATION                      $359B
+# Alphabet Inc.                             $350B
+
+# Total assets snapshot end of 2024 (balance sheet = instantaneous)
+data2 = json.loads(http_get(
+    "https://data.sec.gov/api/xbrl/frames/us-gaap/Assets/USD/CY2024Q4I.json",
+    headers=UA
+))
+# 6,229 companies for this frame
+```
+
+### Full-text search across all filings
+
+```python
+import json
+UA = {"User-Agent": "browser-harness research@example.com"}
+
+# Search for any phrase across filing documents
+# Params: q (quoted phrase), forms (comma-separated), dateRange=custom,
+#         startdt, enddt, size (max 100), from (offset for pagination)
+url = (
+    "https://efts.sec.gov/LATEST/search-index"
+    "?q=%22climate+risk%22"
+    "&forms=10-K"
+    "&dateRange=custom&startdt=2024-01-01"
+    "&size=10&from=0"
+)
+data = json.loads(http_get(url, headers=UA))
+# Note: default http_get UA (Mozilla/5.0) works fine on efts.sec.gov
+
+print(data['hits']['total']['value'])   # e.g. 1438 matching documents
+hits = data['hits']['hits']             # up to 100 per call
+
+for h in hits:
+    src = h['_source']
+    # Key fields: display_names, ciks, form, file_date, adsh (accession), period_ending
+    name = src['display_names'][0] if src.get('display_names') else '?'
+    cik  = src['ciks'][0] if src.get('ciks') else '?'
+    print(f"{name}  form={src['form']}  filed={src['file_date']}  accn={src['adsh']}")
+
+# Pagination: max 100 per page, use from= to walk through results
+# Page 2: from=100, Page 3: from=200, etc.
+for page in range(0, 300, 100):
+    page_url = url + f"&from={page}"
+    page_data = json.loads(http_get(page_url, headers=UA))
+    if not page_data['hits']['hits']:
+        break
+    # process...
+
+# Aggregations — group hits by entity, SIC, state
+aggs = data['aggregations']
+top_entities = aggs['entity_filter']['buckets']   # [{key: "Name (CIK...)", doc_count: N}, ...]
+top_sics     = aggs['sic_filter']['buckets']
+top_states   = aggs['biz_states_filter']['buckets']
+```
+
+### Find a company's CIK by name search (via search aggregations)
+
+```python
+import json, re
+UA = {"User-Agent": "browser-harness research@example.com"}
+
+# Best method: company_tickers.json (fastest, all tickers)
+tickers = json.loads(http_get("https://www.sec.gov/files/company_tickers.json", headers=UA))
+msft = next(v for v in tickers.values() if v['ticker'] == 'MSFT')
+# CIK = msft['cik_str']  → 789019
+
+# Alternative: full-text search aggregations (finds CIK from company name)
+data = json.loads(http_get(
+    "https://efts.sec.gov/LATEST/search-index?q=%22microsoft+corporation%22&forms=10-K",
+    headers=UA
+))
+buckets = data['aggregations']['entity_filter']['buckets']
+# [{'key': 'MICROSOFT CORP  (MSFT)  (CIK 0000789019)', 'doc_count': 11}, ...]
+for b in buckets[:3]:
+    m = re.search(r'\(CIK (\d+)\)', b['key'])
+    if m:
+        print(f"{b['key'][:50]}  →  CIK {m.group(1)}")
+```
+
+### Parallel fetching (multiple companies)
+
+```python
+import json
+from concurrent.futures import ThreadPoolExecutor
+
+UA = {"User-Agent": "browser-harness research@example.com"}
+
+def get_company_meta(ticker_cik):
+    ticker, cik = ticker_cik
+    subs = json.loads(http_get(f"https://data.sec.gov/submissions/CIK{cik}.json", headers=UA))
+    return {"ticker": ticker, "name": subs['name'], "sic": subs['sic']}
+
+companies = [("AAPL", "0000320193"), ("TSLA", "0001318605"), ("MSFT", "0000789019")]
+with ThreadPoolExecutor(max_workers=3) as ex:
+    results = list(ex.map(get_company_meta, companies))
+# Confirmed: 3 requests complete in ~0.28s
+# SEC rate limit: 10 req/sec — stay at max_workers ≤ 8 to be safe
+```
+
+## API reference
+
+| Endpoint | What it returns | UA required |
+|---|---|---|
+| `www.sec.gov/files/company_tickers.json` | All 10,391 tickers → CIK mapping | YES |
+| `data.sec.gov/submissions/CIK{10-digit}.json` | Company meta + ~1000 recent filings | YES |
+| `data.sec.gov/api/xbrl/companyfacts/CIK{10-digit}.json` | All XBRL facts (~5MB) | YES |
+| `data.sec.gov/api/xbrl/companyconcept/CIK{10-digit}/{taxonomy}/{concept}.json` | One concept, all values | YES |
+| `data.sec.gov/api/xbrl/frames/{taxonomy}/{concept}/{unit}/{period}.json` | All companies for one period | YES |
+| `efts.sec.gov/LATEST/search-index?q=...` | Full-text search across filings | NO (Mozilla/5.0 works) |
+| `www.sec.gov/Archives/edgar/data/{cik}/{accn-nodash}/{doc}` | Actual filing document | YES |
+
+`data.sec.gov` accepts `Mozilla/5.0` (the http_get default). `www.sec.gov` (Archives, company_tickers) requires the `"CompanyName email@example.com"` format.
+
+## Rate limits
+
+SEC documents a **10 requests/second** limit. In practice:
+- 12 rapid sequential calls to `data.sec.gov` completed in 2.4s (5 req/s) with no throttling.
+- 3 parallel calls completed in 0.28s without issue.
+- Stay at `max_workers ≤ 8` for ThreadPoolExecutor to respect the 10 req/s ceiling.
+- No per-day or per-hour cap documented; the 10/s limit is the only stated constraint.
+
+## Gotchas
+
+- **`www.sec.gov` returns 403 with `Mozilla/5.0` UA** — The http_get default (`"Mozilla/5.0"`) works on `data.sec.gov` and `efts.sec.gov` but is blocked on `www.sec.gov`. Always pass `headers=UA` where UA includes your company name and email. Confirmed: `"python-requests/2.28"` → 403.
+
+- **`data.sec.gov` is more permissive** — `Mozilla/5.0` works on `data.sec.gov` (submissions, xbrl). But always use the proper UA anyway — SEC's policy page explicitly requires it and they can add stricter checks at any time.
+
+- **XBRL contains duplicate entries per period** — The same fiscal year end date appears multiple times when a company restates or re-files. Each entry has a `filed` date and `accn` (accession). To get the canonical value, deduplicate by `end` keeping the entry with the latest `filed` date.
+
+- **Revenue concept name varies by company and era** — There is no single canonical "revenue" concept. Apple uses `RevenueFromContractWithCustomerExcludingAssessedTax`. Microsoft uses the same for recent years, but older filings used `SalesRevenueNet`. Always check which concepts are actually present: `[k for k in usgaap if 'Revenue' in k]`.
+
+- **`fp` field for annual filings is `'FY'`, but quarterly values also appear in 10-K** — A 10-K re-reports each quarter (fp=Q1, Q2, Q3) plus the full year (fp=FY). Filter on both `form == '10-K'` AND `fp == 'FY'` to get only annual totals.
+
+- **`companyfacts` is ~5MB per company** — For a single metric, use `companyconcept` instead (much smaller). Only use `companyfacts` when you need multiple concepts from the same company.
+
+- **`submissions` recent filings cap at ~1,000** — Very old filings don't appear. If you need historical data before that window, use the `filings.files` array in submissions JSON to find older filing JSON pages (`data.sec.gov/submissions/CIK{cik}-submissions-001.json`, etc.).
+
+- **`adsh` in search results is the accession number** — The search index returns `adsh` (no dashes). To build the document URL, insert dashes: `adsh[:10] + '-' + adsh[10:12] + '-' + adsh[12:]`, or use the `accessionNumber` field from submissions JSON (which already has dashes).
+
+- **`size` param is capped at 100** — Requesting `size=200` silently returns 100 hits. Walk results with `from=0`, `from=100`, etc. Maximum reachable index is 10,000 (Elasticsearch default).
+
+- **Search total `'gte'` relation means >10,000 hits** — When `total['relation'] == 'gte'`, there are more than 10,000 results (only first 10,000 accessible). Narrow with `dateRange` or `forms` filters.
+
+- **`company_tickers.json` covers only exchange-listed companies** — ~10,391 entries. Many SEC filers (private companies, bond issuers, FHLBs) have CIKs but no ticker. Find them via the full-text search aggregations or `submissions` lookup if you have the CIK.
+
+- **Filing document is XBRL-tagged HTML, 1–2MB** — Retrieving the actual 10-K HTML works but is large. For financial data extraction, always prefer the XBRL API endpoints over parsing the document.
+
+- **CIK format gotcha** — `company_tickers.json` returns `cik_str` as an int (`320193`). The submissions and xbrl APIs require a 10-digit zero-padded string in the filename (`CIK0000320193`). Always use `str(cik).zfill(10)` when building URLs.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/shopify-admin/README.md

@@ -0,0 +1,36 @@
+# shopify-admin
+
+Browser-harness patterns for `admin.shopify.com` and embedded Shopify apps.
+
+## Files in this folder
+
+- `embedded-apps.md` — every Shopify app runs in an iframe; how to target it
+- `polaris-inputs.md` — Polaris React inputs reject synthetic value setters; use CDP type_text
+- `knowledge-base.md` — automating the Shopify Knowledge Base App for FAQ entries
+
+## When to use these
+
+You're driving Shopify admin and need to add / edit / configure something. The Shopify admin UI is large and many surfaces are embedded apps — first check whether what you need is in an embedded app (most apps under `admin.shopify.com/store/<store>/apps/<app-slug>/...` are).
+
+## When to skip
+
+- If the operation is read-only product / inventory data → use the **Storefront API** (HTTP) instead, much faster
+- If the store has a custom admin app with API token provisioned → use the **Admin API** (GraphQL or REST) instead, no UI scraping
+- If you're editing theme code → use the **Shopify CLI** (`shopify theme push`) — don't touch the theme editor UI
+
+The browser is the right tool only when:
+- The setting / app exposes no API
+- The change is one-time or rare enough not to justify scripting
+- You're discovering / exploring the admin (e.g., finding selectors for a future automation)
+
+## Authentication
+
+Mike (or the human owner) must be logged into `admin.shopify.com` in the Chrome session that browser-harness attaches to. The harness does NOT log in — it inherits the human's session.
+
+If you hit `accounts.shopify.com` redirect, stop and ask the human to log in. Don't type credentials.
+
+## Polaris is in transition (Jan 2026 onward)
+
+Shopify is migrating its design system from React-based Polaris to Web-Components-based Polaris. Most legacy admin surfaces are still React. Newer surfaces (Catalog Mapping, parts of Settings) may be web components.
+
+Screenshot first. If you see `<s-text-field>` or `<s-button>` web component tags → use the web component pattern. If you see `[class*="Polaris-"]` React class names → use the CDP keystrokes pattern in `polaris-inputs.md`.