catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/eventbrite/scraping.md

@@ -0,0 +1,363 @@
+# Eventbrite — Scraping & Data Extraction
+
+`https://www.eventbrite.com` — public event listings and detail pages, no auth required for HTML scraping. REST API requires an OAuth token.
+
+## Do this first
+
+**Use the search listing URL to get event lists — parse the `ItemList` JSON-LD block, not the HTML.**
+
+```python
+import re, json
+
+headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"}
+html = http_get("https://www.eventbrite.com/d/ca--san-francisco/tech/", headers=headers)
+
+ld_blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
+for block in ld_blocks:
+    parsed = json.loads(block)
+    if isinstance(parsed, dict) and parsed.get('@type') == 'ItemList':
+        for item in parsed['itemListElement']:
+            ev = item['item']
+            print(ev['name'], ev['startDate'], ev['url'])
+        break
+# Returns 18–40 events per page
+```
+
+**For a single event, fetch the detail page and extract the `Event` JSON-LD block.** It contains all fields including `offers` (pricing). There is also a richer `__NEXT_DATA__` block if you need venue coordinates, refund policy, or sales status.
+
+## URL structure
+
+### Search / listing pages
+
+```
+https://www.eventbrite.com/d/{location}/{category}/
+https://www.eventbrite.com/d/{location}/{category}/?page=2
+https://www.eventbrite.com/d/{location}/{category}/?start_date=2026-05-01&end_date=2026-05-31
+```
+
+**Location format:** `{state-abbreviation}--{city}` (lowercase, hyphens for spaces)
+- `ca--san-francisco`
+- `ny--new-york`
+- `ca--los-angeles`
+- Use `online` for virtual events
+
+**Category slugs (confirmed working):**
+- `tech` — Technology events
+- `music` — Music
+- `food--drink` — Food & Drink
+- `health` — Health & Wellness
+- `sports--fitness` — Sports & Fitness
+- `arts--entertainment` — Arts & Entertainment
+- `family--education` — Family & Education
+- `business--professional` — Business & Networking
+- `science--tech` — Science & Technology
+- `community--culture` — Community & Culture
+- `networking` — Networking
+- `events` — All events (broadest, returns ~40/page)
+
+**Filter slugs (replace category):**
+- `free--events` — Free events only
+- `events--today` — Today
+- `events--tomorrow` — Tomorrow
+- `events--this-weekend` — This weekend
+
+**Query params:**
+- `?page=N` — Pagination (page 2+ confirmed working, each returns 18–20 events)
+- `?start_date=YYYY-MM-DD&end_date=YYYY-MM-DD` — Date range filter (confirmed, narrows results)
+
+### Event detail pages
+
+```
+https://www.eventbrite.com/e/{slug}-tickets-{event_id}
+```
+
+Example: `https://www.eventbrite.com/e/icontact-the-tactile-tech-opera-tickets-1982861003639`
+
+- `event_id` is a numeric string (10–13 digits)
+- Extract with: `re.search(r'-tickets-(\d+)$', url).group(1)`
+- Extract slug with: `re.search(r'/e/(.+)-tickets-\d+$', url).group(1)`
+
+Other TLDs (`.ca`, `.co.uk`, etc.) use the same structure — event IDs are globally unique across TLDs.
+
+## Listing page: JSON-LD `ItemList` schema
+
+The first `<script type="application/ld+json">` block on any `/d/` page is an `ItemList`. Each `itemListElement` contains:
+
+```json
+{
+  "position": 1,
+  "@type": "ListItem",
+  "item": {
+    "@type": "Event",
+    "name": "iContact the tactile tech opera",
+    "description": "An immersive performance...",
+    "url": "https://www.eventbrite.com/e/icontact-the-tactile-tech-opera-tickets-1982861003639",
+    "image": "https://img.evbuc.com/...",
+    "startDate": "2026-06-21",
+    "endDate": "2026-06-21",
+    "eventAttendanceMode": "https://schema.org/OfflineEventAttendanceMode",
+    "location": {
+      "@type": "Place",
+      "name": "Little Boxes Theater",
+      "address": {
+        "@type": "PostalAddress",
+        "addressLocality": "San Francisco",
+        "addressRegion": "CA",
+        "addressCountry": "US",
+        "streetAddress": "94107 1661 Tennessee Street",
+        "postalCode": "94107"
+      },
+      "geo": {
+        "@type": "GeoCoordinates",
+        "latitude": "37.7508806",
+        "longitude": "-122.3881427"
+      }
+    }
+  }
+}
+```
+
+Note: listing-page items do NOT include `offers` (pricing) or `organizer`. Fetch the detail page for those.
+
+The second JSON-LD block on listing pages is a `BreadcrumbList` (skip it).
+
+## Detail page: JSON-LD `Event` schema
+
+The detail page has 4 JSON-LD blocks. The `Event` (or `BusinessEvent`) block is the second one and contains the full schema:
+
+```python
+import re, json
+
+headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"}
+html = http_get("https://www.eventbrite.com/e/icontact-the-tactile-tech-opera-tickets-1982861003639", headers=headers)
+
+ld_blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
+event_data = None
+for block in ld_blocks:
+    parsed = json.loads(block)
+    if isinstance(parsed, dict) and parsed.get('@type') in ('Event', 'BusinessEvent', 'MusicEvent', 'EducationEvent'):
+        event_data = parsed
+        break
+
+print(event_data['name'])              # "iContact the tactile tech opera"
+print(event_data['startDate'])         # "2026-06-21T17:05:00-07:00"  (ISO 8601 with TZ)
+print(event_data['endDate'])           # "2026-06-21T20:08:00-07:00"
+print(event_data['eventStatus'])       # "https://schema.org/EventScheduled"
+print(event_data['eventAttendanceMode'])  # "https://schema.org/OfflineEventAttendanceMode"
+print(event_data['location']['name'])  # "Little Boxes Theater"
+print(event_data['location']['address']['streetAddress'])   # "94107 1661 Tennessee Street, San Francisco, CA 94107"
+print(event_data['organizer']['name'])  # "Beth McNamara"
+print(event_data['organizer']['url'])   # "https://www.eventbrite.com/o/beth-mcnamara-120755148166"
+```
+
+Full confirmed schema on detail page:
+```
+name               str     Event title
+description        str     Short summary
+url                str     Canonical event URL
+image              str     Event banner image URL
+startDate          str     ISO 8601 with timezone offset
+endDate            str     ISO 8601 with timezone offset
+eventStatus        str     URI: EventScheduled / EventCancelled / EventPostponed
+eventAttendanceMode str    URI: OfflineEventAttendanceMode / OnlineEventAttendanceMode / MixedEventAttendanceMode
+location.@type     str     "Place" (in-person) or "VirtualLocation" (online)
+location.name      str     Venue name
+location.address.streetAddress   str
+location.address.addressLocality str    City
+location.address.addressRegion   str    State abbreviation
+location.address.addressCountry  str    Country code
+organizer.name     str     Organizer display name
+organizer.url      str     Organizer profile URL
+offers             list    AggregateOffer object(s)
+```
+
+### Offers / pricing
+
+```python
+offers = event_data.get('offers', [])
+if offers:
+    offer = offers[0]   # always a list; typically one AggregateOffer
+    print(offer['@type'])           # "AggregateOffer"
+    print(offer['lowPrice'])        # "50.0"  (string, not float)
+    print(offer['highPrice'])       # "50.0"
+    print(offer['priceCurrency'])   # "USD"
+    print(offer['availability'])    # "InStock" / "SoldOut"
+    print(offer['availabilityStarts'])   # ISO 8601 UTC
+    print(offer['availabilityEnds'])     # ISO 8601 UTC
+
+# Free events: lowPrice="0.0", highPrice="0.0"
+# Free check: float(offer['lowPrice']) == 0.0
+```
+
+`@type` on the event itself varies by format (all scrape identically):
+- `Event` — general
+- `BusinessEvent` — networking, professional
+- `MusicEvent` — concerts
+- `EducationEvent` — classes, workshops
+
+## Detail page: `__NEXT_DATA__` (richer structured data)
+
+Every event detail page embeds a `<script id="__NEXT_DATA__">` block with additional fields not in JSON-LD:
+
+```python
+import re, json
+
+nextjs = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
+nd = json.loads(nextjs.group(1))
+context = nd['props']['pageProps']['context']
+
+bi = context['basicInfo']
+print(bi['id'])               # "1982861003639"  (event ID string)
+print(bi['name'])             # event title
+print(bi['isFree'])           # bool
+print(bi['isOnline'])         # bool
+print(bi['currency'])         # "USD"
+print(bi['status'])           # "live" / "completed" / "canceled"
+print(bi['organizationId'])   # numeric string
+print(bi['formatId'])         # numeric string (event format category)
+print(bi['isProtected'])      # bool — password-protected events
+print(bi['isSeries'])         # bool — recurring series
+print(bi['created'])          # ISO 8601 UTC creation timestamp
+
+# Venue with coordinates
+venue = bi['venue']
+print(venue['name'])                              # "Little Boxes Theater"
+print(venue['address']['city'])                   # "San Francisco"
+print(venue['address']['region'])                 # "CA"
+print(venue['address']['latitude'])               # "37.7508806"
+print(venue['address']['longitude'])              # "-122.3881427"
+print(venue['address']['localizedMultiLineAddressDisplay'])  # list of strings
+
+# Organizer details
+org = bi['organizer']
+print(org['name'])            # "Beth McNamara"
+print(org['url'])             # organizer profile URL
+print(org['numEvents'])       # int
+print(org['verified'])        # bool
+
+# Sales status
+ss = context['salesStatus']
+print(ss['salesStatus'])      # "on_sale" / "sold_out" / "sales_ended"
+print(ss['startSalesDate']['local'])   # local datetime string
+
+# Good to know
+gtk = context['goodToKnow']['highlights']
+print(gtk['ageRestriction'])          # "18+" or null
+print(gtk['durationInMinutes'])       # int (e.g. 183)
+print(gtk['doorTime'])                # local datetime string or null
+print(gtk['locationType'])            # "in_person" or "online"
+
+# Refund policy
+refund = context['goodToKnow']['refundPolicy']
+print(refund['policyType'])           # "custom" / "no_refunds" / "standard"
+print(refund['isRefundAllowed'])      # bool
+print(refund['validDays'])            # int or null
+
+# Full event description (HTML)
+for module in context['structuredContent']['modules']:
+    if module['type'] == 'text':
+        print(module['text'])         # raw HTML, may need BeautifulSoup to strip tags
+```
+
+## Complete workflow: scrape events from a category
+
+```python
+import re, json
+
+def get_events_from_listing(location, category, page=1):
+    """Returns list of event dicts with name, url, startDate, endDate, location."""
+    headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"}
+    url = f"https://www.eventbrite.com/d/{location}/{category}/?page={page}"
+    html = http_get(url, headers=headers)
+    ld_blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
+    for block in ld_blocks:
+        parsed = json.loads(block)
+        if isinstance(parsed, dict) and parsed.get('@type') == 'ItemList':
+            return [item['item'] for item in parsed.get('itemListElement', [])]
+    return []
+
+def get_event_detail(event_url):
+    """Returns full Event JSON-LD + NEXT_DATA context for a single event."""
+    headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"}
+    html = http_get(event_url, headers=headers)
+
+    # JSON-LD Event block
+    ld_blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
+    event_ld = None
+    for block in ld_blocks:
+        parsed = json.loads(block)
+        if isinstance(parsed, dict) and parsed.get('@type') in ('Event', 'BusinessEvent', 'MusicEvent', 'EducationEvent'):
+            event_ld = parsed
+            break
+
+    # NEXT_DATA context
+    nextjs = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
+    context = None
+    if nextjs:
+        nd = json.loads(nextjs.group(1))
+        context = nd['props']['pageProps']['context']
+
+    return event_ld, context
+
+# Usage
+events = get_events_from_listing("ca--san-francisco", "tech", page=1)
+print(f"Found {len(events)} events")  # 18–20 typical
+
+for ev in events[:3]:
+    print(ev['name'], ev['startDate'], ev['url'])
+
+# Deep-fetch one event
+ld, ctx = get_event_detail(events[0]['url'])
+if ld and ld.get('offers'):
+    price = float(ld['offers'][0]['lowPrice'])
+    currency = ld['offers'][0]['priceCurrency']
+    print(f"Price: {price} {currency}")   # 0.0 USD (free) or e.g. 50.0 USD
+```
+
+## Public API: requires auth
+
+The Eventbrite REST API (`https://www.eventbriteapi.com/v3/`) requires an OAuth token for all endpoints:
+
+- `GET /v3/events/{id}/` — HTTP 401 without auth
+- `GET /v3/events/search/` — HTTP 404 (endpoint changed; auth also required)
+
+**Use HTML scraping instead** — the JSON-LD and `__NEXT_DATA__` data is equivalent to the API response and requires no credentials.
+
+If you have a token (`EVENTBRITE_TOKEN`):
+```python
+import os
+token = os.environ.get('EVENTBRITE_TOKEN')
+headers = {
+    "User-Agent": "Mozilla/5.0",
+    "Authorization": f"Bearer {token}"
+}
+data = json.loads(http_get(f"https://www.eventbriteapi.com/v3/events/{event_id}/", headers=headers))
+```
+
+## Gotchas
+
+- **Event URLs in the HTML use relative `/e/` paths, not absolute URLs** — Search listing HTML contains `/e/slug-tickets-id?aff=...` relative paths (with tracking params). Extract event URLs from the JSON-LD `ItemList` instead — they are absolute, clean URLs without tracking params.
+
+- **`re.findall(r'href="https://www.eventbrite.com/e/...')` returns 0 results** — Confirmed: event cards in the HTML do not have `https://www.eventbrite.com/e/` in href attributes. Use JSON-LD extraction only.
+
+- **`__SERVER_DATA__` does not exist** — Both search and detail pages were checked. There is no `window.__SERVER_DATA__` or `window.__redux_state__`. The embedded data is in `<script id="__NEXT_DATA__">` (detail pages only) and JSON-LD (both).
+
+- **Search listing pages have no `__NEXT_DATA__`** — Only event detail pages (`/e/` URLs) have the `__NEXT_DATA__` block. Listing pages (`/d/` URLs) have JSON-LD only.
+
+- **`@type` varies by event format** — Don't filter JSON-LD blocks with `parsed['@type'] == 'Event'` alone. Check for any of: `Event`, `BusinessEvent`, `MusicEvent`, `EducationEvent`. They have identical field structure.
+
+- **`startDate` on listing vs. detail pages differs in precision** — Listing page items show date-only (`"2026-06-21"`). Detail page Event block shows full ISO 8601 with timezone offset (`"2026-06-21T17:05:00-07:00"`). Use detail page for scheduling tasks.
+
+- **`offers` is absent on listing page items** — The `ItemList` does not include pricing. Fetch the detail page for `offers.lowPrice` / `offers.highPrice`.
+
+- **Free events have `lowPrice: "0.0"` and `highPrice: "0.0"`** — Not null or missing. Check `float(offers[0]['lowPrice']) == 0.0` or use `basicInfo.isFree` from `__NEXT_DATA__`.
+
+- **`offers` prices are strings, not floats** — `"50.0"` not `50.0`. Cast with `float(offer['lowPrice'])` before arithmetic.
+
+- **Page size is ~18–20 events per page** — Not a fixed 20. Some pages return fewer. Don't assume page N is empty because it returned < 20.
+
+- **Date filter works but can still return events outside range** — The `?start_date=` / `?end_date=` params narrow results but are not strict; always validate `startDate` from the returned data.
+
+- **Eventbrite CA / UK / AU use different TLDs** — Online event listings may surface `eventbrite.ca`, `eventbrite.co.uk` URLs. The `/e/` structure and JSON-LD schema are identical. Fetch them with the same code.
+
+- **No rate limiting observed** — 8 sequential HTTP requests across 4 pages completed without errors or blocks (avg ~1.5s each). No delay needed for light workloads, but be reasonable for bulk scraping.

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/expedia/automation.md

@@ -0,0 +1,168 @@
+# Expedia — Browser Automation
+
+Field-tested against expedia.co.in on 2026-04-27 using `browser-harness` CDP
+helpers (`goto`, `js`, `click`, `type_text`, `screenshot`).
+
+---
+
+## TL;DR
+
+**Build your search via URL parameters, not the UI.** The date picker, destination
+autocomplete, and traveller widgets are fragile—coordinate clicks frequently
+dismiss them or mis-target. Encode everything you can into a `goto()` URL, then
+use JS clicks only for what the URL can't express (child ages, price filters).
+
+---
+
+## Hotel Search URL Template
+
+```
+https://www.expedia.co.in/Hotel-Search?destination={DEST}&startDate={YYYY-MM-DD}&endDate={YYYY-MM-DD}&rooms={N}&adults={N}&children={N}&childrenAges={age1,age2,...}
+```
+
+Example — 2 adults, 2 children (ages 5 and 7), Tokyo, June 2026:
+
+```python
+goto("https://www.expedia.co.in/Hotel-Search?"
+     "destination=Central+Tokyo,+Tokyo+Prefecture&"
+     "startDate=2026-06-01&endDate=2026-06-07&"
+     "rooms=1&adults=2&children=2&childrenAges=5,7")
+```
+
+**Note:** `childrenAges` in the URL may not always populate the age dropdowns on
+the results page. Verify with a screenshot and set them via JS if needed.
+
+---
+
+## Date Picker — DO NOT USE
+
+The calendar widget is extremely unreliable with coordinate-based clicks:
+
+- Clicking a date cell frequently **closes the entire picker** instead of
+  selecting the date.
+- The picker has month-navigation arrows that are tiny targets.
+- "Flexible dates" mode has a different DOM structure with pill-shaped month
+  selectors that also mis-fire.
+- Dozens of retry attempts across multiple strategies all failed.
+
+**Workaround:** Always pass dates via URL parameters (`startDate`, `endDate`).
+
+---
+
+## Travellers Widget
+
+The travellers stepper panel works with JS `.click()` on the increment/decrement
+buttons.
+
+### Opening the panel
+
+```python
+js("""
+(()=>{
+  let btn = document.querySelector('button[data-testid="travelers-field-trigger"]')
+           || [...document.querySelectorAll('button')].find(b => b.textContent.includes('traveller'));
+  if(btn){ btn.click(); return 'opened'; }
+  return 'not found';
+})()
+""")
+```
+
+### Incrementing children count
+
+```python
+js("""
+(()=>{
+  let span = [...document.querySelectorAll('span')].find(s => s.textContent.trim() === 'Children');
+  if(!span) return 'no Children label';
+  let container = span.closest('div').parentElement;
+  let buttons = container.querySelectorAll('button');
+  let plus = buttons[buttons.length - 1];  // last button is "+"
+  plus.click();
+  return 'incremented';
+})()
+""")
+```
+
+### Setting child ages
+
+Child age dropdowns are `<select>` elements with `aria-label` like
+"Child 1 age", "Child 2 age", etc.
+
+```python
+js("""
+(()=>{
+  let selects = document.querySelectorAll('select');
+  // selects[0] = Child 1 age, selects[1] = Child 2 age, etc.
+  let setter = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set;
+  selects[0].value = '5';
+  selects[0].dispatchEvent(new Event('change', {bubbles:true}));
+  selects[1].value = '7';
+  selects[1].dispatchEvent(new Event('change', {bubbles:true}));
+  return 'ages set';
+})()
+""")
+```
+
+### Closing the panel
+
+```python
+js("""
+(()=>{
+  let btn = [...document.querySelectorAll('button')].find(b => b.textContent.trim() === 'Done');
+  if(btn){ btn.click(); return 'done'; }
+  return 'no Done button';
+})()
+""")
+```
+
+---
+
+## Price Filter
+
+On the results page, the nightly-price filter has two text inputs and two range
+sliders.
+
+| Element | Selector |
+|---------|----------|
+| Min text input | `#price-min` |
+| Max text input | `#price-max` |
+| Min range slider | `input[type="range"][aria-label*="Minimum"]` |
+| Max range slider | `input[type="range"][aria-label*="Maximum"]` |
+
+### Setting max price
+
+The most reliable method is to click the input, select all, type the value, and
+press Enter:
+
+```python
+click(x, y)  # coordinates of #price-max
+js("document.getElementById('price-max').select()")
+type_text("20000")
+press_key("Enter")
+```
+
+Setting the value purely via JS (`dispatchEvent`) does trigger a re-search but
+coordinate-click + type is more reliable for actually applying the filter.
+
+---
+
+## Key Lessons
+
+1. **URL-first strategy** — Encode destination, dates, room count, adults,
+   children, and child ages in the URL. Only use UI interaction for things the
+   URL cannot express.
+
+2. **JS `.click()` over coordinate clicks** — For buttons inside panels
+   (traveller stepper, Done), find elements by text/attribute and call `.click()`
+   in JS. Coordinate clicks on overlay panels are unreliable.
+
+3. **`dispatchEvent` with `{bubbles: true}`** — Required for React-controlled
+   inputs (selects, text fields). Without bubbling, React state won't update.
+
+4. **Wait after navigation** — After `goto()` or pressing Search, call
+   `wait_for_load()` + `time.sleep(3)` before interacting. Expedia loads
+   results asynchronously.
+
+5. **Indian locale** — `expedia.co.in` shows prices in ₹ (INR). The price
+   filter values include the ₹ symbol and commas in the text input but the
+   underlying range slider uses plain integers.