arise-browser 0.2.3 → 0.3.0

package/dist/src/server/routes/health.js

@@ -4,7 +4,7 @@ export function registerHealthRoute(app) {
         return {
             status: "ok",
             connected: session.isConnected,
-            version: "0.2.3",
+            version: "0.3.0",
         };
     });
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "arise-browser",
-  "version": "0.2.3",
+  "version": "0.3.0",
   "description": "AI browser automation engine — persistent refs, multi-strategy actions, behavior recording",
   "type": "module",
   "main": "./dist/src/index.js",

package/skill/arise-browser/SKILL.md

@@ -13,15 +13,46 @@ metadata:
 
 # AriseBrowser
 
-Control a real Chrome browser via HTTP API. Persistent element refs, YAML accessibility snapshots, WebRTC live view.
+You are controlling a **real Chrome browser**, like a human sitting in front of a screen. You see the page through snapshots, and you interact by clicking, typing, and selecting — not by writing JavaScript or constructing URLs.
 
 ## MANDATORY RULES
 
 **You MUST follow these rules. No exceptions.**
 
-1. **Do NOT call any API endpoint until `/health` returns `{"connected":true}`.** The server needs time to start the Docker container and Chrome. Poll `/health` in a loop.
-2. **Every browser task follows: Navigate → Snapshot → Act → Snapshot → Act → Done.** Always snapshot before acting — you need refs from the snapshot to target elements.
-3. **Refs are persistent.** Do NOT re-snapshot just to reuse a ref. Only snapshot when the page changes significantly.
+1. **Wait for ready.** Do NOT call any endpoint until `/health` returns `{"connected":true}`.
+2. **Snapshot is your eyes.** After every navigate or significant action, call `/snapshot` to see what's on the page. Read the snapshot to find element refs (e0, e5, e12...) and understand the page structure.
+3. **Act through refs.** To click a button, select a dropdown, or type in a field — use `/action` with the ref from your snapshot. Do NOT construct URLs with query parameters to change page state. Use `select`, `click`, and `type` actions instead.
+4. **NEVER use `/evaluate` to extract data.** The snapshot already contains all visible text, links, buttons, and form elements in a structured format. `/evaluate` is only for rare edge cases where data is hidden from the accessibility tree.
+5. **NEVER use `/text` as your primary data source.** `/text` returns unstructured plain text that is hard to parse. Use `/snapshot` — it gives you structured elements with refs, roles, names, and links.
+6. **Refs are persistent.** Do NOT re-snapshot just to reuse a ref. Only re-snapshot when the page content changes.
+
+## How to Think
+
+You are a person using a browser. Snapshot is your eyes, action is your hands.
+
+- **To sort results** → find the sort dropdown in the snapshot → use `select` action on its ref
+- **To search** → find the search box ref → `type` your query → `press_key` Enter
+- **To go to next page** → find the "Next" button ref → `click` it
+- **To read product info** → it's already in the snapshot (names, prices, ratings are all there as text)
+
+### Example: Sort Amazon results by Best Sellers
+
+```bash
+# 1. Navigate
+curl -X POST /navigate -d '{"url": "https://amazon.com/s?k=laptop"}'
+
+# 2. Snapshot — see the page
+curl /snapshot
+# → combobox "Sort by:" [ref=e187] with options including "Best Sellers"
+# → link "Product Name" [ref=e226], generic "4.4" [ref=e231], link "$599" [ref=e246]
+
+# 3. Select from dropdown using ref
+curl -X POST /action -d '{"type": "select", "ref": "e187", "value": "exact-aware-popularity-rank"}'
+
+# 4. Snapshot again — results are now sorted
+curl /snapshot
+# → Read product names, prices, ratings directly from snapshot text
+```
 
 ## Step 1: Start the Server
 
@@ -51,20 +82,14 @@ After installation succeeds, inform the user:
    - `6090/tcp` — Neko WebRTC UI (browser live view)
    - `52000-52100/udp` — WebRTC media data
 2. **Watch the browser**: Open `http://<server-ip>:6090` in your browser, password: `neko`
-3. **HTTPS (optional)**: For production, put a reverse proxy in front. Recommended: [Caddy](https://caddyserver.com/) with `tls internal` (self-signed, no domain needed) or your own domain for auto Let's Encrypt.
-4. **Passwords**: Default Neko passwords are `neko` (viewer) and `admin` (admin). Change via `--neko-password` and `--neko-admin-password` flags.
+3. **HTTPS (optional)**: For production, put Caddy in front with `tls internal` (self-signed, no domain needed).
+4. **Passwords**: Default Neko passwords are `neko` (viewer) and `admin` (admin). Change via CLI flags.
 
-## Step 2: Use the Browser
+## Step 2: Core Loop
 
 Base URL: `http://localhost:9867`
 
-Every task follows this loop:
-
-```
-Navigate → Snapshot → Act → Snapshot → Act → ... → Done
-```
-
-### Navigate
+### Navigate to a URL
 
 ```bash
 curl -X POST http://localhost:9867/navigate \
@@ -72,61 +97,55 @@ curl -X POST http://localhost:9867/navigate \
   -d '{"url": "https://example.com"}'
 ```
 
-### Snapshot (get page state)
+### Snapshot — see the page
 
-Returns a YAML accessibility tree with element refs (e0, e5, e12...).
+Returns a YAML accessibility tree. Every interactive element has a ref you can act on.
 
 ```bash
-# Full snapshot
 curl http://localhost:9867/snapshot
+```
 
-# Diff mode — only changes since last snapshot (saves tokens)
-curl "http://localhost:9867/snapshot?diff=true"
+What you'll see in a snapshot:
+```yaml
+- combobox "Sort by:" [ref=e187]        ← dropdown, use select action
+- link "Product Name" [ref=e226]         ← clickable link
+- textbox "Search" [ref=e14]             ← input field, use type action
+- button "Add to cart" [ref=e281]        ← button, use click action
+- generic "4.4" [ref=e231]              ← text content (rating)
+- generic "$599.99" [ref=e246]          ← text content (price)
 ```
 
-### Act on elements
+Use `?diff=true` after the first snapshot to only see changes (saves tokens).
 
-Use refs from the snapshot. Refs are **persistent** — they survive across snapshots, no need to re-snapshot before reusing a ref.
+### Act — interact with elements
+
+Use the ref from your snapshot:
 
 ```bash
-# Click
+# Click a link or button
 curl -X POST http://localhost:9867/action -H "Content-Type: application/json" \
-  -d '{"type": "click", "ref": "e5"}'
+  -d '{"type": "click", "ref": "e226"}'
 
-# Type text
+# Type in a text field
 curl -X POST http://localhost:9867/action -H "Content-Type: application/json" \
-  -d '{"type": "type", "ref": "e12", "text": "search query"}'
+  -d '{"type": "type", "ref": "e14", "text": "search query"}'
 
-# Press key
+# Press a key (Enter, Tab, Escape, etc.)
 curl -X POST http://localhost:9867/action -H "Content-Type: application/json" \
   -d '{"type": "press_key", "key": "Enter"}'
 
-# Scroll
+# Select from a dropdown
 curl -X POST http://localhost:9867/action -H "Content-Type: application/json" \
-  -d '{"type": "scroll", "direction": "down", "amount": 500}'
+  -d '{"type": "select", "ref": "e187", "value": "option-value"}'
 
-# Hover
+# Scroll down
 curl -X POST http://localhost:9867/action -H "Content-Type: application/json" \
-  -d '{"type": "hover", "ref": "e7"}'
-
-# Select dropdown
-curl -X POST http://localhost:9867/action -H "Content-Type: application/json" \
-  -d '{"type": "select", "ref": "e3", "value": "option1"}'
+  -d '{"type": "scroll", "direction": "down", "amount": 500}'
 ```
 
-### Extract content
+### Repeat
 
-```bash
-# Page text
-curl http://localhost:9867/text
-
-# Screenshot (JPEG)
-curl http://localhost:9867/screenshot > screenshot.jpg
-
-# Execute JavaScript
-curl -X POST http://localhost:9867/evaluate -H "Content-Type: application/json" \
-  -d '{"expression": "document.title"}'
-```
+After each action that changes the page, snapshot again to see the result. Then act on the new refs.
 
 ## Step 3: Stop
 
@@ -138,11 +157,11 @@ The Docker container is automatically stopped and cleaned up.
 
 ## Tips
 
+- **Read the snapshot carefully.** Product names, prices, ratings, links — they're all there. No need for JavaScript or regex.
 - Use `?diff=true` after the first snapshot to save tokens.
-- Refs persist across snapshots — don't re-snapshot just to reuse a ref.
 - Batch actions: `POST /actions` with `{"actions": [...], "stopOnError": true}`.
 - Tabs: `GET /tabs`, `POST /tab` with `{"action": "create|switch|close"}`.
-- Use `tabId` param on any endpoint to target a specific tab without switching.
+- Screenshot (`GET /screenshot`) is useful to show the user what you see, but do NOT use it as your primary data source.
 
 ## Troubleshooting
 
@@ -151,9 +170,5 @@ The Docker container is automatically stopped and cleaned up.
 | First run slow | Docker pulling Neko image (~700MB), wait ~2 min |
 | Health returns `connected: false` | Chrome crashed — restart arise-browser |
 | Neko UI loads but no video | Open UDP 52000-52100 in firewall/security group |
-| Neko UI click no response | Use admin password `admin`, or restart container (implicit hosting enabled) |
-| Action returns error | Snapshot first to get valid refs, then act |
-
-## Full API Reference
-
-See [references/api.md](references/api.md) for all endpoints, parameters, and advanced features (recording, PDF export, batch actions).
+| Action returns error | Snapshot first to get valid refs, then act on them |
+| Can't find an element | Scroll down and snapshot again — element may be below the fold |