opensteer 0.9.4 → 0.9.5

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "opensteer",
-  "version": "0.9.4",
+  "version": "0.9.5",
   "description": "Opensteer browser automation, replay, and reverse-engineering toolkit.",
   "license": "MIT",
   "type": "module",
@@ -57,7 +57,7 @@
     "skills": "^1.4.6",
     "ws": "^8.18.0",
     "@opensteer/engine-playwright": "0.8.9",
-    "@opensteer/runtime-core": "0.2.3"
+    "@opensteer/runtime-core": "0.2.4"
   },
   "optionalDependencies": {
     "webcrack": "^2.15.1"
@@ -71,9 +71,9 @@
     }
   },
   "devDependencies": {
-    "@opensteer/browser-core": "0.7.9",
-    "@opensteer/protocol": "0.8.3",
-    "@opensteer/engine-abp": "0.8.9"
+    "@opensteer/browser-core": "0.7.10",
+    "@opensteer/engine-abp": "0.8.9",
+    "@opensteer/protocol": "0.8.4"
   },
   "scripts": {
     "build": "tsup && node ../../scripts/copy-opensteer-local-view-assets.mjs && node ../../scripts/sync-package-skills.mjs && node ../../scripts/sync-package-readme.mjs",

package/skills/opensteer/SKILL.md

@@ -6,25 +6,48 @@ argument-hint: "[goal]"
 
 # Opensteer
 
-Opensteer gives AI agents a real Chromium browser — local or cloud. Use it when normal code is not enough because the task depends on a live browser session.
+Opensteer gives AI agents a real Chromium browser. Use it when the task depends on a live browser session — clicks, forms, extraction, cookies, network capture, or browser-backed fetch.
 
-Default workflow:
+## Core Workflow
 
-1. CLI to explore the site and discover behavior.
-2. Save stable targets with `persist`.
-3. SDK to write the final reusable TypeScript.
+Follow this order. Do not skip steps.
 
-Do not stop at manual exploration if the user needs automation.
+1. **Open** a browser in a workspace.
+2. **Snapshot** to see the page and get element numbers.
+3. **Interact** using element numbers from the latest snapshot. Every action requires `--persist <key>`.
+4. **Re-snapshot** after navigation or UI changes before reusing element numbers.
+5. **Extract** data using a template with `--persist <key>`.
+6. **Write SDK code** that replays persisted targets — no templates in the SDK, only persist keys.
+7. **Close** the browser when done.
 
-## Setup
+```bash
+opensteer open https://example.com --workspace demo
+opensteer snapshot action --workspace demo
+opensteer input 5 "laptop" --workspace demo --press-enter --persist "search input"
+opensteer click 7 --workspace demo --persist "search button"
+opensteer snapshot extraction --workspace demo
+opensteer extract '{"items":[{"title":13,"price":14},{"title":22,"price":23},{"title":31,"price":32}]}' --workspace demo --persist "search results"
+```
+
+```ts
+import { Opensteer } from "opensteer";
+
+const opensteer = new Opensteer({ workspace: "demo", rootDir: process.cwd() });
+
+await opensteer.open("https://example.com");
+await opensteer.input({ persist: "search input", text: "laptop", pressEnter: true });
+await opensteer.click({ persist: "search button" });
+const data = await opensteer.extract({ persist: "search results" });
+await opensteer.close();
+```
 
-Install the Opensteer skill so the coding agent can use it:
+## Setup
 
 ```bash
 opensteer skills install
 ```
 
-This registers the skill with the agent's tool system. Only needed once per environment.
+Only needed once per environment.
 
 ## When To Use
 
@@ -37,17 +60,6 @@ This registers the skill with the agent's tool system. Only needed once per envi
 
 If the user wants to manually drive a browser and record the flow, use the `recorder` skill instead.
 
-## Core Rules
-
-1. Always use a workspace for stateful commands: `--workspace <id>` or `OPENSTEER_WORKSPACE`.
-2. Re-snapshot after navigation or big UI changes before reusing element numbers.
-3. CLI to discover, SDK for the final implementation.
-4. Use `persist` for stable reusable targets and extraction payloads.
-5. Use `exec` for SDK code and API experiments. Use `evaluate` only for page-context JavaScript.
-6. If `fetch()` fails with auth errors, inspect `state()`, `cookies()`, and `storage()` before changing transport.
-7. Keep output simple. Prefer ordinary TypeScript with `Opensteer`, no extra abstraction.
-8. Close the browser when done. Do not leave headed browsers running. Use `opensteer browser delete --workspace <id>` or SDK cleanup when the session does not need to stay open.
-
 ## Choose A Path
 
 ```
@@ -67,26 +79,86 @@ What does the task need?
 
 Use this when the goal is clicking, typing, navigating, or extracting visible data.
 
-### CLI exploration
+### Persist is required
+
+Every `click`, `hover`, `input`, `scroll`, and `extract` command requires `--persist <key>`. This saves a stable element descriptor so the action is replayable across sessions. Name the key after what the element is:
 
 ```bash
-opensteer open https://example.com --workspace demo
-opensteer snapshot action --workspace demo
-opensteer input 5 "laptop" --workspace demo --press-enter --persist "search input"
 opensteer click 7 --workspace demo --persist "search button"
-opensteer snapshot extraction --workspace demo
-opensteer extract '{"items":[{"name":{"element":13},"price":{"element":14}}]}' \
-  --workspace demo \
-  --persist "search results"
+opensteer input 5 "laptop" --workspace demo --press-enter --persist "search input"
+opensteer scroll down 500 --workspace demo --persist "page scroll"
 ```
 
-Element numbers come from `c="N"` markers in the snapshot HTML.
+### Element numbers
+
+Element numbers come from `c="N"` markers in the snapshot HTML. They are only valid for the current snapshot. After navigation or DOM changes, snapshot again to get fresh numbers.
+
+```bash
+opensteer snapshot action --workspace demo      # for interactions
+opensteer snapshot extraction --workspace demo  # for data extraction
+```
+
+Read the full snapshot output. Do not pipe it through `head`, `grep`, or `sed` — filtering destroys the structural context you need to identify which elements belong to the same card.
+
+### Extraction templates
+
+The `extract` command takes a JSON template that describes the fields in one or more items. Opensteer merges the structural pattern across all provided examples and generalizes to every matching item on the page.
+
+**Template format:**
+
+- Bare number: `13` reads text content of element `c="13"`.
+- Object with attribute: `{"c": 13, "attr": "href"}` reads an attribute from that element.
+- Selector: `{"selector": "#price"}` targets by CSS selector.
+- Page source: `{"source": "current_url"}` reads page metadata.
+
+**How many items to include:**
+
+**Lazy — 3 items from 3 different positions (recommended for reusable SDK descriptors).** Give one entry per card for 3 different cards. Opensteer compares the 3 examples, cancels out position noise, and produces a descriptor that matches all similar items. Use this when the goal is a persist key the SDK can replay later.
+
+```bash
+opensteer extract '{
+  "products": [
+    {"title": 47, "price": 51, "url": {"c": 47, "attr": "href"}},
+    {"title": 62, "price": 66, "url": {"c": 62, "attr": "href"}},
+    {"title": 78, "price": 83, "url": {"c": 78, "attr": "href"}}
+  ]
+}' --workspace demo --persist "search results"
+```
+
+**Eager — all visible items (use when you need the full data immediately).** Include every item visible in the snapshot. This returns all data in one shot from the current session. The descriptor is still saved under `--persist` and can be replayed, but the generalization is weaker than the 3-item approach.
+
+```bash
+opensteer extract '{
+  "products": [
+    {"title": 47, "price": 51, "url": {"c": 47, "attr": "href"}},
+    {"title": 62, "price": 66, "url": {"c": 62, "attr": "href"}},
+    {"title": 78, "price": 83, "url": {"c": 78, "attr": "href"}},
+    ...continue for every visible card...
+  ]
+}' --workspace demo --persist "search results"
+```
+
+**Rule: all fields in each array entry must come from the same card.** Never take a field from card 1 and another field from card 2 within the same entry — that produces a broken descriptor.
+
+Wrong — title from card 1, price from card 2 mixed in the same entry:
+
+```bash
+# DO NOT DO THIS — fields across cards in one entry
+opensteer extract '{"products":[{"title":47,"price":66}]}' --workspace demo --persist "search results"
+```
+
+For non-array fields at the top level, point to the elements directly:
+
+```bash
+opensteer extract '{"pageTitle":3,"totalResults":8,"url":{"source":"current_url"}}' \
+  --workspace demo --persist "page metadata"
+```
 
 ### SDK implementation
 
-```ts
-import { Opensteer } from "opensteer";
+The SDK `extract()` method replays a previously persisted template. It does not accept inline templates — those belong in the CLI exploration phase.
 
+```ts
 const opensteer = new Opensteer({ workspace: "demo", rootDir: process.cwd() });
 
 await opensteer.open("https://example.com");
@@ -95,7 +167,7 @@ await opensteer.click({ persist: "search button" });
 const data = await opensteer.extract({ persist: "search results" });
 ```
 
-Use `selector` in SDK code only when a stable CSS selector is cleaner than `persist`.
+Use `selector` in SDK action code only when a stable CSS selector is cleaner than persist.
 
 ## Network Path
 
@@ -106,7 +178,7 @@ Use this when the goal is to find or replay a site API.
 ```bash
 opensteer open https://example.com --workspace demo
 opensteer goto https://example.com/search --workspace demo --capture-network page-load
-opensteer input 5 "laptop" --workspace demo --press-enter --capture-network search
+opensteer input 5 "laptop" --workspace demo --press-enter --persist "search input" --capture-network search
 opensteer network query --workspace demo --capture search --json
 opensteer network detail rec_123 --workspace demo --probe
 ```
@@ -174,10 +246,7 @@ Flags: `--inline`, `--external`, `--dynamic`, `--workers` to filter by source ty
 ### Beautify and deobfuscate
 
 ```bash
-# Format minified code
 opensteer scripts beautify <artifactId> --workspace demo --persist
-
-# Deobfuscate packed/obfuscated code
 opensteer scripts deobfuscate <artifactId> --workspace demo --persist
 ```
 
@@ -208,7 +277,7 @@ opensteer scripts sandbox art_ghi789 --workspace demo
 
 ## Computer-Use
 
-Use this only when DOM targeting is not enough.
+Use this only when DOM targeting is not enough — canvas, WebGL, or elements that cannot be reached by selector.
 
 ```bash
 opensteer computer click 245 380 --workspace demo --capture-network action
@@ -230,11 +299,10 @@ After coordinate-based actions, switch back to normal extraction or request anal
 Use when handling OAuth popups, multi-page flows, or any task that opens new tabs.
 
 ```bash
-opensteer tab list --workspace demo              # List all open tabs
-opensteer tab new https://example.com --workspace demo   # Open new tab
+opensteer tab list --workspace demo
+opensteer tab new https://example.com --workspace demo
 opensteer tab 2 --workspace demo                 # Switch to tab 2
-opensteer tab close 3 --workspace demo           # Close tab 3
-opensteer tab close --workspace demo             # Close current tab
+opensteer tab close 3 --workspace demo
 ```
 
 ```ts
@@ -250,11 +318,11 @@ Re-snapshot after switching tabs — element numbers are per-page.
 
 Each workspace has one browser. Three modes:
 
-| Mode                     | What it does                                    | Data persists?                                                        |
-| ------------------------ | ----------------------------------------------- | --------------------------------------------------------------------- |
-| **Persistent** (default) | Browser tied to workspace, survives restarts    | Yes — cookies, localStorage, logins, history, extensions all retained |
-| **Temporary**            | Headless browser in `/tmp`, cleaned up on close | No                                                                    |
-| **Attach**               | Connects to an already-running browser via CDP  | Depends on that browser                                               |
+| Mode                     | What it does                                    | Data persists?          |
+| ------------------------ | ----------------------------------------------- | ----------------------- |
+| **Persistent** (default) | Browser tied to workspace, survives restarts    | Yes                     |
+| **Temporary**            | Headless browser in `/tmp`, cleaned up on close | No                      |
+| **Attach**               | Connects to an already-running browser via CDP  | Depends on that browser |
 
 ### Headless vs headed
 
@@ -266,99 +334,69 @@ opensteer open https://example.com --workspace demo --headless false
 
 Use headed mode for debugging or when the user wants to watch. For hands-free automation, keep headless and use `opensteer view` if a human needs to observe.
 
-### Persistent sessions
-
-When you `opensteer open` with a workspace, the browser's full Chrome user-data directory lives at `~/.opensteer/workspaces/<id>/browser/user-data/`. Everything Chrome normally persists (cookies, localStorage, IndexedDB, history, extensions) survives between runs.
-
-Re-running `opensteer open --workspace demo` reconnects to the existing browser if it's still alive, or launches a fresh one with the same profile if it died.
-
 ### Profile cloning
 
 Clone a real user's Chrome profile to start a workspace with their logins already active:
 
 ```bash
-# Discover available local browsers and profiles
 opensteer browser discover
-
-# Clone a profile into a workspace
 opensteer browser clone --workspace demo \
   --source-user-data-dir "$HOME/Library/Application Support/Google/Chrome" \
   --source-profile-directory Default
 ```
 
-This copies cookies, localStorage, extensions, and settings from the source browser. Caches and lock files are skipped. The source browser does not need to be closed — cloning while running is safe.
-
-### Attach to an existing browser
-
-```bash
-opensteer open https://example.com --workspace demo --attach-endpoint http://localhost:9222
-```
+This copies cookies, localStorage, extensions, and settings. The source browser does not need to be closed.
 
 ### Workspace lifecycle
 
 ```bash
-opensteer browser status --workspace demo    # Check if browser is running
+opensteer browser status --workspace demo
 opensteer browser reset --workspace demo     # Wipe browser data, keep workspace
 opensteer browser delete --workspace demo    # Delete workspace entirely
 ```
 
 ## Cloud Mode
 
-Run the browser on Opensteer's cloud infrastructure instead of locally. Use cloud mode when you need browsers that run 24/7, or when the local machine should not run Chromium.
-
-### Setup
+Run the browser on Opensteer's cloud infrastructure instead of locally.
 
 ```bash
-export OPENSTEER_API_KEY=osk_your_key_here    # Required
-export OPENSTEER_PROVIDER=cloud               # Or use --provider cloud per command
+export OPENSTEER_API_KEY=osk_your_key_here
+export OPENSTEER_PROVIDER=cloud
 ```
 
-### Usage
-
 All CLI commands work the same with `--provider cloud`:
 
 ```bash
 opensteer open https://example.com --workspace demo --provider cloud
 opensteer snapshot action --workspace demo
-opensteer click 5 --workspace demo
+opensteer click 5 --workspace demo --persist "nav link"
 ```
 
-### Export local browser profile to cloud
-
-Sync a local browser's cookies to a cloud browser profile so the cloud session starts logged in:
+Export a local profile to cloud:
 
 ```bash
-# Reads cookies from local Chrome, decrypts them, uploads to cloud
 opensteer browser clone --workspace demo \
   --source-user-data-dir "$HOME/Library/Application Support/Google/Chrome" \
   --source-profile-directory Default \
   --provider cloud
 ```
 
-The cookies are extracted from the local SQLite database, decrypted, packaged into a portable format, and uploaded. The cloud browser then starts with those cookies applied.
-
 ## Local View
 
-When Opensteer runs headless, a human cannot see what the browser is doing. Local view streams live screenshots from headless sessions to a browser-based viewer.
+Stream live screenshots from headless sessions to a browser-based viewer.
 
 ```bash
 opensteer view                   # Start viewer service, print URL
 opensteer view stop              # Stop the viewer service
-opensteer view --auto            # Auto-start viewer on every browser launch
-opensteer view --no-auto         # Only start viewer when manually requested
+opensteer view --auto            # Auto-start on every browser launch
+opensteer view --no-auto         # Only start when manually requested
 ```
 
-The viewer is a local web UI that shows:
-
-- Live JPEG stream of the active browser tab
-- Tab bar with switching
-- Navigation controls (back, forward, reload, URL bar)
-
-Local view is a passive observer — it connects to the browser's existing CDP endpoint. Starting or stopping it has zero impact on running browser sessions.
+Local view is a passive observer. Starting or stopping it has zero impact on running sessions.
 
 ## Interaction Capture & Replay
 
-Record a trace of browser interactions (clicks, typing, network, DOM changes) and replay them deterministically. Useful for building repeatable test flows or comparing behavior across runs.
+Record browser interactions and replay them deterministically.
 
 ```bash
 opensteer interaction capture --workspace demo --key "login-flow" --duration 30000
@@ -375,14 +413,11 @@ Commands that use `--persist` save artifacts to the workspace. Read them back wi
 opensteer artifact read <artifactId> --workspace demo
 ```
 
-Artifacts are created by `extract --persist`, `scripts capture --persist`, `scripts beautify --persist`, and other persist-enabled commands.
-
-## Useful SDK Surface
+## SDK Surface
 
 - `open(url)`, `goto(url, { captureNetwork? })`, `close()`
-- `snapshot("action" | "extraction")`
 - `click()`, `hover()`, `input()`, `scroll()`
-- `extract()`
+- `extract({ persist })` — replay-only, no inline templates
 - `listPages()`, `newPage()`, `activatePage()`, `closePage()`
 - `network.query()`, `network.detail()`
 - `waitForPage()`
@@ -395,8 +430,13 @@ Artifacts are created by `extract --persist`, `scripts capture --persist`, `scri
 
 ## Guardrails
 
-- Snapshot before using element numbers. Snapshot again after UI changes.
+- Always snapshot before using element numbers. Snapshot again after UI changes.
+- Always include `--persist <key>` on click, hover, input, scroll, and extract.
+- Extraction templates: use 3 items from 3 different positions for reusable descriptors; use all visible items when you need the full data immediately. All fields in each array entry must come from the same card/row.
+- Do not pass templates to the SDK `extract()` — use persist keys only.
+- Re-snapshot after navigation before reusing element numbers.
 - Do not use `evaluate` for API work — use `exec` or `fetch`.
+- If `fetch()` fails with auth errors, check `state()`, `cookies()`, `storage()` first.
 - Do not keep the result as a manual-only workflow if the user needs reusable automation.
 - Prefer a small final script over a large framework.
 - Close browsers when done. Do not leave headed browser windows open.