opensteer 0.8.14 → 0.8.16

package/skills/opensteer/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: opensteer
-description: "Handles Opensteer browser automation, structured DOM extraction, and browser-backed API reverse engineering. Use when the user mentions Opensteer, browser automation, real Chromium sessions, DOM extraction, network capture, replay, cookies, reverse-engineering a site API, or browser-grade fetch."
+description: "Handles Opensteer browser automation, structured DOM extraction, and browser-backed API reverse engineering. Use when the user mentions Opensteer, browser automation, real Chromium sessions, DOM extraction, network capture, reverse-engineering a site API, or browser-grade fetch."
 argument-hint: "[goal]"
 ---
 
@@ -9,7 +9,7 @@ argument-hint: "[goal]"
 Opensteer gives agents a real Chromium browser for two jobs normal code cannot do:
 
 1. **DOM automation** — interact with pages and extract structured data via snapshot-based element targeting.
-2. **API reverse engineering** — capture real browser traffic, identify APIs, test transport portability, and write replay code.
+2. **API reverse engineering** — capture real browser traffic, identify APIs, test transport portability, and write reusable code.
 
 The workflow is always: **explore with CLI, then write reusable code with SDK.**
 
@@ -56,7 +56,7 @@ opensteer hover 3 --workspace demo --persist "menu trigger"
 opensteer scroll down 400 --workspace demo
 ```
 
-`--persist <name>` saves the element's structural DOM path for deterministic SDK replay. Element number is always required on CLI — persist is save-only, not a targeting mode.
+`--persist <key>` saves the element's structural DOM path for deterministic SDK replay. Element number is always required on CLI — persist is save-only, not a targeting mode.
 
 ### Step 3: Extract
 
@@ -85,7 +85,7 @@ const opensteer = new Opensteer({ workspace: "demo", rootDir: process.cwd() });
 try {
   await opensteer.open("https://example.com");
 
-  // Replay by persist name — no element numbers or snapshots needed
+  // Replay by persist key — no element numbers or snapshots needed
   await opensteer.input({ persist: "search input", text: "laptop", pressEnter: true });
   await opensteer.click({ persist: "search button" });
 
@@ -122,40 +122,51 @@ opensteer network query --workspace demo --capture search --hostname api.example
 
 `--json` filters to JSON and GraphQL responses only. Other filters: `--url`, `--path`, `--method`, `--status`, `--type`, `--before`, `--after`, `--limit`.
 
-### Step 3: Inspect
+### Step 3: Inspect and probe transport
 
 ```bash
 opensteer network detail rec_123 --workspace demo
+opensteer network detail rec_123 --probe --workspace demo
 ```
 
-Shows: URL, method, request headers, cookies sent, request/response body preview, GraphQL metadata, redirect chain.
+The first command shows: URL, method, request headers, cookies sent, request/response body preview, GraphQL metadata, redirect chain.
 
-### Step 4: Test replay
+Add `--probe` to also test which transport works for this API:
+
+| Transport     | Meaning                        | SDK usage                                       |
+| ------------- | ------------------------------ | ----------------------------------------------- |
+| `direct-http` | Plain HTTP works               | `this.fetch(url)` (default)                     |
+| `matched-tls` | Needs TLS fingerprint matching | `this.fetch(url, { transport: "matched-tls" })` |
+| `page-http`   | Needs a live browser page      | `this.fetch(url, { transport: "page" })`        |
+
+### Step 4: Check browser state (if 401/403)
 
 ```bash
-opensteer replay rec_123 --workspace demo
-opensteer replay rec_123 --workspace demo --query keyword=headphones --query count=10
+opensteer state example.com --workspace demo
 ```
 
-Replay tries transports automatically and reports which succeeded:
+Look for session cookies, CSRF tokens, or storage-backed auth that the request depends on.
 
-| Transport | Meaning | SDK usage |
-|---|---|---|
-| `direct-http` | Plain HTTP works | `opensteer.fetch(url)` (default) |
-| `matched-tls` | Needs TLS fingerprint matching | `opensteer.fetch(url, { transport: "matched-tls" })` |
-| `page-http` | Needs a live browser page | `opensteer.fetch(url, { transport: "page" })` |
+### Step 5: Test and write code with exec
 
-### Step 5: Check browser state (if replay returns 401/403)
+Use `exec` to test the API call with the SDK. The code you write here is the same code that goes in your final script:
 
 ```bash
-opensteer cookies example.com --workspace demo
-opensteer storage example.com --workspace demo
-opensteer state example.com --workspace demo
+opensteer exec "
+  const r = await this.fetch('https://api.example.com/search', {
+    method: 'POST',
+    headers: { 'content-type': 'application/json' },
+    body: JSON.stringify({ keyword: 'laptop', count: 24 }),
+  });
+  return { status: r.status, data: await r.json() };
+" --workspace demo
 ```
 
-Look for session cookies, CSRF tokens, or storage-backed auth that the request depends on.
+`exec` runs JavaScript with the Opensteer SDK bound as `this`. It supports `await` and has access to `this.fetch()`, `this.cookies()`, `this.evaluate()`, and all other SDK methods.
+
+IMPORTANT: Use `exec` instead of `evaluate` for API calls. `evaluate` runs inside the browser page where anti-bot scripts can detect and block programmatic requests. `exec` runs through the framework's transport stack which automatically finds the least detectable path.
 
-### Step 6: Write SDK code
+### Step 6: Write the final SDK script
 
 ```bash
 opensteer close --workspace demo
@@ -165,31 +176,17 @@ opensteer close --workspace demo
 import { Opensteer } from "opensteer";
 const opensteer = new Opensteer({ workspace: "demo", rootDir: process.cwd() });
 
-async function ensureSession() {
-  const cookies = await opensteer.cookies("example.com");
-  if (cookies.has("session")) return;
-  await opensteer.goto("https://example.com");
-}
-
 export async function search(keyword: string) {
-  await ensureSession();
   const response = await opensteer.fetch("https://api.example.com/search", {
-    query: { keyword, count: 24 },
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify({ keyword, count: 24 }),
   });
   return response.json();
 }
 ```
 
-If replay showed a required transport, carry it into `fetch()`:
-
-```ts
-const response = await opensteer.fetch("https://api.example.com/search", {
-  query: { keyword: "laptop" },
-  transport: "matched-tls",
-});
-```
-
-IMPORTANT: `opensteer.fetch()` does not require opening a browser page. It works directly with the session's cookies and transport stack. Only call `opensteer.open()` or `opensteer.goto()` if you need to establish session cookies first.
+`opensteer.fetch()` accepts standard Web Fetch API syntax (`body` as string, `headers` as `Record<string, string>`). It auto-selects the best transport and includes browser cookies by default.
 
 ---
 
@@ -244,6 +241,14 @@ opensteer tab close 2 --workspace demo
 
 ## Run JavaScript
 
+Use `exec` for API calls and SDK operations (runs in Node.js, avoids bot detection):
+
+```bash
+opensteer exec "await this.fetch('https://api.example.com/data').then(r => r.json())" --workspace demo
+```
+
+Use `evaluate` only for DOM inspection (runs inside the browser page):
+
 ```bash
 opensteer evaluate "document.title" --workspace demo
 ```
@@ -264,28 +269,26 @@ opensteer computer screenshot --workspace demo
 
 ## CLI Quick Reference
 
-| Command | Positional args | Key flags |
-|---|---|---|
-| `open <url>` | url | `--headless`, `--provider`, `--attach-endpoint`, `--attach-header` |
-| `close` | — | — |
-| `status` | — | — |
-| `goto <url>` | url | `--capture-network` |
-| `snapshot [mode]` | action \| extraction | — |
-| `click <element>` | element number | `--persist`, `--capture-network`, `--button` |
-| `hover <element>` | element number | `--persist`, `--capture-network` |
-| `input <element> <text>` | element, text | `--persist`, `--press-enter`, `--capture-network` |
-| `scroll <dir> <amount>` | direction, amount | `--element`, `--persist`, `--capture-network` |
-| `extract <schema>` | JSON schema | `--persist` |
-| `evaluate <script>` | JS expression | — |
-| `network query` | — | `--capture`, `--url`, `--hostname`, `--json`, `--limit`, +6 filters |
-| `network detail <id>` | recordId | — |
-| `replay <id>` | recordId | `--query`, `--header`, `--body`, `--variables` |
-| `fetch <url>` | url | `--method`, `--header`, `--query`, `--body`, `--transport`, `--cookies` |
-| `cookies [domain]` | domain (optional) | — |
-| `storage [domain]` | domain (optional) | — |
-| `state [domain]` | domain (optional) | — |
-| `tab list / new / <n> / close` | varies | — |
-| `computer click/type/key/scroll/move/drag/screenshot/wait` | varies | `--capture-network` |
+| Command                                                    | Positional args      | Key flags                                                               |
+| ---------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------- |
+| `open <url>`                                               | url                  | `--headless`, `--provider`, `--attach-endpoint`, `--attach-header`      |
+| `close`                                                    | —                    | —                                                                       |
+| `status`                                                   | —                    | —                                                                       |
+| `goto <url>`                                               | url                  | `--capture-network`                                                     |
+| `snapshot [mode]`                                          | action \| extraction | —                                                                       |
+| `click <element>`                                          | element number       | `--persist`, `--capture-network`, `--button`                            |
+| `hover <element>`                                          | element number       | `--persist`, `--capture-network`                                        |
+| `input <element> <text>`                                   | element, text        | `--persist`, `--press-enter`, `--capture-network`                       |
+| `scroll <dir> <amount>`                                    | direction, amount    | `--element`, `--persist`, `--capture-network`                           |
+| `extract <schema>`                                         | JSON schema          | `--persist`                                                             |
+| `evaluate <script>`                                        | JS expression        | —                                                                       |
+| `network query`                                            | —                    | `--capture`, `--url`, `--hostname`, `--json`, `--limit`, +6 filters     |
+| `network detail <id>`                                      | recordId             | `--probe`                                                               |
+| `fetch <url>`                                              | url                  | `--method`, `--header`, `--query`, `--body`, `--transport`, `--cookies` |
+| `state [domain]`                                           | domain (optional)    | —                                                                       |
+| `exec <expression>`                                        | JS expression        | —                                                                       |
+| `tab list / new / <n> / close`                             | varies               | —                                                                       |
+| `computer click/type/key/scroll/move/drag/screenshot/wait` | varies               | `--capture-network`                                                     |
 
 ## SDK Quick Reference
 
@@ -312,20 +315,16 @@ await opensteer.extract({ persist: "name", schema: { ... } });   // inline schem
 // Network discovery
 const records = await opensteer.network.query({ capture: "label", limit: 20 });
 const detail = await opensteer.network.detail(recordId);
-const replay = await opensteer.network.replay(recordId, { query: { k: "v" } });
 
-// Fetch — works without a page open
+// Fetch — standard Web Fetch API syntax, auto-selects transport
 const response = await opensteer.fetch(url, {
-  query?: { key: "value" },
-  method?: "POST",
-  headers?: { Authorization: "Bearer ..." },
-  body?: { ... },
-  transport?: "auto" | "direct" | "matched-tls" | "page",
+  method: "POST",
+  headers: { "content-type": "application/json" },
+  body: JSON.stringify({ keyword: "laptop" }),
 });
 
 // Browser state
 const cookies = await opensteer.cookies("domain.com"); // .has(), .get(), .serialize()
-const storage = await opensteer.storage("domain.com", "local");
 const state = await opensteer.state("domain.com");
 
 // Snapshots
@@ -337,11 +336,11 @@ const html = await opensteer.snapshot("extraction");
 
 ## Common Issues
 
-| Symptom | Fix |
-|---|---|
-| Element numbers wrong after navigation | Re-snapshot before using element numbers |
-| `replay` returns 401/403 | Check `cookies`, `storage`, `state` — request depends on session tokens |
-| `replay` works but `fetch()` fails | Use the transport replay discovered: `transport: "matched-tls"` or `"page"` |
-| Direct HTTP blocked, browser transport works | Site uses TLS fingerprinting — use `transport: "matched-tls"` |
-| Extract returns empty data | Element numbers changed — re-snapshot and rebuild the schema |
-| `fetch()` fails with no session | Call `opensteer.goto(url)` first to establish cookies, then `fetch()` |
+| Symptom                                      | Fix                                                                                                |
+| -------------------------------------------- | -------------------------------------------------------------------------------------------------- |
+| Element numbers wrong after navigation       | Re-snapshot before using element numbers                                                           |
+| `fetch()` returns 401/403                    | Check `state` — request depends on session cookies or tokens                                       |
+| Direct HTTP blocked, browser transport works | Site uses TLS fingerprinting — use `transport: "matched-tls"`                                      |
+| Extract returns empty data                   | Element numbers changed — re-snapshot and rebuild the schema                                       |
+| `fetch()` fails with no session              | Call `opensteer.goto(url)` first to establish cookies, then `fetch()`                              |
+| Using `evaluate` for API calls               | Use `exec` instead — `evaluate` runs inside the page where anti-bot scripts can intercept requests |