opensteer 0.8.13 → 0.8.15

package/skills/opensteer/references/sdk-reference.md

@@ -1,17 +1,6 @@
 # Opensteer SDK Reference
 
-This covers all task paths (DOM, request capture, browser admin). For task triage, see SKILL.md.
-
-Use the SDK when the workflow should become reusable TypeScript code in the repository.
-
-## Sections
-
-- [Construction](#construction)
-- [DOM Automation And Extraction](#dom-automation-and-extraction)
-- [Browser Admin](#browser-admin)
-- [Request Capture, Plans, And Recipes](#request-capture-plans-and-recipes)
-- [Common Methods](#common-methods)
-- [Rules](#rules)
+Use the SDK when the result should become reusable TypeScript in the repository.
 
 ## Construction
 
@@ -19,143 +8,127 @@ Use the SDK when the workflow should become reusable TypeScript code in the repo
 import { Opensteer } from "opensteer";
 
 const opensteer = new Opensteer({
-  workspace: "github-sync",
+  workspace: "demo",
   rootDir: process.cwd(),
-  browser: "persistent",
-  launch: {
-    headless: false,
-  },
-  context: {
-    locale: "en-US",
-  },
 });
 ```
 
-- `workspace` creates a repo-local persistent root under `.opensteer/workspaces/<id>`.
-- Omitting `workspace` creates a temporary root.
-- `browser` can be `persistent`, `temporary`, or `{ mode: "attach", endpoint?, freshTab? }`.
-- `opensteer.browser.status()`, `clone()`, `reset()`, and `delete()` manage the persistent workspace browser.
-- `close()` shuts the current session and, for persistent workspaces, closes the live browser process.
-- `disconnect()` detaches local runtime handles and leaves the workspace/browser files intact.
-- Network history is SQLite-backed and auto-persisted. Records are written to SQLite as they are captured during `goto()`, `click()`, and other actions. The database initializes on first use.
-- Generic workspace and browser helpers do not require SQLite capability unless they touch network history persistence.
-- The current public SDK does not expose `Opensteer.attach()`, cloud session helpers, or the ABP engine.
+Key options:
 
-## DOM Automation And Extraction
+- `workspace`: persistent repo-local browser state
+- `rootDir`: where `.opensteer` lives
+- `browser`: local browser mode or attach config
+- `provider`: cloud config when applicable
 
-Opensteer uses a two-phase workflow: **explore** with the CLI, then **replay** with the SDK.
+## DOM Automation
 
-### Phase 1 — Exploration (one-time, via CLI or setup script)
+Explore with the CLI first. Use the snapshot `html` output to find `c="N"` element numbers.
 
-Run `opensteer snapshot action --workspace demo` from the CLI first. Read the `html` field in the JSON output — it is a clean filtered DOM with `c="N"` attributes. Use those counter numbers as the `element` parameter below. The SDK also exposes `snapshot()`, but this guide keeps discovery in the CLI so the DOM HTML is easy to inspect from the terminal.
+Persist a DOM action target during exploration:
 
 ```ts
-import { Opensteer } from "opensteer";
-
-const opensteer = new Opensteer({
-  workspace: "demo",
-  rootDir: process.cwd(),
-});
-
-await opensteer.open("https://example.com");
-
-// element numbers come from c="N" values in the snapshot html field
 await opensteer.click({
   element: 3,
-  description: "primary button", // caches the element path
+  persist: "primary button",
 });
 
 await opensteer.input({
   element: 7,
-  description: "search input", // caches the element path
+  persist: "search input",
   text: "laptop",
   pressEnter: true,
 });
+```
+
+Replay it later without element numbers:
 
-await opensteer.extract({
-  description: "page summary",
+```ts
+await opensteer.click({ persist: "primary button" });
+await opensteer.input({ persist: "search input", text: "laptop", pressEnter: true });
+```
+
+Persist works for extraction too:
+
+```ts
+const summary = await opensteer.extract({
+  persist: "page summary",
   schema: {
     title: { selector: "title" },
     url: { source: "current_url" },
   },
 });
-
-await opensteer.close();
 ```
 
-### Phase 2 — Deterministic replay (the actual reusable script)
+Rules:
 
-Use `description` alone for everything — resolves from cached descriptors:
+- Use `persist` for reusable DOM targets and extraction descriptors.
+- Use `selector` only as a low-level escape hatch.
+
+## Network Discovery
 
 ```ts
-const opensteer = new Opensteer({
-  workspace: "demo",
-  rootDir: process.cwd(),
+await opensteer.goto("https://example.com/search", {
+  captureNetwork: "search",
 });
 
-await opensteer.open("https://example.com");
-
-await opensteer.click({ description: "primary button" });
-await opensteer.input({ description: "search input", text: "laptop", pressEnter: true });
-const data = await opensteer.extract({ description: "page summary" });
+const records = await opensteer.network.query({
+  capture: "search",
+  limit: 20,
+});
 
-await opensteer.close();
+const detail = await opensteer.network.detail(records.records[0]!.recordId);
+const replay = await opensteer.network.replay(records.records[0]!.recordId, {
+  query: { keyword: "headphones" },
+});
 ```
 
-DOM rules:
+Use:
 
-- Deterministic scripts use `description` for all interactions and extractions — no snapshots, no selectors.
-- `element + description` persists a DOM action descriptor. Bare `description` replays it later.
-- `description + schema` writes or updates a persisted extraction descriptor. Bare `description` replays it later.
-- Use `element` targets only during the exploration phase with a fresh snapshot from the CLI.
-- Keep DOM data collection in `extract()`, not `evaluate()` or raw page DOM parsing, when the result can be expressed as structured fields.
-- CSS selectors exist as a low-level escape hatch but are not recommended for reusable scripts.
+- `network.query()` to shortlist requests
+- `network.detail()` to inspect one request deeply
+- `network.replay()` to confirm the transport and response shape
 
-Supported extraction field shapes:
+## Browser State
 
-- `{ element: N }` — requires a prior CLI snapshot; use during exploration only
-- `{ element: N, attribute: "href" }`
-- `{ selector: ".price" }`
-- `{ selector: "img.hero", attribute: "src" }`
-- `{ source: "current_url" }`
+```ts
+const cookies = await opensteer.cookies("example.com");
+const localStorage = await opensteer.storage("example.com", "local");
+const sessionStorage = await opensteer.storage("example.com", "session");
+const browserState = await opensteer.state("example.com");
+```
 
-For arrays, provide 1-2 representative objects. The extractor auto-generalizes from these templates to find ALL matching rows on the page:
+`cookies()` returns a small cookie-jar helper:
 
 ```ts
-const results = await opensteer.extract({
-  description: "search results",
-  schema: {
-    items: [
-      { name: { element: 13 }, price: { element: 14 } },
-      { name: { element: 22 }, price: { element: 23 } },
-    ],
-  },
-});
-// results.items contains ALL matching rows on the page, not just the 2 templates
+cookies.has("session");
+cookies.get("session");
+cookies.getAll();
+cookies.serialize();
 ```
 
-Do not use `prompt` or semantic placeholder values such as `"string"` in the current public SDK. The extractor expects explicit schema objects, arrays, and field descriptors.
-
-### What extract() Returns
+## Session-Aware Fetch
 
-`extract()` returns a plain JSON object matching your schema shape:
+`fetch()` is the main replay primitive for final code.
 
 ```ts
-// Flat schema:
-{ title: "Search Results", url: "https://..." }
-
-// Array schema (auto-generalized from 1-2 templates):
-{
-  items: [
-    { name: "Apple AirPods Max", price: "$549.99" },
-    { name: "Apple AirPods Pro", price: "$249.99" },
-    { name: "Apple AirPods 4", price: "$129.99" },
-    // ... ALL matching rows
-  ]
-}
+const response = await opensteer.fetch("https://api.example.com/search", {
+  query: {
+    keyword: "laptop",
+    count: 24,
+  },
+});
+
+const data = await response.json();
 ```
 
-Use `extract()` for structured data. Do NOT use `evaluate()` or raw DOM parsing when `extract()` can express the result.
+If exploration showed a required transport:
+
+```ts
+const response = await opensteer.fetch("https://api.example.com/search", {
+  query: { keyword: "laptop" },
+  transport: "matched-tls",
+});
+```
 
 ## Browser Admin
 
@@ -170,244 +143,15 @@ if (!status.live) {
 }
 ```
 
-- `browser.clone()` is only for persistent workspace browsers.
-- Clone before `open()` when the workflow needs local authenticated browser state.
-- `browser.reset()` clears cloned browser state but keeps the workspace.
-- `browser.delete()` removes workspace browser files.
-
-## Request Capture, Plans, And Recipes
-
-For the complete pipeline with mandatory phases, see [Request Plan Pipeline](request-workflow.md). This section covers SDK method signatures and examples for reusable scripts.
-
-The deliverable of a request capture workflow is a persisted request plan tested via `request()`. `rawRequest()` is a diagnostic probe — not the deliverable.
-
-### Capture, Probe, and Infer
-
-```ts
-await opensteer.open();
-await opensteer.goto({
-  url: "https://example.com/app",
-  captureNetwork: "page-load",
-});
-
-await opensteer.click({
-  selector: "button.load-products",
-  description: "load products",
-  captureNetwork: "products-load",
-});
-
-const records = await opensteer.queryNetwork({
-  capture: "products-load",
-  includeBodies: true,
-  limit: 20,
-});
-
-// DIAGNOSTIC ONLY — probe transport to determine portability.
-// Do NOT return this data as the final answer. Proceed to inferRequestPlan.
-const response = await opensteer.rawRequest({
-  transport: "direct-http",
-  url: "https://example.com/api/products",
-  method: "POST",
-  body: {
-    json: { page: 1 },
-  },
-});
-
-// Infer plan with the transport you proved works
-await opensteer.inferRequestPlan({
-  recordId: records.records[0]!.recordId,
-  key: "products.search",
-  version: "v1",
-  transport: "direct-http",
-});
-
-// Replay the plan — this is the deliverable
-await opensteer.request("products.search", {
-  query: { q: "laptop" },
-});
-```
-
-### Read and Fix Plans
-
-After inferring a plan, read it to validate auth and annotate parameters:
-
-```ts
-// Read the inferred plan
-const plan = await opensteer.getRequestPlan({
-  key: "products.search",
-  version: "v1",
-});
-
-// Inspect plan.payload.auth — if auth.strategy is set but the API is public,
-// rewrite the plan with auth removed
-await opensteer.writeRequestPlan({
-  key: "products.search",
-  version: "v1",
-  tags: ["products", "search"],
-  provenance: {
-    source: "manual",
-    notes: "Auth removed — API is public. Parameters annotated.",
-  },
-  payload: {
-    ...plan.payload,
-    auth: undefined, // Remove spurious auth classification
-    parameters: [
-      { name: "q", in: "query", required: true, description: "Search keyword" },
-      { name: "count", in: "query", defaultValue: "24", description: "Results per page" },
-      { name: "offset", in: "query", defaultValue: "0", description: "Pagination offset" },
-    ],
-  },
-});
-```
+## Recommended Rules
 
-### Auth Recipes
+- Explore with the CLI first, then write reusable SDK code.
+- Use `captureNetwork` on the real browser actions that trigger the traffic.
+- Let `replay` tell you the required transport instead of guessing.
+- Keep the final artifact as code, not as shell commands or giant logs.
 
-When an API genuinely requires auth, create an auth recipe that acquires tokens automatically:
-
-```ts
-// Write an auth recipe that acquires a bearer token
-await opensteer.writeAuthRecipe({
-  key: "example.auth",
-  version: "v1",
-  payload: {
-    description: "Acquire guest bearer token for example.com API",
-    steps: [
-      {
-        kind: "directRequest",
-        request: {
-          url: "https://example.com/api/oauth/token",
-          transport: "direct-http",
-          method: "POST",
-          body: { json: { grant_type: "client_credentials" } },
-        },
-        capture: {
-          bodyJsonPointer: { pointer: "/access_token", saveAs: "token" },
-        },
-      },
-    ],
-    outputs: {
-      headers: { Authorization: "Bearer {{token}}" },
-    },
-  },
-});
-
-// Bind the auth recipe to a plan by rewriting the plan with auth.recipe
-const plan = await opensteer.getRequestPlan({ key: "products.search" });
-await opensteer.writeRequestPlan({
-  key: "products.search",
-  version: "v1",
-  payload: {
-    ...plan.payload,
-    auth: {
-      strategy: "bearer-token",
-      recipe: { key: "example.auth", version: "v1" },
-      failurePolicy: { on: "status", status: "401", action: "recover" },
-    },
-  },
-});
-
-// Now request.execute automatically runs the auth recipe on 401
-const result = await opensteer.request("products.search", {
-  query: { q: "laptop" },
-});
-```
+## What Not To Do
 
-**Recipe step types:** `directRequest`, `sessionRequest`, `request`, `readCookie`, `readStorage`, `evaluate`, `waitForNetwork`, `waitForCookie`, `goto`, `solveCaptcha`, `hook`. Each step can `capture` values; `outputs` maps captured variables to `headers`, `query`, `params`, or `body` overrides.
-
-### Rules
-
-- `captureNetwork` is supported on `goto()`, `click()`, `scroll()`, `input()`, and `hover()`. It is NOT supported on `open()`. Use `open()` then `goto({ url, captureNetwork })` to name initial navigation capture.
-- Query by capture first, then query all traffic to catch async requests that fire after page load.
-- Probe discovered APIs with `rawRequest()` using `direct-http` first, then `context-http`. `rawRequest()` is diagnostic — always proceed to `inferRequestPlan`.
-- Persistence is automatic; use `tagNetwork()` when you want to label a slice of already-persisted history for later lookup.
-- Use recipes when replay needs deterministic setup work. Use auth recipes when the setup is specifically auth-related. They live in separate registries.
-
-### Input Shapes
-
-- `rawRequest` `headers` MUST be an array: `[{ name: "Authorization", value: "Bearer ..." }]`. NOT `{ Authorization: "Bearer ..." }`.
-- `rawRequest` `body` MUST be one of: `{ json: { ... } }`, `{ text: "..." }`, or `{ base64: "..." }`. NOT a raw string or object.
-- `rawRequest()` may populate parsed JSON on `data`. If it does not, decode `response.body.data` with `Buffer.from(..., "base64").toString("utf8")`.
-- Recipe step `request` fields accept `{ key: value }` objects for headers (unlike `rawRequest`).
-
-### Common Errors
-
-| Error                                                        | Cause                                                 | Fix                                                               |
-| ------------------------------------------------------------ | ----------------------------------------------------- | ----------------------------------------------------------------- |
-| `"captureNetwork is not allowed"`                            | Used `captureNetwork` on `open()`                    | Move to `goto({ url, captureNetwork })`                           |
-| `"must be array"` on `rawRequest`                            | Headers passed as `{key: value}`                      | Use `[{name, value}]` array                                       |
-| `"must match exactly one supported shape"`                   | Body passed as raw string                             | Wrap in `{json: {...}}` or `{text: "..."}`                        |
-| `"Specify exactly one of element, selector, or description"` | `scroll()` called without a target                    | Add `selector: "body"` or a `description`                         |
-| `"registry record already exists"`                           | `inferRequestPlan` called twice with same key+version | Catch the error or use a new version                              |
-| `"no stored extraction descriptor"`                          | `extract()` called with `description` but no `schema` | Always provide `schema` unless a descriptor was previously stored |
-
-## Common Methods
-
-Session and page control:
-
-- `new Opensteer({ workspace?, rootDir?, browser?, launch?, context? })`
-- `open(url | { url?, workspace?, browser?, launch?, context? })`
-- `goto(url | { url, captureNetwork? })`
-- `listPages()`
-- `newPage({ url?, openerPageRef? })`
-- `activatePage({ pageRef })`
-- `closePage({ pageRef })`
-- `waitForPage({ openerPageRef?, urlIncludes?, timeoutMs? })`
-
-Interaction and extraction:
-
-- `click({ element | selector | description, captureNetwork? })`
-- `hover({ element | selector | description, captureNetwork? })`
-- `input({ element | selector | description, text, pressEnter?, captureNetwork? })`
-- `scroll({ element | selector | description, direction, amount, captureNetwork? })`
-- `extract({ description, schema? })`
-
-Inspection and evaluation:
-
-- `evaluate(script | { script, pageRef?, args? })`
-- `evaluateJson({ script, pageRef?, args? })`
-- `waitForNetwork({ ...filters, pageRef?, includeBodies?, timeoutMs? })`
-- `waitForResponse({ ...filters, pageRef?, includeBodies?, timeoutMs? })`
-- `queryNetwork({ ...filters, includeBodies?, limit? })`
-- `tagNetwork({ tag, ...filters })`
-- `clearNetwork({ tag? })`
-
-Request capture and replay:
-
-- `rawRequest({ transport?, pageRef?, url, method?, headers?, body?, followRedirects? })`
-- `inferRequestPlan({ recordId, key, version, transport? })`
-- `writeRequestPlan({ key, version, payload, tags?, provenance?, freshness? })`
-- `getRequestPlan({ key, version? })`
-- `listRequestPlans({ key? })`
-- `request(key, { path?, query?, headers?, body? })`
-- `writeRecipe({ key, version, payload, tags?, provenance? })`
-- `getRecipe({ key, version? })`
-- `listRecipes({ key? })`
-- `runRecipe({ key, version?, input? })`
-- `writeAuthRecipe({ key, version, payload, tags?, provenance? })`
-- `getAuthRecipe({ key, version? })`
-- `listAuthRecipes({ key? })`
-- `runAuthRecipe({ key, version?, input? })`
-
-Browser helpers:
-
-- `discoverLocalCdpBrowsers({ timeoutMs? })`
-- `inspectCdpEndpoint({ endpoint, headers?, timeoutMs? })`
-- `browser.status()`
-- `browser.clone({ sourceUserDataDir, sourceProfileDirectory? })`
-- `browser.reset()`
-- `browser.delete()`
-
-Lifecycle:
-
-- `close()`
-- `disconnect()`
-
-## Rules
-
-- Wrap long-running browser ownership in `try/finally` and call `close()`.
-- Use `captureNetwork` on actions that trigger requests you may inspect later.
-- Use `description` for all interactions and extractions in deterministic scripts.
-- Use `description` plus `schema` to persist an extraction descriptor. Bare `description` replays it.
-- Use `element` targets only during CLI exploration with a fresh snapshot. Deterministic scripts use `description`.
-- The SDK does expose `snapshot()`, but this workflow keeps element discovery in the CLI with `snapshot action`.
-- Prefer Opensteer methods over raw Playwright so browser, extraction, and replay semantics stay consistent.
+- Do not build new abstractions on top of simple `fetch()` code unless the task really needs them.
+- Do not bypass Opensteer with raw Playwright when Opensteer already captured the request.
+- Do not dump giant raw response blobs into logs or prompts when the filtered previews already show the useful structure.