opensteer 0.6.13 → 0.7.1

package/README.md

@@ -1,208 +1,152 @@
-# Opensteer
+# Opensteer Package
 
-Browser automation framework for AI agents to explore websites and build complex scrapers directly in your codebase.
+`opensteer` is the main product surface for the repository. It exposes:
 
-Opensteer enables AI agents like Claude Code and Codex to interact with browsers and Electron applications, building scrapers directly in your local codebase. It provides a token-efficient suite of tools and agent skills designed for seamless integration.
+- a semantic SDK with session continuity inside one process
+- a thin JSON-first CLI
+- a local per-session service so CLI commands can share one live browser session
+- browser-native observation and instrumentation primitives
+- deterministic replay through request plans, recipes, and saved evidence
+- HTML-first snapshots, DOM action replay, computer-use actions, descriptor persistence, traces,
+  and artifacts
+
+The package is organized around three lanes:
+
+- `Interact`: open pages, navigate, evaluate, inspect DOM state, manage pages, use computer actions
+- `Observe / Instrument`: capture network, capture scripts, add init scripts, route requests, replace scripts
+- `Replay / Execute`: `direct-http`, `context-http`, `page-http`, reverse workflows, request plans, and recipes
 
 ## Install
 
-Main setup (recommended):
+CLI:
 
 ```bash
 npm i -g opensteer
 opensteer skills install
 ```
 
-SDK package (when importing `Opensteer` in app code):
+SDK:
 
 ```bash
+pnpm add opensteer
+pnpm exec playwright install chromium
+
 # npm
 npm install opensteer
 
-# pnpm
-pnpm add opensteer
-
-# bun
-bun add opensteer
+# Optional ABP backend for `--engine abp`
+pnpm add @opensteer/engine-abp
 ```
 
-## Requirements
-
-- Node.js `>=20`
-- Playwright-supported browser runtime
-- Model provider API key for LLM-powered resolution/extraction/CUA
-
-If browser binaries are missing:
-
-```bash
-npx playwright install chromium
-```
+`opensteer skills install` installs the packaged first-party skill pack through the upstream
+`skills` CLI.
 
-## What It Does
+`opensteer` installs the Playwright-backed local engine by default. Add
+`@opensteer/engine-abp` only when you need the ABP backend.
 
-- Unified local/cloud execution with the same API surface
-- Descriptor-aware actions with selector persistence for replay
-- Structured extraction with typed schemas
-- CUA agent support (`openai`, `anthropic`, `google`)
-- Stealth cursor preview for interactive actions (CLI default on, SDK default off)
+Cloud features require access to an Opensteer Cloud deployment. This repository includes cloud
+client code and shared contracts; the managed Opensteer Cloud service is operated separately.
 
-## Quick Start: SDK
+## SDK
 
 ```ts
 import { Opensteer } from "opensteer";
 
-const opensteer = new Opensteer({ name: "quickstart" });
+const opensteer = new Opensteer({
+  name: "docs-example",
+  rootDir: process.cwd(),
+  browser: { headless: true },
+});
 
 try {
-  await opensteer.launch();
-  await opensteer.goto("https://example.com");
-
-  await opensteer.snapshot({ mode: "action" });
-  await opensteer.click({ description: "main call to action" });
-
-  await opensteer.snapshot({ mode: "extraction" });
-  const data = await opensteer.extract({
-    description: "hero section",
-    schema: { title: "string", href: "string" },
+  await opensteer.open({
+    url: "https://example.com",
+    browser: {
+      headless: true,
+    },
+  });
+  const snapshot = await opensteer.snapshot("action");
+  const firstButton = snapshot.counters.find((counter) => counter.tagName === "BUTTON");
+  if (firstButton) {
+    await opensteer.click({
+      element: firstButton.element,
+      description: "primary button",
+    });
+  }
+
+  const extracted = await opensteer.extract({
+    description: "page summary",
+    schema: {
+      url: { source: "current_url" },
+      title: { selector: "title" },
+    },
   });
 
-  console.log(data);
+  console.log(snapshot.html);
+  console.log(extracted);
 } finally {
   await opensteer.close();
 }
 ```
 
-Enable cursor preview in SDK:
-
-```ts
-const opensteer = new Opensteer({
-  name: "quickstart",
-  cursor: { enabled: true },
-});
-```
-
-## Quick Start: CUA Agent
+Browser-backed replay:
 
 ```ts
 import { Opensteer } from "opensteer";
 
-const opensteer = new Opensteer({ model: "openai/computer-use-preview" });
+const opensteer = new Opensteer({
+  name: "browser-backed-replay",
+  rootDir: process.cwd(),
+  browser: { headless: true },
+});
 
 try {
-  await opensteer.launch();
-  const agent = opensteer.agent({ mode: "cua" });
-  const result = await agent.execute({
-    instruction: "Go to Hacker News and open the top story.",
-    maxSteps: 20,
+  await opensteer.open("https://example.com/app");
+  const token = await opensteer.evaluate<string>({
+    script: "() => window.exampleToken",
   });
-  console.log(result.message);
+
+  const response = await opensteer.rawRequest({
+    transport: "context-http",
+    url: "https://example.com/api/items",
+    method: "POST",
+    body: {
+      json: { token },
+    },
+  });
+
+  console.log(response.data);
 } finally {
   await opensteer.close();
 }
 ```
 
-## Quick Start: CLI
-
-```bash
-# Open a browser session and bind a selector namespace
-opensteer open https://example.com --session demo --name quickstart
-
-# Action snapshot + interaction
-opensteer snapshot action --session demo
-opensteer click --description "main call to action" --session demo
-
-# Cursor controls
-opensteer cursor status --session demo
-opensteer cursor off --session demo
-
-# Extraction snapshot + structured extract
-opensteer snapshot extraction --session demo
-opensteer extract '{"title":"string","href":"string"}' --description "hero section" --session demo
-
-# Close session
-opensteer close --session demo
-```
-
-For non-interactive runs, set `OPENSTEER_SESSION` or `OPENSTEER_CLIENT_ID`.
-Runtime daemon routing for `OPENSTEER_SESSION` is scoped by canonical `cwd`
-(`realpath(cwd)`) + logical session id.
-
-Cursor defaults:
-
-- CLI sessions: enabled by default (toggle with `--cursor` or `opensteer cursor on|off`)
-- SDK instances: disabled by default (set `cursor.enabled: true` to opt in)
-
-## For AI Agents
-
-Use this workflow to keep scripts replayable and maintainable:
-
-1. Use Opensteer APIs (`goto`, `snapshot`, `click`, `input`, `extract`) instead of raw Playwright calls.
-2. Keep namespace consistent: SDK `name` must match CLI `--name`.
-3. Take `snapshot({ mode: "action" })` before actions and `snapshot({ mode: "extraction" })` before extraction.
-4. Prefer `description` targeting for persistence and deterministic reruns.
-5. Always wrap runs in `try/finally` and call `close()`.
+Attach to an existing CLI-opened local session:
 
-First-party skills:
-
-- [skills/opensteer/SKILL.md](skills/opensteer/SKILL.md)
-- [skills/electron/SKILL.md](skills/electron/SKILL.md)
-- [skills/README.md](skills/README.md)
-
-Install the Opensteer skill pack:
-
-```bash
-opensteer skills install
-```
-
-Claude Code marketplace plugin:
-
-```text
-/plugin marketplace add steerlabs/opensteer
-/plugin install opensteer@opensteer-marketplace
-```
-
-## Cloud Mode
-
-Opensteer defaults to local mode. Enable cloud mode with env or constructor options:
-
-```bash
-OPENSTEER_MODE=cloud
-OPENSTEER_API_KEY=<your_api_key>
-```
+```ts
+import { Opensteer } from "opensteer";
 
-- Interactive CLI login:
+const opensteer = Opensteer.attach({
+  name: "docs-example",
+  rootDir: process.cwd(),
+});
 
-```bash
-opensteer auth login
-opensteer auth status
-opensteer auth logout
+try {
+  const state = await opensteer.open();
+  console.log(state.url);
+} finally {
+  await opensteer.disconnect();
+}
 ```
 
-- `opensteer auth login` opens your default browser when possible. Use
-  `--no-browser` on remote shells, containers, or CI and paste the printed URL
-  into a browser manually. In `--json` mode, login prompts go to stderr and the
-  final JSON result stays on stdout.
-- Saved machine logins remain scoped per resolved cloud API host (`baseUrl`).
-  The CLI also remembers the last selected cloud host, so `opensteer auth
-  status`, `opensteer auth logout`, and other cloud commands reuse it by
-  default unless `--base-url` or env vars select a different host.
-
-- `OPENSTEER_BASE_URL` overrides the default cloud host
-- `OPENSTEER_ACCESS_TOKEN` provides bearer auth for cloud commands
-- `OPENSTEER_AUTH_SCHEME` supports `api-key` (default) or `bearer`
-- Credential precedence: explicit flags > environment variables > saved machine login
-- `OPENSTEER_CLOUD_PROFILE_ID` optionally launches into a specific cloud browser profile
-- `OPENSTEER_CLOUD_PROFILE_REUSE_IF_ACTIVE` (`true|false`) optionally reuses an active profile session
-- `cloud: true` or a `cloud` options object overrides `OPENSTEER_MODE`
-- Cloud mode is fail-fast (no automatic fallback to local)
-- `Opensteer.from(page)`, `uploadFile`, `exportCookies`, and `importCookies` are local-only
-
-Select a cloud browser profile in SDK:
+Launch a cloud session with a specific browser profile:
 
 ```ts
+import { Opensteer } from "opensteer";
+
 const opensteer = new Opensteer({
   cloud: {
-    accessToken: process.env.OPENSTEER_ACCESS_TOKEN,
+    apiKey: process.env.OPENSTEER_API_KEY!,
     browserProfile: {
       profileId: "bp_123",
       reuseIfActive: true,
@@ -211,56 +155,184 @@ const opensteer = new Opensteer({
 });
 ```
 
-Sync local profile cookies into a cloud profile:
+Sync cookies from a live Chromium browser into an existing cloud profile:
+
+```ts
+import { OpensteerCloudClient } from "opensteer";
+
+const client = new OpensteerCloudClient({
+  apiKey: process.env.OPENSTEER_API_KEY!,
+  baseUrl: process.env.OPENSTEER_BASE_URL ?? "https://api.opensteer.dev",
+});
+
+await client.syncBrowserProfileCookies({
+  profileId: "bp_123",
+  attachEndpoint: "9222",
+  domains: ["github.com"],
+});
+```
+
+## CLI
 
 ```bash
+opensteer open https://example.com --name docs-example --headless true
+opensteer open https://example.com --name docs-example --browser attach-live --attach-endpoint 9222
+opensteer open https://example.com --name docs-example --browser snapshot-session \
+  --source-user-data-dir "~/Library/Application Support/Google/Chrome" \
+  --source-profile-directory Default
+opensteer open https://example.com --name docs-example --browser snapshot-authenticated \
+  --source-user-data-dir "~/Library/Application Support/Google/Chrome" \
+  --source-profile-directory "Profile 1"
+opensteer browser discover
+opensteer browser inspect --endpoint 9222
+opensteer local-profile list
+opensteer local-profile inspect --user-data-dir "~/Library/Application Support/Opensteer Chrome"
+opensteer local-profile unlock --user-data-dir "~/Library/Application Support/Opensteer Chrome"
 opensteer profile sync \
-  --from-profile-dir ~/Library/Application\ Support/Google/Chrome/Default \
-  --to-profile-id bp_123 \
+  --profile-id bp_123 \
+  --attach-endpoint 9222 \
   --domain github.com
+opensteer open https://example.com --name docs-example --engine abp
+opensteer snapshot action --name docs-example
+opensteer click 3 --name docs-example --description "primary button"
+opensteer extract --name docs-example --description "page summary" \
+  --schema '{"url":{"source":"current_url"},"title":{"selector":"title"}}'
+opensteer computer '{"type":"screenshot"}' --name docs-example
+opensteer close --name docs-example
 ```
 
-## Resolution and Replay
+CLI long flags are canonical kebab-case, for example `--root-dir`, `--network-tag`, and
+`--press-enter`. Unknown flags and flags used on the wrong command fail fast instead of being
+silently ignored.
 
-For descriptor-aware actions (`click`, `input`, `hover`, `select`, `scroll`):
+Action and data commands print JSON to stdout. Help commands print human-readable usage text.
+Browser state does not live in the CLI process. It lives
+in the local session service recorded under `.opensteer/runtime/sessions/<name>/service.json`.
+`opensteer computer` prints compact screenshot metadata and points at the persisted image through
+`screenshot.path` for local shells and `screenshot.payload.uri` for the canonical file-backed
+location.
 
-1. Reuse persisted selector path from `description`
-2. Try snapshot counter (`element`)
-3. Try explicit CSS selector (`selector`)
-4. Use LLM resolution (`description` required)
-5. Return actionable error
+Use `--engine <playwright|abp>` on `open` to choose the backend for a new session. You can also
+set `OPENSTEER_ENGINE=abp` to change the default engine for `open` in the current shell. Engine
+selection is fixed when the session service starts, so `OPENSTEER_ENGINE` and `--engine` only
+affect `open`, not commands like `snapshot` or `click` that attach to an existing session.
+When using `--engine abp`, Opensteer accepts the ABP launch options it can actually honor:
+`--headless`, `--executable-path`, and equivalent `--browser-json` fields for `headless`, `args`,
+and `executablePath`. Unsupported shared browser/context options fail fast instead of being
+ignored.
 
-When step 2-4 succeeds and `description` is present, selector paths are cached
-in `.opensteer/selectors/<namespace>` for deterministic replay.
+Use `opensteer local-profile inspect` to diagnose whether a live Chromium profile is safe to reuse
+as a snapshot source or whether it should be attached via CDP instead. `opensteer local-profile unlock`
+remains limited to explicit `stale_lock` recovery; Opensteer does not mutate or take ownership of
+real user-data-dirs as part of `open`.
 
-## Docs
+## Connect To Real Browser
 
-- [Getting Started](docs/getting-started.md)
-- [API Reference](docs/api-reference.md)
-- [CLI Reference](docs/cli-reference.md)
-- [Cloud Integration](docs/cloud-integration.md)
-- [Selectors and Storage](docs/selectors.md)
-- [HTML Cleaning and Snapshot Modes](docs/html-cleaning.md)
-- [Live Web Validation Suite](docs/live-web-tests.md)
-- [Skills](docs/skills.md)
+- `managed` launches a fresh isolated Chrome/Chromium process with a temporary `user-data-dir`.
+- `snapshot-session` copies a source profile into a temporary owned browser directory without full authenticated OS-integrated state. Use it when persisted cookies/storage are enough.
+- `snapshot-authenticated` copies a source profile into a temporary owned browser directory and preserves the authenticated browser state needed for harder replay cases.
+- `attach-live` connects to an already-running Chrome/Chromium instance. Pass `endpoint` for an explicit CDP target, or omit it to auto-discover a locally attachable browser.
 
-## Development
+When you are launching a browser yourself for `attach-live`, prefer a dedicated profile directory:
 
 ```bash
-pnpm install
-pnpm typecheck
-pnpm test
-pnpm build
+/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
+  --user-data-dir="$HOME/Library/Application Support/Opensteer Chrome" \
+  --remote-debugging-port=9222
 ```
 
-## Community
+When attaching to an existing browser, Opensteer may land on an already-open tab such as `chrome://newtab`. Pass `--fresh-tab` when you want a clean working page immediately after attach.
 
-- [Contributing](CONTRIBUTING.md)
-- [Code of Conduct](CODE_OF_CONDUCT.md)
-- [Security Policy](SECURITY.md)
-- [Discussions](https://github.com/steerlabs/opensteer/discussions)
-- [Changelog](CHANGELOG.md)
+## Transport Guide
 
-## License
+- `direct-http`: use when the request is replayable without a browser
+- `context-http`: use when browser cookies or browser session state are required
+- `page-http`: use when request execution must happen inside the live page JavaScript world
+
+`goto()` plus `waitForNetwork()` is a separate pattern. It is how you observe the
+page's own traffic; it is not a transport.
+
+## Session Root
+
+By default, Opensteer writes into:
+
+```text
+<cwd>/.opensteer
+```
+
+Important subtrees:
+
+```text
+.opensteer/
+  artifacts/
+  traces/
+  registry/
+  runtime/
+    sessions/
+```
 
-[MIT](LICENSE)
+## Public Methods
+
+- `Opensteer.attach({ name?, rootDir? })`
+- `discoverLocalCdpBrowsers({ timeoutMs? })`
+- `inspectCdpEndpoint({ endpoint, headers?, timeoutMs? })`
+- `inspectLocalBrowserProfile({ userDataDir? })`
+- `unlockLocalBrowserProfile({ userDataDir })`
+- `open(url | { url?, name?, browser?, context? })`
+- `goto(url | { url, networkTag? })`
+- `evaluate(script | { script, pageRef?, args? })`
+- `evaluateJson({ script, pageRef?, args? })`
+- `waitForNetwork({ ...filters, pageRef?, includeBodies?, timeoutMs? })`
+- `waitForResponse({ ...filters, pageRef?, includeBodies?, timeoutMs? })`
+- `listPages()`
+- `newPage({ url?, openerPageRef? })`
+- `activatePage({ pageRef })`
+- `closePage({ pageRef })`
+- `waitForPage({ openerPageRef?, urlIncludes?, timeoutMs? })`
+- `snapshot("action" | "extraction")`
+- `click({ element | selector | description, networkTag? })`
+- `hover({ element | selector | description, networkTag? })`
+- `input({ element | selector | description, text, networkTag? })`
+- `scroll({ element | selector | description, direction, amount, networkTag? })`
+- `extract({ description, schema? })`
+- `queryNetwork({ source?, recordId?, requestId?, actionId?, tag?, url?, hostname?, path?, method?, status?, resourceType?, pageRef?, includeBodies?, limit? })`
+- `saveNetwork({ tag, ...filters })`
+- `clearNetwork({ tag? })`
+- `captureScripts({ pageRef?, includeInline?, includeExternal?, includeDynamic?, includeWorkers?, urlFilter?, persist? })`
+- `addInitScript({ script, args?, pageRef? })`
+- `route({ urlPattern, resourceTypes?, times?, handler })`
+- `interceptScript({ urlPattern, handler, times? })`
+- `rawRequest({ transport?, pageRef?, url, method?, headers?, body?, followRedirects? })`
+- `inferRequestPlan({ recordId, key, version, lifecycle? })`
+- `writeRequestPlan({ key, version, payload, lifecycle?, tags?, provenance?, freshness? })`
+- `getRequestPlan({ key, version? })`
+- `listRequestPlans({ key? })`
+- `writeRecipe({ key, version, payload, tags?, provenance? })`
+- `getRecipe({ key, version? })`
+- `listRecipes({ key? })`
+- `runRecipe({ key, version?, input? })`
+- `request(key, { path?, query?, headers?, body? })`
+- `computerExecute({ action, screenshot?, networkTag? })`
+- `disconnect()`
+- `close()`
+
+`element` targets use counters from the latest snapshot. `description` replays a stored descriptor.
+`selector` resolves a CSS selector directly and, when not explicitly scoped, searches the current
+page before falling back to child frames.
+
+Use `disconnect()` for attached sessions when you want to release the SDK handle but keep the
+underlying session alive. Use `close()` when you want to destructively end the session.
+
+Profile inspection is independent from session ownership. `inspectLocalBrowserProfile()` returns a
+structured status union (`available`, `unsupported_default_user_data_dir`, `opensteer_owned`,
+`browser_owned`, `stale_lock`) that launch, CLI, and SDK all consume. Failed owned launches throw
+`OpensteerLocalProfileUnavailableError` with the inspection attached for programmatic handling.
+
+The reverse-engineering workflow is: perform a browser action, inspect traffic with
+`queryNetwork()`, optionally instrument with `addInitScript()`, `route()`, or
+`captureScripts()`, experiment with `rawRequest()`, promote a record with
+`inferRequestPlan()`, then replay with `request()`.
+
+`route()` and `interceptScript()` are only available on owned local SDK sessions.
+They are not available on attached or cloud proxy sessions because they rely on a
+live in-process route handler.