opensteer 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+## Unreleased
+
+- Breaking: removed legacy `ai` config from `OpensteerConfig`; use top-level `model` instead.
+- Breaking: `OPENSTEER_AI_MODEL` is no longer supported; use `OPENSTEER_MODEL`.
+- Breaking: `OPENSTEER_RUNTIME` is no longer supported; use `OPENSTEER_MODE`.
+- Breaking: mode selection now uses `mode: 'local' | 'remote'` and remote credentials use `remote.apiKey`.
+- Opensteer now enables built-in LLM resolve/extract by default with model `gpt-5.1`.
+- Remote mode now falls back to `OPENSTEER_API_KEY` when `remote.apiKey` is omitted.
+- Mutating actions now include smart best-effort post-action wait with per-action
+  profiles and optional per-call overrides via `wait`.
+- Added structured interaction diagnostics via `OpensteerActionError` for
+  descriptor-aware interaction methods (`click`, `dblclick`, `rightclick`,
+  `hover`, `input`, `select`, `scroll`, `uploadFile`).
+- Added `ActionFailure` types (`ActionFailureCode`, `retryable`,
+  `classificationSource`, optional `details`) to support programmatic handling
+  of action failures.
+- Added DOM actionability probe + Playwright call-log classification to report
+  reasons like `BLOCKED_BY_INTERCEPTOR`, `NOT_VISIBLE`, `NOT_EDITABLE`, and
+  timeout/stale-target cases more accurately.
+- Remote action failures now accept optional structured failure details and map
+  them to `OpensteerActionError` when available.
+
+## 0.1.0
+
+- Initial open-source release.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 TrendUp AI, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,156 @@
+# Opensteer
+
+Lean browser automation SDK for coding agents and script replay.
+
+`opensteer` provides descriptor-aware actions (`click`, `dblclick`,
+`rightclick`, `hover`, `input`, `select`, `scroll`, `extract`,
+`extractFromPlan`, `uploadFile`), observation (`snapshot`, `state`,
+`screenshot`), navigation (`goto`), and convenience methods for tabs, cookies,
+keyboard, element info, and wait.
+
+For anything not covered, use raw Playwright via `opensteer.page` and
+`opensteer.context`.
+
+## Install
+
+```bash
+# npm
+npm install opensteer playwright
+# pnpm
+pnpm add opensteer playwright
+```
+
+## Quickstart
+
+```ts
+import { Opensteer } from "opensteer";
+
+const opensteer = new Opensteer({ name: "my-scraper" }); // defaults to model: 'gpt-5.1'
+await opensteer.launch({ headless: false });
+
+await opensteer.goto("https://example.com");
+const html = await opensteer.snapshot();
+
+await opensteer.click({ description: "login-button" });
+await opensteer.input({ description: "email", text: "user@example.com" });
+await opensteer.page.keyboard.press("Enter");
+
+await opensteer.close();
+```
+
+## Core Model
+
+- `opensteer.page`: raw Playwright `Page`
+- `opensteer.context`: raw Playwright `BrowserContext`
+- Opensteer methods: descriptor-aware operations that can persist selectors
+- Selector storage: `.opensteer/selectors/<namespace>`
+
+## Resolution Chain
+
+For actions like `click`/`input`/`hover`/`select`/`scroll`:
+
+1. Use persisted path for `description` (if present)
+2. Use `element` counter from snapshot
+3. Use explicit CSS `selector`
+4. Use built-in LLM resolution (`description` required)
+5. Throw
+
+When steps 2-4 resolve and `description` is provided, the path is persisted.
+
+## Smart Post-Action Wait
+
+Mutating actions (`click`, `input`, `select`, `scroll`, etc.) include a
+best-effort post-action wait so delayed visual updates are usually settled
+before the method resolves.
+
+You can disable or tune this per call:
+
+```ts
+await opensteer.click({ description: "Save button", wait: false });
+
+await opensteer.click({
+  description: "Save button",
+  wait: { timeout: 9000, settleMs: 900, includeNetwork: true, networkQuietMs: 400 },
+});
+```
+
+## Action Failure Diagnostics
+
+Descriptor-aware interaction methods (`click`, `dblclick`, `rightclick`,
+`hover`, `input`, `select`, `scroll`, `uploadFile`) throw
+`OpensteerActionError` when an interaction cannot be completed.
+
+The error includes structured failure metadata for agent/tooling decisions:
+
+- `error.failure.code` (`ActionFailureCode`)
+- `error.failure.message`
+- `error.failure.retryable`
+- `error.failure.classificationSource`
+- `error.failure.details` (for blocker and observation details when available)
+
+```ts
+import { Opensteer, OpensteerActionError } from "opensteer";
+
+try {
+  await opensteer.click({ description: "Save button" });
+} catch (err) {
+  if (err instanceof OpensteerActionError) {
+    console.error(err.failure.code); // e.g. BLOCKED_BY_INTERCEPTOR
+    console.error(err.failure.message);
+    console.error(err.failure.classificationSource);
+  }
+  throw err;
+}
+```
+
+## Snapshot Modes
+
+```ts
+await opensteer.snapshot(); // action mode (default)
+await opensteer.snapshot({ mode: "extraction" });
+await opensteer.snapshot({ mode: "clickable" });
+await opensteer.snapshot({ mode: "scrollable" });
+await opensteer.snapshot({ mode: "full" });
+```
+
+## Two Usage Patterns
+
+### Explore (coding agent, no API key required)
+
+Use `snapshot()` + `element` counters while exploring in real time, then persist
+stable descriptions for replay.
+
+### Run (script replay / built-in LLM)
+
+Opensteer uses built-in LLM resolve/extract by default. You can override the
+default model with top-level `model` or `OPENSTEER_MODEL`.
+
+```ts
+const opensteer = new Opensteer({
+  name: "run-mode",
+  model: "gpt-5-mini",
+});
+```
+
+## Mode Selection
+
+Opensteer defaults to local mode.
+
+- `OPENSTEER_MODE=local` runs local Playwright.
+- `OPENSTEER_MODE=remote` runs remote mode (requires `OPENSTEER_API_KEY`).
+- `mode: "remote"` in constructor config always forces remote mode.
+
+Remote mode is fail-fast: it does not automatically fall back to local mode.
+
+## Docs
+
+- `docs/getting-started.md`
+- `docs/api-reference.md`
+- `docs/remote-integration.md`
+- `docs/html-cleaning.md`
+- `docs/selectors.md`
+- `docs/live-web-tests.md`
+
+## License
+
+MIT

package/bin/opensteer.mjs

@@ -0,0 +1,388 @@
+#!/usr/bin/env node
+
+import { connect } from 'net'
+import { spawn } from 'child_process'
+import { existsSync, readFileSync, unlinkSync, mkdirSync } from 'fs'
+import { join, dirname } from 'path'
+import { homedir } from 'os'
+import { fileURLToPath } from 'url'
+
+const __dirname = dirname(fileURLToPath(import.meta.url))
+
+const RUNTIME_DIR = join(homedir(), '.opensteer')
+const SOCKET_PATH = join(RUNTIME_DIR, 'opensteer.sock')
+const PID_PATH = join(RUNTIME_DIR, 'opensteer.pid')
+const SERVER_SCRIPT = join(__dirname, '..', 'dist', 'cli', 'server.js')
+
+const CONNECT_TIMEOUT = 15000
+const POLL_INTERVAL = 100
+const RESPONSE_TIMEOUT = 120000
+
+function parseArgs(argv) {
+    const args = argv.slice(2)
+    if (args.length === 0 || args[0] === '--help' || args[0] === '-h') {
+        printHelp()
+        process.exit(0)
+    }
+
+    const command = args[0]
+    const flags = {}
+    const positional = []
+
+    for (let i = 1; i < args.length; i++) {
+        const arg = args[i]
+        if (arg.startsWith('--')) {
+            const key = arg.slice(2)
+            const next = args[i + 1]
+            if (next !== undefined && !next.startsWith('--')) {
+                flags[key] = parseValue(next)
+                i++
+            } else {
+                flags[key] = true
+            }
+        } else {
+            positional.push(arg)
+        }
+    }
+
+    return { command, flags, positional }
+}
+
+function parseValue(str) {
+    if (str === 'true') return true
+    if (str === 'false') return false
+    const num = Number(str)
+    if (!Number.isNaN(num) && str.trim() !== '') return num
+    return str
+}
+
+function buildRequest(command, flags, positional) {
+    const id = 1
+    const globalFlags = {}
+    for (const key of ['name', 'headless', 'json', 'connect-url', 'channel', 'profile-dir']) {
+        if (key in flags) {
+            globalFlags[key] = flags[key]
+            delete flags[key]
+        }
+    }
+
+    const args = { ...globalFlags, ...flags }
+
+    switch (command) {
+        case 'open':
+        case 'navigate':
+            args.url = positional[0] || args.url
+            break
+
+        case 'click':
+        case 'dblclick':
+        case 'rightclick':
+        case 'hover':
+        case 'select':
+        case 'scroll':
+        case 'get-text':
+        case 'get-value':
+        case 'get-attrs':
+            if (positional[0] !== undefined && args.element === undefined) {
+                args.element = Number(positional[0])
+            }
+            break
+
+        case 'input': {
+            // input 12 "text" or input "text" --element 12
+            if (positional.length >= 2) {
+                const first = Number(positional[0])
+                if (!Number.isNaN(first)) {
+                    args.element = args.element ?? first
+                    args.text = args.text ?? positional[1]
+                } else {
+                    args.text = args.text ?? positional[0]
+                    args.element = args.element ?? Number(positional[1])
+                }
+            } else if (positional.length === 1) {
+                args.text = args.text ?? positional[0]
+            }
+            break
+        }
+
+        case 'press':
+            args.key = positional[0] || args.key
+            break
+
+        case 'type':
+            args.text = positional[0] || args.text
+            break
+
+        case 'get-html':
+            if (positional[0] && !args.selector) {
+                args.selector = positional[0]
+            }
+            break
+
+        case 'tab-new':
+            args.url = positional[0] || args.url
+            break
+
+        case 'tab-switch':
+        case 'tab-close':
+            args.index =
+                positional[0] !== undefined ? Number(positional[0]) : args.index
+            break
+
+        case 'cookies-export':
+        case 'cookies-import':
+        case 'screenshot':
+            args.file = positional[0] || args.file
+            break
+
+        case 'eval':
+            args.expression = positional[0] || args.expression
+            break
+
+        case 'wait-for':
+            args.text = positional[0] || args.text
+            break
+
+        case 'wait-selector':
+            args.selector = positional[0] || args.selector
+            break
+
+        case 'extract':
+            if (positional[0] && !args.schema) {
+                try {
+                    args.schema = JSON.parse(positional[0])
+                } catch {
+                    error(`Invalid JSON schema: ${positional[0]}`)
+                }
+            }
+            break
+
+        case 'snapshot':
+            args.mode = positional[0] || args.mode
+            break
+    }
+
+    return { id, command, args }
+}
+
+function isServerRunning() {
+    if (!existsSync(PID_PATH)) return false
+    try {
+        const pid = parseInt(readFileSync(PID_PATH, 'utf-8').trim(), 10)
+        process.kill(pid, 0)
+        return true
+    } catch {
+        cleanStaleFiles()
+        return false
+    }
+}
+
+function cleanStaleFiles() {
+    try {
+        unlinkSync(SOCKET_PATH)
+    } catch { }
+    try {
+        unlinkSync(PID_PATH)
+    } catch { }
+}
+
+function startServer() {
+    mkdirSync(RUNTIME_DIR, { recursive: true })
+
+    const child = spawn('node', [SERVER_SCRIPT], {
+        detached: true,
+        stdio: ['ignore', 'ignore', 'ignore'],
+    })
+    child.unref()
+}
+
+function waitForSocket(timeout) {
+    return new Promise((resolve, reject) => {
+        const start = Date.now()
+
+        function poll() {
+            if (Date.now() - start > timeout) {
+                reject(new Error('Timed out waiting for server to start'))
+                return
+            }
+
+            if (existsSync(SOCKET_PATH)) {
+                resolve()
+                return
+            }
+
+            setTimeout(poll, POLL_INTERVAL)
+        }
+
+        poll()
+    })
+}
+
+function sendCommand(request) {
+    return new Promise((resolve, reject) => {
+        const socket = connect(SOCKET_PATH)
+        let buffer = ''
+        let settled = false
+
+        const timer = setTimeout(() => {
+            if (!settled) {
+                settled = true
+                socket.destroy()
+                reject(new Error('Response timeout'))
+            }
+        }, RESPONSE_TIMEOUT)
+
+        socket.on('connect', () => {
+            socket.write(JSON.stringify(request) + '\n')
+        })
+
+        socket.on('data', (chunk) => {
+            buffer += chunk.toString()
+            const idx = buffer.indexOf('\n')
+            if (idx !== -1) {
+                const line = buffer.slice(0, idx)
+                clearTimeout(timer)
+                settled = true
+                socket.end()
+                try {
+                    resolve(JSON.parse(line))
+                } catch {
+                    reject(new Error('Invalid JSON response from server'))
+                }
+            }
+        })
+
+        socket.on('error', (err) => {
+            if (!settled) {
+                clearTimeout(timer)
+                settled = true
+                reject(err)
+            }
+        })
+
+        socket.on('close', () => {
+            if (!settled) {
+                clearTimeout(timer)
+                settled = true
+                reject(new Error('Connection closed before response'))
+            }
+        })
+    })
+}
+
+function output(data) {
+    process.stdout.write(JSON.stringify(data) + '\n')
+}
+
+function error(msg) {
+    process.stderr.write(JSON.stringify({ ok: false, error: msg }) + '\n')
+    process.exit(1)
+}
+
+function printHelp() {
+    console.log(`Usage: opensteer <command> [options]
+
+Navigation:
+  open <url>                Open browser and navigate to URL
+  navigate <url>            Navigate and wait for visual stability
+  back                      Go back
+  forward                   Go forward
+  reload                    Reload page
+  close                     Close browser and server
+
+Observation:
+  snapshot [--mode action]  Get page snapshot
+  state                     Get page URL, title, and snapshot
+  screenshot [file]         Take screenshot
+
+Actions:
+  click [element]           Click element
+  dblclick [element]        Double-click element
+  rightclick [element]      Right-click element
+  hover [element]           Hover over element
+  input [element] <text>    Input text into element
+  select [element]          Select option from dropdown
+  scroll [element]          Scroll page or element
+
+Keyboard:
+  press <key>               Press key
+  type <text>               Type text into focused element
+
+Element Info:
+  get-text [element]        Get element text
+  get-value [element]       Get element value
+  get-attrs [element]       Get element attributes
+  get-html [selector]       Get page or element HTML
+
+Tabs:
+  tabs                      List tabs
+  tab-new [url]             Open new tab
+  tab-switch <index>        Switch to tab
+  tab-close [index]         Close tab
+
+Cookies:
+  cookies [--url]           Get cookies
+  cookie-set                Set cookie (--name, --value, ...)
+  cookies-clear             Clear all cookies
+  cookies-export <file>     Export cookies to file
+  cookies-import <file>     Import cookies from file
+
+Utility:
+  eval <expression>         Evaluate JavaScript
+  wait-for <text>           Wait for text to appear
+  wait-selector <selector>  Wait for selector
+  extract <schema-json>     Extract structured data
+
+Global Flags:
+  --name <namespace>        Storage namespace (default: "cli")
+  --headless                Launch browser in headless mode
+  --connect-url <url>       Connect to a running browser (e.g. http://localhost:9222)
+  --channel <browser>       Use installed browser (chrome, chrome-beta, msedge)
+  --profile-dir <path>      Browser profile directory for logged-in sessions
+  --element <N>             Target element by counter
+  --selector <css>          Target element by CSS selector
+  --description <text>      Description for selector persistence
+  --help                    Show this help
+
+Environment:
+  OPENSTEER_MODE            Runtime mode: "local" (default) or "remote"
+  OPENSTEER_API_KEY         Required when remote mode is selected
+  OPENSTEER_BASE_URL        Override remote control-plane base URL
+`)
+}
+
+async function main() {
+    const { command, flags, positional } = parseArgs(process.argv)
+    const request = buildRequest(command, flags, positional)
+
+    if (!isServerRunning()) {
+        if (!existsSync(SERVER_SCRIPT)) {
+            error(
+                `Server script not found: ${SERVER_SCRIPT}. Run the build script first.`
+            )
+        }
+        startServer()
+        try {
+            await waitForSocket(CONNECT_TIMEOUT)
+        } catch {
+            error('Failed to start server. Check that the build is complete.')
+        }
+    }
+
+    try {
+        const response = await sendCommand(request)
+
+        if (response.ok) {
+            output({ ok: true, ...response.result })
+        } else {
+            process.stderr.write(
+                JSON.stringify({ ok: false, error: response.error }) + '\n'
+            )
+            process.exit(1)
+        }
+    } catch (err) {
+        error(err.message || 'Connection failed')
+    }
+}
+
+main()

package/dist/chunk-3H5RRIMZ.js

@@ -0,0 +1,69 @@
+// src/extract-field-plan.ts
+var CURRENT_URL_SENTINEL = "CURRENT_URL";
+var COUNTER_KEY = "$c";
+function flattenExtractionDataToFieldPlan(data) {
+  const fields = {};
+  flattenExtractionDataToFieldPlanRecursive(data, "", fields);
+  return fields;
+}
+function flattenExtractionDataToFieldPlanRecursive(value, prefix, out) {
+  if (value == null) return;
+  if (typeof value === "number" && Number.isFinite(value)) {
+    const key = String(prefix || "").trim();
+    if (!key) return;
+    out[key] = { element: Math.trunc(value) };
+    return;
+  }
+  if (typeof value === "string" && value.trim().toUpperCase() === CURRENT_URL_SENTINEL) {
+    const key = String(prefix || "").trim();
+    if (!key) return;
+    out[key] = { source: "current_url" };
+    return;
+  }
+  const counterLeaf = parseCounterLeafDescriptor(value);
+  if (counterLeaf) {
+    const key = String(prefix || "").trim();
+    if (!key) return;
+    out[key] = counterLeaf;
+    return;
+  }
+  if (Array.isArray(value)) {
+    for (let i = 0; i < value.length; i++) {
+      const nextPrefix = prefix ? `${prefix}[${i}]` : `[${i}]`;
+      flattenExtractionDataToFieldPlanRecursive(value[i], nextPrefix, out);
+    }
+    return;
+  }
+  if (typeof value !== "object") return;
+  for (const [key, child] of Object.entries(
+    value
+  )) {
+    const nextPrefix = prefix ? `${prefix}.${key}` : key;
+    flattenExtractionDataToFieldPlanRecursive(child, nextPrefix, out);
+  }
+}
+function parseCounterLeafDescriptor(value) {
+  if (!value || typeof value !== "object" || Array.isArray(value)) {
+    return null;
+  }
+  const record = value;
+  if (!Object.hasOwn(record, COUNTER_KEY)) {
+    return null;
+  }
+  const counter = record.$c;
+  if (typeof counter !== "number" || !Number.isFinite(counter)) {
+    return null;
+  }
+  const rawAttribute = record.$a;
+  const normalizedAttribute = typeof rawAttribute === "string" ? rawAttribute.trim() : "";
+  return normalizedAttribute ? {
+    element: Math.trunc(counter),
+    attribute: normalizedAttribute
+  } : {
+    element: Math.trunc(counter)
+  };
+}
+
+export {
+  flattenExtractionDataToFieldPlan
+};