@simular-ai/simulang-js 8.0.0 → 9.0.0

package/CHANGELOG.md

@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [9.0.0] - 2026-06-29
+
+### Added
+
+- `Image.fromBase64()` now accepts GIF and WebP data URLs/payloads.
+- `Image.base64DataUrl()` and `Screenshot.base64DataUrl()` for returning MIME-prefixed data URLs.
+- Bundled OpenRouter config now advertises `openrouter_claude_opus` for both `AskModel` and `GroundingModel`, including Claude-specific grounding coordinate scaling and image size/format limits.
+- `StateSatisfiesModel` for evaluating screen state against a natural-language condition through the dedicated state-satisfaction provider.
+
+### Changed
+
+- `Image.base64()` and `Screenshot.base64()` now return raw base64 without a `data:image/...;base64,` prefix, matching the Rust API. Use `base64DataUrl()` when a MIME-prefixed data URL is needed.
+
+## [8.1.0] - 2026-06-23
+
+### Added
+
+- `AccessibilityNode.parent()`, `ancestors()`, direct/strict ancestry checks, and `lowestCommonAncestor(other)` for navigating accessibility-tree relationships. `parent()` returns `null` when a node has no parent, `ancestors()` stops only when the parent chain is fully resolved, and both throw on lookup failure; `lowestCommonAncestor` returns a `[node, level]` tuple where `level` is from the reached parentless node, returns `null` only for resolved unrelated trees, and throws if either parent chain cannot be resolved.
+- `Window.screenshot(hideCursor)` — captures just this window's pixels from its own backing store (macOS `ScreenCaptureKit` window filter / Windows `Windows.Graphics.Capture`), so occluding windows don't bleed through and hardware-accelerated content (Chrome, Electron, D3D apps) is captured correctly; off-display or minimized windows throw.
+- `Window.ground(model, concept)` — locate a concept within this window and return its global desktop coordinates (sugar for `screenshot(true).ground(model, concept)`); restricting the search to the window's bounds is faster and more accurate than grounding a full-screen screenshot.
+- `AccessibilityNode.url` — the node's raw hyperlink target as a `string`, or `null` when the node isn't a link, has no target, or the platform doesn't expose it.
+- `Image.drawDot(x, y, radius, red, green, blue)` and `Screenshot.drawDot(...)` — paint a filled opaque RGB disc at an image-pixel coordinate (companion to `drawGrid`; `radius` 0 paints a single pixel, out-of-bounds points are no-ops). Handy for annotating grounding results.
+
+### Changed
+
+- Frame-relative input on a `Window` (`moveMouse`, `click`) now throws when the target point maps off every connected display, instead of letting the OS silently clamp the cursor to a screen edge and click the wrong location.
+
 ## [8.0.0] - 2026-06-05
 
 ### Added

package/CLAUDE.md

@@ -2,7 +2,7 @@
 
 Node.js bindings for the Rust `simulang-rs` crate (via napi-rs). Cross-platform
 desktop automation: apps, windows, accessibility trees, mouse/keyboard,
-screenshots, clipboard, audio, and VLM/STT model access.
+screenshots, clipboard, audio, and VLM/LLM/STT model access.
 
 This file ships in the npm tarball alongside `index.d.ts` and is versioned
 with it.
@@ -10,7 +10,7 @@ with it.
 ## Where the API is documented
 
 Read **`index.d.ts`** first — it is the source of truth. Every class,
-function, and enum is fully typed (~1500 lines) and carries JSDoc covering
+function, and enum is fully typed (~2100 lines) and carries JSDoc covering
 idioms, lifecycle rules, platform quirks, and inter-API trade-offs that types
 alone can't express. The JSDoc is generated from doc comments,
 so the per-symbol guidance there is authoritative — trust it over any
@@ -24,12 +24,22 @@ restatement elsewhere.
 - Many objects are **handles to platform resources** (windows, audio devices,
   accessibility trees, file/directory handles). Their lifetime matters;
   dropping them can free the underlying resource.
-- Coordinates are uniform across platforms: **top-left origin, global
-  physical pixels** (also called device pixels — the raw hardware pixels of
-  the display, not the logical / CSS / point units used in browsers and
-  some desktop UI frameworks). On a 2× HiDPI display, a 1920×1080-logical
-  screen is 3840×2160 in these coordinates. Image and screenshot dimensions
-  are likewise in physical pixels.
+- Coordinates live on the **global desktop**: top-left origin at `(0, 0)` on
+  the primary monitor, in **OS-native units** — the unit is **not** the same
+  on every platform:
+  - **Windows / Linux**: **physical pixels** (raw hardware pixels).
+  - **macOS**: **logical points** — on a 2× Retina display one point spans two
+    hardware pixels, so a 1920×1080-logical screen is `1920×1080` here, not
+    `3840×2160`.
+
+  These are the native units the OS input/accessibility APIs expect, **not**
+  the browser logical/CSS pixel. Within a single OS every API speaks that OS's
+  unit, so coordinates round-trip between `MouseController`, the various
+  `boundingBox()` methods, `Screenshot.toGlobalDesktopCoordinates()`, and
+  grounding output **without conversion**; only code crossing into a different
+  coordinate system (e.g. an Electron overlay measured in CSS pixels) must
+  account for the per-OS unit. Monitors arranged to the left of / above the
+  primary contribute **negative** coordinates, so don't assume `x, y >= 0`.
 
 ## Logging is on by default
 

package/index.d.ts

@@ -61,6 +61,13 @@ export declare class AccessibilityNode {
   get automationId(): string
   /** Whether the node accepts user input. */
   get isEnabled(): boolean
+  /**
+   * Hyperlink target of this node, or `null` when the node is not a link,
+   * the link has no target, or the platform backend does not expose the
+   * target through the accessibility API. The string is the raw URL as
+   * reported by the platform (no parsing or normalisation).
+   */
+  get url(): string | null
   /**
    * Live bounding box of the element on the global desktop, in the canonical
    * coordinate space (OS-native units; see
@@ -74,6 +81,31 @@ export declare class AccessibilityNode {
    * simulang-rs's convention of treating walk failures as "no children").
    */
   children(): Array<AccessibilityNode>
+  /**
+   * Direct parent node, or `null` if this node has no parent. Parent lookup
+   * failures throw.
+   */
+  parent(): AccessibilityNode | null
+  /**
+   * Parent chain for this node, nearest parent first, excluding this node.
+   * Stops when a node has no parent; parent lookup failures throw.
+   */
+  ancestors(): Array<AccessibilityNode>
+  /** Whether this node is the direct parent of `other`. */
+  isParentOf(other: AccessibilityNode): boolean
+  /** Whether this node is a direct child of `other`. */
+  isChildOf(other: AccessibilityNode): boolean
+  /** Whether this node is a strict ancestor of `other`. */
+  isAncestorOf(other: AccessibilityNode): boolean
+  /** Whether this node is a strict descendant of `other`. */
+  isDescendantOf(other: AccessibilityNode): boolean
+  /**
+   * Lowest shared ancestor for this node and `other`, plus that ancestor's
+   * level from the reached parentless node (`parentless = 0`, its children
+   * `= 1`). Returns `null` when both chains resolve and no shared ancestor
+   * exists; lookup failures throw.
+   */
+  lowestCommonAncestor(other: AccessibilityNode): [AccessibilityNode, number] | null
   /**
    * Render this node's accessibility subtree as an indented
    * Playwright-style aria snapshot string.
@@ -315,13 +347,15 @@ export declare class AskModel {
   /**
    * First LLM model advertised by the first LLM-capable provider in the
    * loaded configuration whose credentials are currently available.
-   * Throws if no provider in the loaded config advertises an LLM service.
+   * Providers with missing credentials are skipped; throws if no provider
+   * remains available.
    */
   static default(): AskModel
   /**
    * Resolve a model alias against the loaded config (e.g.
-   * `"openrouter_gpt_4o_mini"` from the bundled `openrouter` provider, or
-   * any alias declared by a user provider). Throws if the alias is unknown.
+   * `"openrouter_gpt_4o_mini"` / `"openrouter_claude_opus"` from the
+   * bundled `openrouter` provider, or any alias declared by a user provider).
+   * Throws if the alias is unknown.
    */
   static byAlias(alias: string): AskModel
   /**
@@ -352,8 +386,8 @@ export declare class AskModel {
    * - `text`: optional accessibility-tree (or any) text included as
    *   structural context. Pass `null` / omit to skip.
    * - `images`: zero or more images to attach. Each is encoded as a
-   *   base64 data URL and sent as an `image_url` chat content part. Pass
-   *   `null` / omit for none.
+   *   base64 data URL in a model-supported format and sent as an
+   *   `image_url` chat content part. Pass `null` / omit for none.
    *
    * Returns the trimmed assistant response on success.
    */
@@ -532,14 +566,14 @@ export declare class File {
 export declare class GroundingModel {
   /**
    * First VLM model advertised by the first VLM provider in the loaded
-   * config. Throws if no provider in the loaded config advertises a VLM
-   * service.
+   * configuration whose credentials are currently available. Providers with
+   * missing credentials are skipped; throws if no provider remains available.
    */
   static default(): GroundingModel
   /**
    * Resolve a model alias against the loaded config (e.g. `"ui_venus_30b"`,
-   * `"ui_tars_7b"`, or any alias declared by a user provider). Throws if
-   * the alias is unknown.
+   * `"ui_tars_7b"`, `"openrouter_claude_opus"`, or any alias declared by a
+   * user provider). Throws if the alias is unknown.
    */
   static byAlias(alias: string): GroundingModel
   /**
@@ -595,6 +629,21 @@ export declare class Image {
    * Grid squares have the specified `width` and `height`.
    */
   drawGrid(width: number, height: number): void
+  /**
+   * Paints a filled disc on the image. Useful for visualizing point
+   * coordinates returned from grounding, layout queries, etc.
+   *
+   * `x` / `y` are image-pixel coordinates of the disc's centre. `radius`
+   * is the disc radius in pixels (`0` paints a single pixel at the
+   * centre). `(red, green, blue)` is the fill color; alpha is always 255
+   * (opaque replacement of the underlying pixel).
+   *
+   * Coordinates that fall outside the image bounds (negative, or past the
+   * width / height) silently produce no pixel, so the helper is safe to
+   * call with the raw output of a grounding model even at the edge of the
+   * captured rect.
+   */
+  drawDot(x: number, y: number, radius: number, red: number, green: number, blue: number): void
   /**
    * Compress the image by converting it to JPEG with the specified quality.
    *
@@ -616,19 +665,21 @@ export declare class Image {
   shrink(nwidth: number, nheight: number): void
   /** Returns the image dimensions as `[width, height]` in pixels. */
   get dimensions(): [number, number]
+  /** Returns the image encoded as raw base64, without a MIME prefix. */
+  base64(): string
   /**
    * Returns the image encoded as a base64 data URL.
    *
    * The result includes the MIME prefix, for example
-   * `data:image/png;base64,...` or `data:image/jpeg;base64,...`.
+   * `data:image/png;base64,...`, `data:image/jpeg;base64,...`,
+   * `data:image/gif;base64,...`, or `data:image/webp;base64,...`.
    */
-  base64(): string
+  base64DataUrl(): string
   /**
    * Decodes a base64 image string into an image.
    *
-   * Accepts either a raw base64 payload or a data URL such as
-   * `data:image/png;base64,...`, `data:image/jpeg;base64,...`, or
-   * `data:image/jpg;base64,...`.
+   * Accepts either a raw base64 payload or a PNG, JPEG/JPG, GIF, or WebP
+   * data URL.
    */
   static fromBase64(base64: string): Image
   /**
@@ -1072,6 +1123,21 @@ export declare class Screenshot {
    * Grid squares have the specified `width` and `height`.
    */
   drawGrid(width: number, height: number): void
+  /**
+   * Paints a filled disc on the image. Useful for visualizing point
+   * coordinates returned from grounding, layout queries, etc.
+   *
+   * `x` / `y` are image-pixel coordinates of the disc's centre. `radius`
+   * is the disc radius in pixels (`0` paints a single pixel at the
+   * centre). `(red, green, blue)` is the fill color; alpha is always 255
+   * (opaque replacement of the underlying pixel).
+   *
+   * Coordinates that fall outside the image bounds (negative, or past the
+   * width / height) silently produce no pixel, so the helper is safe to
+   * call with the raw output of a grounding model even at the edge of the
+   * captured rect.
+   */
+  drawDot(x: number, y: number, radius: number, red: number, green: number, blue: number): void
   /**
    * Compress the image by converting it to JPEG with the specified quality.
    *
@@ -1093,13 +1159,16 @@ export declare class Screenshot {
   shrink(nwidth: number, nheight: number): void
   /** Returns the screenshot dimensions as `[width, height]` in pixels. */
   get dimensions(): [number, number]
+  /** Returns the image encoded as raw base64, without a MIME prefix. */
+  base64(): string
   /**
    * Returns the image encoded as a base64 data URL.
    *
    * The result includes the MIME prefix, for example
-   * `data:image/png;base64,...` or `data:image/jpeg;base64,...`.
+   * `data:image/png;base64,...`, `data:image/jpeg;base64,...`,
+   * `data:image/gif;base64,...`, or `data:image/webp;base64,...`.
    */
-  base64(): string
+  base64DataUrl(): string
   /**
    * Converts a point in this screenshot to global desktop coordinates.
    *
@@ -1148,6 +1217,44 @@ export declare class ScreenshotCoordinateType {
   static normalized(range: number): ScreenshotCoordinateType
 }
 
+/**
+ * A dedicated state-satisfaction model that evaluates whether observed screen
+ * state satisfies a natural-language condition.
+ *
+ * This speaks Simular's `v1/perception/state_satisfies` endpoint. It is
+ * separate from `AskModel`, which remains an OpenAI-compatible
+ * chat-completions client.
+ */
+export declare class StateSatisfiesModel {
+  /**
+   * First state-satisfaction model advertised by the first capable provider
+   * in the loaded configuration whose credentials are currently available.
+   * Throws if no provider advertises the state-satisfies service.
+   */
+  static default(): StateSatisfiesModel
+  /**
+   * Resolve a model alias against the loaded config. Throws if the alias is
+   * unknown.
+   */
+  static byAlias(alias: string): StateSatisfiesModel
+  /** Convenience shim for the bundled `simular_state_satisfies` alias. */
+  static simularStateSatisfies(): StateSatisfiesModel
+  /**
+   * Every state-satisfaction model alias accepted by `byAlias` on this
+   * machine, deduplicated and sorted alphabetically.
+   */
+  static availableAliases(): Array<string>
+  /** The provider-configured model identifier. */
+  get name(): string
+  /** Check that the provider credentials work. Throws if they do not. */
+  checkAuth(): void
+  /**
+   * Evaluate `condition` against the supplied accessibility text and raw
+   * screenshot base64.
+   */
+  stateSatisfies(condition: string, text: string, image: string): boolean
+}
+
 /** A speech-to-text model that can transcribe audio. */
 export declare class SttModel {
   /**
@@ -1292,6 +1399,16 @@ export declare class Window {
    * top-left of [`Window.boundingBox`], which includes the title bar and
    * other OS chrome.
    *
+   * Two validation passes run before any input is synthesized, and either
+   * failing throws:
+   * 1. `(x, y)` must lie inside the window frame (`[0, width) × [0,
+   *    height)`).
+   * 2. The resulting global point must fall on at least one connected
+   *    display. Windows dragged partly off-screen can map an in-frame
+   *    point to a coordinate the OS would silently clamp to a display
+   *    edge — erroring out is strictly more useful than moving the cursor
+   *    somewhere unintended and clicking the wrong thing.
+   *
    * Does **not** focus the window. Call [`Window.focus`] (or
    * [`Instance.focus`]) first if the click target requires the window
    * to be active.
@@ -1310,7 +1427,9 @@ export declare class Window {
    * Move the cursor to `(x, y)` inside this window and synthesise a
    * mouse button event. Sugar for `moveMouse(x, y); button(...)`.
    *
-   * Coordinates are window-frame-relative (see [`Window.moveMouse`]).
+   * Coordinates are window-frame-relative (see [`Window.moveMouse`]), and
+   * are subject to the same bounds / on-screen validation: an off-frame or
+   * off-screen target throws instead of clicking the wrong location.
    * Does not focus the window.
    */
   click(x: number, y: number, button: Button, direction: Direction): void
@@ -1333,6 +1452,43 @@ export declare class Window {
    * [`AccessibilityTree.snapshot`] instead.
    */
   snapshot(): string
+  /**
+   * Capture just this window's pixels.
+   *
+   * Reads from the window's own backing store, so overlapping windows do
+   * **not** bleed through and the whole window is captured even where it is
+   * occluded or extends past a display edge — partially off-screen and
+   * multi-display-spanning windows are captured in full. The window must be
+   * on-screen at capture time: a minimized or fully off-display window
+   * throws, because there is nothing to capture.
+   */
+  screenshot(hideCursor: boolean): Screenshot
+  /**
+   * Locate `concept` inside this window's pixels and return its **global
+   * desktop coordinates** `[x, y]` in OS-native units (may be negative on
+   * multi-monitor setups; see [`MouseController`]), ready to feed straight
+   * into [`MouseController.moveMouse`] / click helpers.
+   *
+   * Sugar for `screenshot(true).ground(model, concept)` — the cursor is
+   * hidden because it can occlude or distract the model. Restricts the
+   * model's search to this window's bounds, which is both faster (fewer
+   * pixels to upload) and more accurate (no risk of grounding onto a
+   * concept that happens to be elsewhere on the screen) than grounding a
+   * full-screen screenshot.
+   *
+   * Drop down to [`Window.screenshot`] + [`Screenshot.ground`] directly
+   * when you want to keep the cursor visible, reuse the screenshot across
+   * multiple `ground` calls, or shrink / compress the image before
+   * sending it.
+   *
+   * The screenshot includes any portion of the window that's past the
+   * desktop edge (the capture reads the window's own backing store, not a
+   * display rectangle). A `concept` the model locates in that off-screen
+   * region returns coordinates the OS won't route a click to;
+   * [`Window.moveMouse`] catches this and errors out instead of clicking
+   * the nearest on-screen point.
+   */
+  ground(model: GroundingModel, concept: string): [number, number]
   /**
    * Search this window's accessibility subtree by *concept text*, using
    * bag-of-words paired-Jaccard scoring against each node's