@simular-ai/simulang-js 8.1.0 → 10.0.0

package/CHANGELOG.md

@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [10.0.0] - 2026-07-03
+
+### Added
+
+- `Window.focused()` — the window that currently has keyboard focus, or `null` when nothing is focused (no windows open, focus on the desktop, or the frontmost app has no focused window). The read-side counterpart to the instance method `Window.focus()`. Backed by `GetForegroundWindow` (Windows), the frontmost app's `AXFocusedWindow` (macOS), and `_NET_ACTIVE_WINDOW` (Linux); throws only on genuine backend failures.
+- `ImageGenModel` for generating images from a text prompt through a provider's `image_gen` service. `generate(prompt, quality?, aspectRatio?, images?)` returns a `[image, description]` tuple — the generated `Image` and the model's text description. `quality` (`ImageQuality.Fast` / `.Pro`) and `aspectRatio` (`ImageAspectRatio.Square`, `.Landscape16x9`, …) are type-safe enums defaulting to `Fast` / `Square`. Optional reference images are downscaled / transcoded and size-checked against the model's configured limits before sending. Discover aliases with `ImageGenModel.availableAliases()`; the bundled `simular_cloud` provider advertises `simular_image_gen`.
+- `Image.drawBox(bounds, thickness, red, green, blue)` and `Screenshot.drawBox(...)` — paint the outline of an axis-aligned `BoundingBox` (e.g. the result of `Window.boundingBox()` or `AccessibilityNode.boundingBox()`) with an inset border `thickness` pixels wide in an opaque RGB color. On `Image` the box is in image pixels; on `Screenshot` it is in **global desktop** coordinates (converted to image pixels internally), so a grounding or element box can be passed straight in. Out-of-bounds pixels are clipped; throws on zero thickness or a degenerate box (`right <= left` / `bottom <= top`). Companion to `drawDot` for annotating bounding boxes.
+
+### Breaking
+
+- `Screenshot.drawDot(x, y, ...)` now interprets `(x, y)` as **global desktop** coordinates (the same space `Screenshot.ground` returns and `MouseController` consumes) rather than screenshot image pixels; the point is converted to image pixels internally, inverting the capture offset and any resampling. This lets a `Screenshot.ground` result or an element's `boundingBox()` corner be drawn without manual conversion. Update call sites that passed raw image-pixel coordinates to `Screenshot.drawDot` to pass global desktop coordinates instead. `Image.drawDot` is unchanged (still image pixels).
+
+## [9.0.0] - 2026-06-29
+
+### Added
+
+- `Image.fromBase64()` now accepts GIF and WebP data URLs/payloads.
+- `Image.base64DataUrl()` and `Screenshot.base64DataUrl()` for returning MIME-prefixed data URLs.
+- Bundled OpenRouter config now advertises `openrouter_claude_opus` for both `AskModel` and `GroundingModel`, including Claude-specific grounding coordinate scaling and image size/format limits.
+- `StateSatisfiesModel` for evaluating screen state against a natural-language condition through the dedicated state-satisfaction provider.
+
+### Changed
+
+- `Image.base64()` and `Screenshot.base64()` now return raw base64 without a `data:image/...;base64,` prefix, matching the Rust API. Use `base64DataUrl()` when a MIME-prefixed data URL is needed.
+
 ## [8.1.0] - 2026-06-23
 
 ### Added

package/CLAUDE.md

@@ -2,7 +2,7 @@
 
 Node.js bindings for the Rust `simulang-rs` crate (via napi-rs). Cross-platform
 desktop automation: apps, windows, accessibility trees, mouse/keyboard,
-screenshots, clipboard, audio, and VLM/STT model access.
+screenshots, clipboard, audio, and VLM/LLM/STT model access.
 
 This file ships in the npm tarball alongside `index.d.ts` and is versioned
 with it.
@@ -10,7 +10,7 @@ with it.
 ## Where the API is documented
 
 Read **`index.d.ts`** first — it is the source of truth. Every class,
-function, and enum is fully typed (~1500 lines) and carries JSDoc covering
+function, and enum is fully typed (~2100 lines) and carries JSDoc covering
 idioms, lifecycle rules, platform quirks, and inter-API trade-offs that types
 alone can't express. The JSDoc is generated from doc comments,
 so the per-symbol guidance there is authoritative — trust it over any
@@ -24,12 +24,22 @@ restatement elsewhere.
 - Many objects are **handles to platform resources** (windows, audio devices,
   accessibility trees, file/directory handles). Their lifetime matters;
   dropping them can free the underlying resource.
-- Coordinates are uniform across platforms: **top-left origin, global
-  physical pixels** (also called device pixels — the raw hardware pixels of
-  the display, not the logical / CSS / point units used in browsers and
-  some desktop UI frameworks). On a 2× HiDPI display, a 1920×1080-logical
-  screen is 3840×2160 in these coordinates. Image and screenshot dimensions
-  are likewise in physical pixels.
+- Coordinates live on the **global desktop**: top-left origin at `(0, 0)` on
+  the primary monitor, in **OS-native units** — the unit is **not** the same
+  on every platform:
+  - **Windows / Linux**: **physical pixels** (raw hardware pixels).
+  - **macOS**: **logical points** — on a 2× Retina display one point spans two
+    hardware pixels, so a 1920×1080-logical screen is `1920×1080` here, not
+    `3840×2160`.
+
+  These are the native units the OS input/accessibility APIs expect, **not**
+  the browser logical/CSS pixel. Within a single OS every API speaks that OS's
+  unit, so coordinates round-trip between `MouseController`, the various
+  `boundingBox()` methods, `Screenshot.toGlobalDesktopCoordinates()`, and
+  grounding output **without conversion**; only code crossing into a different
+  coordinate system (e.g. an Electron overlay measured in CSS pixels) must
+  account for the per-OS unit. Monitors arranged to the left of / above the
+  primary contribute **negative** coordinates, so don't assume `x, y >= 0`.
 
 ## Logging is on by default
 

package/index.d.ts

@@ -347,13 +347,15 @@ export declare class AskModel {
   /**
    * First LLM model advertised by the first LLM-capable provider in the
    * loaded configuration whose credentials are currently available.
-   * Throws if no provider in the loaded config advertises an LLM service.
+   * Providers with missing credentials are skipped; throws if no provider
+   * remains available.
    */
   static default(): AskModel
   /**
    * Resolve a model alias against the loaded config (e.g.
-   * `"openrouter_gpt_4o_mini"` from the bundled `openrouter` provider, or
-   * any alias declared by a user provider). Throws if the alias is unknown.
+   * `"openrouter_gpt_4o_mini"` / `"openrouter_claude_opus"` from the
+   * bundled `openrouter` provider, or any alias declared by a user provider).
+   * Throws if the alias is unknown.
    */
   static byAlias(alias: string): AskModel
   /**
@@ -384,8 +386,8 @@ export declare class AskModel {
    * - `text`: optional accessibility-tree (or any) text included as
    *   structural context. Pass `null` / omit to skip.
    * - `images`: zero or more images to attach. Each is encoded as a
-   *   base64 data URL and sent as an `image_url` chat content part. Pass
-   *   `null` / omit for none.
+   *   base64 data URL in a model-supported format and sent as an
+   *   `image_url` chat content part. Pass `null` / omit for none.
    *
    * Returns the trimmed assistant response on success.
    */
@@ -564,14 +566,14 @@ export declare class File {
 export declare class GroundingModel {
   /**
    * First VLM model advertised by the first VLM provider in the loaded
-   * config. Throws if no provider in the loaded config advertises a VLM
-   * service.
+   * configuration whose credentials are currently available. Providers with
+   * missing credentials are skipped; throws if no provider remains available.
    */
   static default(): GroundingModel
   /**
    * Resolve a model alias against the loaded config (e.g. `"ui_venus_30b"`,
-   * `"ui_tars_7b"`, or any alias declared by a user provider). Throws if
-   * the alias is unknown.
+   * `"ui_tars_7b"`, `"openrouter_claude_opus"`, or any alias declared by a
+   * user provider). Throws if the alias is unknown.
    */
   static byAlias(alias: string): GroundingModel
   /**
@@ -642,6 +644,21 @@ export declare class Image {
    * captured rect.
    */
   drawDot(x: number, y: number, radius: number, red: number, green: number, blue: number): void
+  /**
+   * Draws the outline of the axis-aligned rectangle `bounds` on the image.
+   * Useful for visualizing bounding boxes returned from grounding, element /
+   * layout queries, ground-truth annotations, etc.
+   *
+   * `bounds` is in image-pixel coordinates and covers `[left, right) ×
+   * [top, bottom)` (`right` / `bottom` exclusive, matching a `BoundingBox`).
+   * The border is `thickness` pixels wide, drawn inset within that range so
+   * nothing is painted outside it, in the opaque RGB `(red, green, blue)`
+   * color. Pixels that fall outside the image bounds are silently clipped, so
+   * it is safe to call with a box that extends past the image. Throws when
+   * `thickness` is `0` or `bounds` is degenerate (`right <= left` or
+   * `bottom <= top`).
+   */
+  drawBox(bounds: BoundingBox, thickness: number, red: number, green: number, blue: number): void
   /**
    * Compress the image by converting it to JPEG with the specified quality.
    *
@@ -663,19 +680,21 @@ export declare class Image {
   shrink(nwidth: number, nheight: number): void
   /** Returns the image dimensions as `[width, height]` in pixels. */
   get dimensions(): [number, number]
+  /** Returns the image encoded as raw base64, without a MIME prefix. */
+  base64(): string
   /**
    * Returns the image encoded as a base64 data URL.
    *
    * The result includes the MIME prefix, for example
-   * `data:image/png;base64,...` or `data:image/jpeg;base64,...`.
+   * `data:image/png;base64,...`, `data:image/jpeg;base64,...`,
+   * `data:image/gif;base64,...`, or `data:image/webp;base64,...`.
    */
-  base64(): string
+  base64DataUrl(): string
   /**
    * Decodes a base64 image string into an image.
    *
-   * Accepts either a raw base64 payload or a data URL such as
-   * `data:image/png;base64,...`, `data:image/jpeg;base64,...`, or
-   * `data:image/jpg;base64,...`.
+   * Accepts either a raw base64 payload or a PNG, JPEG/JPG, GIF, or WebP
+   * data URL.
    */
   static fromBase64(base64: string): Image
   /**
@@ -687,6 +706,79 @@ export declare class Image {
   ground(model: GroundingModel, concept: string): [number, number]
 }
 
+/**
+ * A model for generating images from a text prompt (with optional reference
+ * images), speaking Simular's `v1/image-gen` wire contract.
+ *
+ * This targets a dedicated Simular endpoint rather than an `OpenAI`-compatible
+ * one; the endpoint picks the concrete backend model from the [`ImageQuality`]
+ * argument, so the resolved model `name` is nominal (used only for `name`).
+ * Reference images are downscaled / transcoded and size-checked against the
+ * model's configured image limits before sending, exactly as the VLM/LLM image
+ * paths do.
+ */
+export declare class ImageGenModel {
+  /**
+   * First image-gen model advertised by the first capable provider in the
+   * loaded configuration whose credentials are currently available.
+   * Providers that advertise the service but lack working credentials are
+   * skipped. Throws only if no provider is left after that filter.
+   */
+  static default(): ImageGenModel
+  /**
+   * Resolve a model alias against the loaded configuration.
+   *
+   * The alias is looked up across every loaded provider in alphabetical
+   * order; the first provider that advertises this alias wins. Throws if the
+   * alias is unknown.
+   */
+  static byAlias(alias: string): ImageGenModel
+  /** Convenience shim for the bundled Simular image-gen alias. */
+  static simularImageGen(): ImageGenModel
+  /**
+   * Every image-gen model alias advertised by the loaded configuration,
+   * deduplicated and sorted alphabetically. Use to discover what `byAlias`
+   * will accept on the current machine.
+   */
+  static availableAliases(): Array<string>
+  /**
+   * The model identifier from provider configuration. Nominal — the
+   * endpoint selects the concrete model from the [`ImageQuality`] argument.
+   */
+  get name(): string
+  /**
+   * Check that provider credentials work. Throws (and logs a warning) if they
+   * do not.
+   *
+   * Call right after creating the model so a bad key fails fast:
+   *
+   * ```ts
+   * try { model.checkAuth() } catch { process.exit(1) }
+   * ```
+   */
+  checkAuth(): void
+  /**
+   * Generate an image from `prompt`.
+   *
+   * - `prompt`: a description of the image to generate.
+   * - `quality`: selects the model tier; defaults to `ImageQuality.Fast`.
+   * - `aspectRatio`: the output shape; defaults to `ImageAspectRatio.Square`.
+   * - `images`: reference images (count capped by the model's configured
+   *   `max_images_per_request` image limit) that condition the result; each is
+   *   downscaled / transcoded to a supported format and size-checked before
+   *   sending. Pass `null` / omit for none.
+   *
+   * Returns a `[image, description]` tuple: the first generated [`Image`] and
+   * the model's text description (an empty string when the model returns none).
+   */
+  generate(
+    prompt: string,
+    quality?: ImageQuality | undefined | null,
+    aspectRatio?: ImageAspectRatio | undefined | null,
+    images?: Array<Image> | undefined | null,
+  ): [Image, string]
+}
+
 /** Represents an opened application instance. */
 export declare class Instance {
   /** Get the process ID of the opened application (returns 0 when unknown). */
@@ -1120,20 +1212,39 @@ export declare class Screenshot {
    */
   drawGrid(width: number, height: number): void
   /**
-   * Paints a filled disc on the image. Useful for visualizing point
-   * coordinates returned from grounding, layout queries, etc.
+   * Paints a filled disc on the screenshot. Useful for visualizing point
+   * coordinates returned from grounding, element / layout queries, etc.
    *
-   * `x` / `y` are image-pixel coordinates of the disc's centre. `radius`
-   * is the disc radius in pixels (`0` paints a single pixel at the
-   * centre). `(red, green, blue)` is the fill color; alpha is always 255
-   * (opaque replacement of the underlying pixel).
+   * `x` / `y` are **global desktop** coordinates (the same space
+   * `Screenshot.ground` returns and `MouseController` consumes), which are
+   * converted to image pixels — inverting the capture offset and any
+   * resampling — before drawing. So a `ground(...)` result or an element's
+   * `boundingBox()` corner can be passed straight in. `radius` is the disc
+   * radius in pixels (`0` paints a single pixel at the centre).
+   * `(red, green, blue)` is the fill color; alpha is always 255 (opaque
+   * replacement of the underlying pixel).
    *
-   * Coordinates that fall outside the image bounds (negative, or past the
-   * width / height) silently produce no pixel, so the helper is safe to
-   * call with the raw output of a grounding model even at the edge of the
-   * captured rect.
+   * Coordinates that map outside the image bounds silently produce no pixel,
+   * so the helper is safe to call even for points that fall outside the
+   * captured region.
    */
   drawDot(x: number, y: number, radius: number, red: number, green: number, blue: number): void
+  /**
+   * Draws the outline of the axis-aligned rectangle `bounds` on the
+   * screenshot. Useful for visualizing bounding boxes returned from grounding,
+   * element / window `boundingBox()` queries, ground-truth annotations, etc.
+   *
+   * `bounds` is in **global desktop** coordinates (the same space element /
+   * window `boundingBox()` and grounding results use); its corners are
+   * converted to image pixels — inverting the capture offset and any
+   * resampling — before drawing, so an element's `boundingBox()` can be passed
+   * straight in. It covers `[left, right) × [top, bottom)` (`right` / `bottom`
+   * exclusive). The border is `thickness` pixels wide, drawn inset, in the
+   * opaque RGB `(red, green, blue)` color. Pixels that map outside the image
+   * bounds are silently clipped. Throws when `thickness` is `0` or `bounds` is
+   * degenerate (`right <= left` or `bottom <= top`).
+   */
+  drawBox(bounds: BoundingBox, thickness: number, red: number, green: number, blue: number): void
   /**
    * Compress the image by converting it to JPEG with the specified quality.
    *
@@ -1155,13 +1266,16 @@ export declare class Screenshot {
   shrink(nwidth: number, nheight: number): void
   /** Returns the screenshot dimensions as `[width, height]` in pixels. */
   get dimensions(): [number, number]
+  /** Returns the image encoded as raw base64, without a MIME prefix. */
+  base64(): string
   /**
    * Returns the image encoded as a base64 data URL.
    *
    * The result includes the MIME prefix, for example
-   * `data:image/png;base64,...` or `data:image/jpeg;base64,...`.
+   * `data:image/png;base64,...`, `data:image/jpeg;base64,...`,
+   * `data:image/gif;base64,...`, or `data:image/webp;base64,...`.
    */
-  base64(): string
+  base64DataUrl(): string
   /**
    * Converts a point in this screenshot to global desktop coordinates.
    *
@@ -1210,6 +1324,44 @@ export declare class ScreenshotCoordinateType {
   static normalized(range: number): ScreenshotCoordinateType
 }
 
+/**
+ * A dedicated state-satisfaction model that evaluates whether observed screen
+ * state satisfies a natural-language condition.
+ *
+ * This speaks Simular's `v1/perception/state_satisfies` endpoint. It is
+ * separate from `AskModel`, which remains an OpenAI-compatible
+ * chat-completions client.
+ */
+export declare class StateSatisfiesModel {
+  /**
+   * First state-satisfaction model advertised by the first capable provider
+   * in the loaded configuration whose credentials are currently available.
+   * Throws if no provider advertises the state-satisfies service.
+   */
+  static default(): StateSatisfiesModel
+  /**
+   * Resolve a model alias against the loaded config. Throws if the alias is
+   * unknown.
+   */
+  static byAlias(alias: string): StateSatisfiesModel
+  /** Convenience shim for the bundled `simular_state_satisfies` alias. */
+  static simularStateSatisfies(): StateSatisfiesModel
+  /**
+   * Every state-satisfaction model alias accepted by `byAlias` on this
+   * machine, deduplicated and sorted alphabetically.
+   */
+  static availableAliases(): Array<string>
+  /** The provider-configured model identifier. */
+  get name(): string
+  /** Check that the provider credentials work. Throws if they do not. */
+  checkAuth(): void
+  /**
+   * Evaluate `condition` against the supplied accessibility text and raw
+   * screenshot base64.
+   */
+  stateSatisfies(condition: string, text: string, image: string): boolean
+}
+
 /** A speech-to-text model that can transcribe audio. */
 export declare class SttModel {
   /**
@@ -1296,6 +1448,24 @@ export declare class Window {
    * - **Linux**: always `null` (no per-point window hit-test).
    */
   static fromPoint(x: number, y: number): Window | null
+  /**
+   * The window that currently has keyboard focus, if any.
+   *
+   * Returns `null` when nothing is focused (e.g. no windows open, focus is on
+   * the desktop, or the frontmost app has no focused window) rather than
+   * treating that as an error. Throwing is reserved for genuine backend
+   * failures.
+   *
+   * Platform notes:
+   * - **Windows**: `GetForegroundWindow` — the top-level window of the
+   *   foreground thread. Never throws; `null` only when there is no
+   *   foreground window at all.
+   * - **macOS**: the frontmost running application's `AXFocusedWindow`
+   *   attribute. `null` when there is no frontmost app, or the frontmost app
+   *   has no focused window (e.g. all its windows are minimized).
+   * - **Linux**: EWMH `_NET_ACTIVE_WINDOW` root-window property.
+   */
+  static focused(): Window | null
   /** Window title (may be empty). */
   get title(): string
   /** Process ID that owns this window. */
@@ -1655,6 +1825,37 @@ export declare enum FocusPolicy {
 
 export declare function hasScreenCapturePermission(): boolean
 
+/** Aspect ratio of the generated image. */
+export declare enum ImageAspectRatio {
+  /** `1:1` square. Default. */
+  Square = 0,
+  /** `16:9` widescreen landscape. */
+  Landscape16x9 = 1,
+  /** `9:16` tall portrait. */
+  Portrait9x16 = 2,
+  /** `4:3` landscape. */
+  Landscape4x3 = 3,
+  /** `3:4` portrait. */
+  Portrait3x4 = 4,
+  /** `3:2` landscape. */
+  Landscape3x2 = 5,
+  /** `2:3` portrait. */
+  Portrait2x3 = 6,
+}
+
+/**
+ * Output image quality / model tier for [`ImageGenModel.generate`].
+ *
+ * The image-gen endpoint selects the underlying model from this value, so it
+ * is a per-request parameter rather than a config-resolved model name.
+ */
+export declare enum ImageQuality {
+  /** Faster, cheaper tier. Default. */
+  Fast = 0,
+  /** Higher-fidelity tier. */
+  Pro = 1,
+}
+
 /**
  * Initialize the logger.
  *