visual-ai-assertions 0.13.1 → 0.16.0

package/README.md

@@ -1,6 +1,6 @@
 # visual-ai-assertions
 
-AI-powered visual assertions for E2E tests. Send screenshots — or short video recordings — to Claude, GPT, or Gemini and get structured, typed results.
+AI-powered visual assertions for E2E tests. Send screenshots — or short video recordings — to Claude, GPT, Gemini — or Grok, Kimi, and Qwen via OpenRouter — and get structured, typed results.
 
 ## Installation
 
@@ -11,6 +11,7 @@ npm install visual-ai-assertions
 # Optional: install additional provider SDKs
 npm install @anthropic-ai/sdk    # for Claude
 npm install @google/genai        # for Gemini
+# OpenRouter (Grok, Kimi, Qwen, ...) uses the OpenAI SDK — no extra install
 
 # Zod is a peer dependency
 npm install zod
@@ -346,6 +347,31 @@ How it works: the library samples frames with ffmpeg and sends them to the provi
 
 **ffmpeg setup.** Video support works out of the box — `fluent-ffmpeg`, `@ffmpeg-installer/ffmpeg`, and `@ffprobe-installer/ffprobe` ship as regular dependencies and bundle platform-specific ffmpeg/ffprobe binaries. If you ran `npm install` you already have everything you need. On platforms where the prebuilt binary is unavailable (or if you've pruned dependencies), `check()` and `ask()` throw `VisualAIVideoError` (import from `visual-ai-assertions` to `instanceof`-narrow it) when called with video input.
 
+### Pre-sampled Frames
+
+When you already have an array of screenshots — or can't pass a video file — hand `check()` and `ask()` a `FramesInput` (`{ frames, fps? }`) instead of a `MediaInput`. Each frame is a plain `ImageInput` (buffer, data URL, base64, file path, or URL) or a `{ image, timestampSeconds }` pair. Frames are treated as the same chronological timeline a sampled video produces — no ffmpeg is loaded on this path.
+
+```typescript
+const frames = [await page.screenshot(), await page.screenshot()];
+
+const result = await ai.check({ frames }, ['A success toast with text "Saved" appears']);
+console.log(result.frames);
+// { count: 2, timestampsSeconds: [0, 1], durationSeconds: 1 }
+
+// Control timestamps: give an explicit fps, or per-frame timestampSeconds
+await ai.ask(
+  {
+    frames: [
+      { image: before, timestampSeconds: 0 },
+      { image: after, timestampSeconds: 2.5 },
+    ],
+  },
+  "What changed between the frames?",
+);
+```
+
+Timestamps for bare frames are derived as `index / fps` (default `fps` is `1`); a per-frame `timestampSeconds` overrides that. The frame count is subject to the same 60-frame hard cap as video sampling, and the `video` sampling option is ignored for this path.
+
 ### Formatting & Assertion Helpers
 
 ```typescript
@@ -416,11 +442,12 @@ The `VisualAIKnownError` union and `isVisualAIKnownError()` helper are useful wh
 
 ### API Keys
 
-| Provider  | Environment Variable |
-| --------- | -------------------- |
-| Anthropic | `ANTHROPIC_API_KEY`  |
-| OpenAI    | `OPENAI_API_KEY`     |
-| Google    | `GOOGLE_API_KEY`     |
+| Provider   | Environment Variable |
+| ---------- | -------------------- |
+| Anthropic  | `ANTHROPIC_API_KEY`  |
+| OpenAI     | `OPENAI_API_KEY`     |
+| Google     | `GOOGLE_API_KEY`     |
+| OpenRouter | `OPENROUTER_API_KEY` |
 
 ### Optional Configuration
 
@@ -473,11 +500,12 @@ type SupportedMimeType = "image/jpeg" | "image/png" | "image/webp" | "image/gif"
 
 **Default models:**
 
-| Provider  | Default Model            |
-| --------- | ------------------------ |
-| Anthropic | `claude-sonnet-4-6`      |
-| OpenAI    | `gpt-5-mini`             |
-| Google    | `gemini-3-flash-preview` |
+| Provider   | Default Model            |
+| ---------- | ------------------------ |
+| Anthropic  | `claude-sonnet-4-6`      |
+| OpenAI     | `gpt-5.4-mini`           |
+| Google     | `gemini-3-flash-preview` |
+| OpenRouter | `qwen/qwen3.6-flash`     |
 
 ## Reasoning Effort
 
@@ -497,6 +525,7 @@ When omitted, each provider uses its default behavior. The `"xhigh"` level enabl
 | Anthropic (other)                         | `thinking.type: "adaptive"` + `output_config.effort`  | `effort: "max"`      |
 | OpenAI                                    | `reasoning.effort` (Responses API)                    | `effort: "xhigh"`    |
 | Google                                    | `thinkingConfig.thinkingBudget` (1024 / 8192 / 24576) | `24576` (max budget) |
+| OpenRouter                                | `reasoning.effort` (normalized low/medium/high)       | `effort: "high"`     |
 
 ## Supported Models
 
@@ -533,11 +562,27 @@ All listed models support image/vision input. Pass any model ID to the `model` c
 
 | Model                 | Model ID                 | Input $/MTok | Output $/MTok | Notes                             |
 | --------------------- | ------------------------ | ------------ | ------------- | --------------------------------- |
+| Gemini 3.6 Flash      | `gemini-3.6-flash`       | $1.50        | $7.50         | Newest GA flash; fewer out-tokens |
 | Gemini 3.5 Flash      | `gemini-3.5-flash`       | $1.50        | $9            | Strongest agentic & coding model  |
+| Gemini 3.5 Flash Lite | `gemini-3.5-flash-lite`  | $0.30        | $2.50         | GA — fast, cheap, agentic tier    |
 | Gemini 3.1 Pro        | `gemini-3.1-pro-preview` | $2           | $12           | Preview — most advanced reasoning |
 | Gemini 3.1 Flash Lite | `gemini-3.1-flash-lite`  | $0.25        | $1.50         | GA — lightweight and cheap        |
 | Gemini 3 Flash        | `gemini-3-flash-preview` | $0.50        | $3            | **Default** — fast and capable    |
 
+### OpenRouter
+
+Any [OpenRouter](https://openrouter.ai/models) model slug (always `vendor/model`) is accepted — the vendor prefix is how the library recognizes an OpenRouter model. The models below are tested and have pricing built in. Note that OpenRouter may route a request to different upstream hosts with different quantizations; keep that in mind when comparing benchmark numbers.
+
+| Model          | Model ID                    | Input $/MTok | Output $/MTok | Notes                                 |
+| -------------- | --------------------------- | ------------ | ------------- | ------------------------------------- |
+| Grok 4.5       | `x-ai/grok-4.5`             | $2           | $6            | xAI flagship, 500K context            |
+| Kimi K3        | `moonshotai/kimi-k3`        | $3           | $15           | Moonshot flagship, 1M context         |
+| Kimi K2.7 Code | `moonshotai/kimi-k2.7-code` | $0.82        | $3.75         | Agentic/coding tier with vision       |
+| Qwen3.7 Plus   | `qwen/qwen3.7-plus`         | $0.32        | $1.28         | Cost-effective, GUI/screen-reading    |
+| Qwen3.6 Flash  | `qwen/qwen3.6-flash`        | $0.19        | $1.13         | **Default** — cheap flash vision tier |
+
+`qwen/qwen3.7-max` is not listed because it accepts no image input on OpenRouter.
+
 ## License
 
 MIT

package/dist/index.cjs

@@ -76,7 +76,8 @@ var ReasoningEffort = {
 var Provider = {
   ANTHROPIC: "anthropic",
   OPENAI: "openai",
-  GOOGLE: "google"
+  GOOGLE: "google",
+  OPENROUTER: "openrouter"
 };
 var Model = {
   Anthropic: {
@@ -101,29 +102,47 @@ var Model = {
     GPT_5_MINI: "gpt-5-mini"
   },
   Google: {
+    GEMINI_3_6_FLASH: "gemini-3.6-flash",
     GEMINI_3_5_FLASH: "gemini-3.5-flash",
+    GEMINI_3_5_FLASH_LITE: "gemini-3.5-flash-lite",
     GEMINI_3_1_PRO_PREVIEW: "gemini-3.1-pro-preview",
     GEMINI_3_1_FLASH_LITE: "gemini-3.1-flash-lite",
     GEMINI_3_FLASH_PREVIEW: "gemini-3-flash-preview"
+  },
+  /**
+   * Models routed through OpenRouter (https://openrouter.ai). Slugs always
+   * carry a vendor prefix (`vendor/model`), which is how provider inference
+   * recognizes them. All listed models accept image input.
+   */
+  OpenRouter: {
+    GROK_4_5: "x-ai/grok-4.5",
+    KIMI_K3: "moonshotai/kimi-k3",
+    KIMI_K2_7_CODE: "moonshotai/kimi-k2.7-code",
+    QWEN_3_7_PLUS: "qwen/qwen3.7-plus",
+    QWEN_3_6_FLASH: "qwen/qwen3.6-flash"
   }
 };
 var DEFAULT_MODELS = {
   [Provider.ANTHROPIC]: Model.Anthropic.SONNET_4_6,
   [Provider.OPENAI]: Model.OpenAI.GPT_5_4_MINI,
-  [Provider.GOOGLE]: Model.Google.GEMINI_3_FLASH_PREVIEW
+  [Provider.GOOGLE]: Model.Google.GEMINI_3_FLASH_PREVIEW,
+  [Provider.OPENROUTER]: Model.OpenRouter.QWEN_3_6_FLASH
 };
 var DEFAULT_MAX_TOKENS = 4096;
 var OPENAI_REASONING_MAX_TOKENS = 16384;
 var MODEL_TO_PROVIDER = new Map([
   ...Object.values(Model.Anthropic).map((m) => [m, Provider.ANTHROPIC]),
   ...Object.values(Model.OpenAI).map((m) => [m, Provider.OPENAI]),
-  ...Object.values(Model.Google).map((m) => [m, Provider.GOOGLE])
+  ...Object.values(Model.Google).map((m) => [m, Provider.GOOGLE]),
+  ...Object.values(Model.OpenRouter).map((m) => [m, Provider.OPENROUTER])
 ]);
 var VALID_PROVIDERS = Object.values(Provider);
 var PROVIDER_DEFAULT_REASONING = {
   openai: "medium",
   anthropic: "off",
-  google: "off"
+  google: "off",
+  // Varies by upstream model; the driver sends no reasoning field unless configured.
+  openrouter: "off"
 };
 var Content = {
   /** Detects Lorem ipsum, TODO, TBD, and similar placeholder text */
@@ -589,6 +608,13 @@ function mapEffort(level, model) {
   if (level !== "xhigh") return level;
   return XHIGH_CAPABLE_MODELS.has(model) ? "xhigh" : "max";
 }
+var BUDGET_THINKING_MODELS = /* @__PURE__ */ new Set([Model.Anthropic.HAIKU_4_5]);
+var EFFORT_TO_BUDGET_TOKENS = {
+  low: 1024,
+  medium: 4096,
+  high: 8192,
+  xhigh: 16384
+};
 var AnthropicDriver = class {
   client;
   model;
@@ -644,10 +670,16 @@ var AnthropicDriver = class {
         ]
       };
       if (this.reasoningEffort) {
-        requestParams.thinking = { type: "adaptive" };
-        requestParams.output_config = {
-          effort: mapEffort(this.reasoningEffort, this.model)
-        };
+        if (BUDGET_THINKING_MODELS.has(this.model)) {
+          const budgetTokens = EFFORT_TO_BUDGET_TOKENS[this.reasoningEffort];
+          requestParams.thinking = { type: "enabled", budget_tokens: budgetTokens };
+          requestParams.max_tokens = Math.max(this.maxTokens, budgetTokens + DEFAULT_MAX_TOKENS);
+        } else {
+          requestParams.thinking = { type: "adaptive" };
+          requestParams.output_config = {
+            effort: mapEffort(this.reasoningEffort, this.model)
+          };
+        }
       }
       const message = await client.messages.create(requestParams);
       const textBlock = message.content.find((block) => block.type === "text");
@@ -888,6 +920,109 @@ var OpenAIDriver = class {
   }
 };
 
+// src/providers/openrouter.ts
+var OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1";
+var OPENROUTER_REASONING_EFFORT = {
+  low: "low",
+  medium: "medium",
+  high: "high",
+  xhigh: "high"
+};
+var OpenRouterDriver = class {
+  client;
+  model;
+  maxTokens;
+  apiKeyOrEnv;
+  reasoningEffort;
+  constructor(config) {
+    this.model = config.model;
+    this.maxTokens = config.maxTokens;
+    this.client = null;
+    this.apiKeyOrEnv = config.apiKey;
+    this.reasoningEffort = config.reasoningEffort;
+  }
+  async getClient() {
+    if (this.client) return this.client;
+    let OpenAI;
+    try {
+      const mod = await import("openai");
+      OpenAI = mod.default;
+    } catch {
+      throw new VisualAIConfigError(
+        "OpenAI SDK not installed (required for the OpenRouter provider). Run: npm install openai"
+      );
+    }
+    const apiKey = this.apiKeyOrEnv ?? process.env.OPENROUTER_API_KEY;
+    if (!apiKey) {
+      throw new VisualAIAuthError(
+        "OpenRouter API key not found. Set OPENROUTER_API_KEY or pass apiKey in config."
+      );
+    }
+    this.client = new OpenAI({ apiKey, baseURL: OPENROUTER_BASE_URL });
+    return this.client;
+  }
+  async sendMessage(images, prompt, options) {
+    const client = await this.getClient();
+    const imageParts = images.map((img) => ({
+      type: "image_url",
+      image_url: { url: `data:${img.mimeType};base64,${img.base64}` }
+    }));
+    try {
+      const responseFormat = options?.responseSchema ? {
+        type: "json_schema",
+        json_schema: {
+          name: "visual_ai_response",
+          strict: true,
+          schema: options.responseSchema
+        }
+      } : { type: "json_object" };
+      const requestParams = {
+        model: this.model,
+        max_tokens: this.maxTokens,
+        response_format: responseFormat,
+        messages: [
+          {
+            role: "user",
+            content: [...imageParts, { type: "text", text: prompt }]
+          }
+        ],
+        // OpenRouter-specific: include token accounting in the response.
+        usage: { include: true }
+      };
+      if (this.reasoningEffort) {
+        requestParams.reasoning = { effort: OPENROUTER_REASONING_EFFORT[this.reasoningEffort] };
+      }
+      const response = await client.chat.completions.create(requestParams);
+      const choice = response.choices?.[0];
+      if (!choice?.message) {
+        throw new VisualAIProviderError("OpenRouter returned an empty response (no choices).");
+      }
+      const text = choice.message.content ?? "";
+      if (choice.finish_reason === "length") {
+        throw new VisualAITruncationError(
+          `Response truncated: OpenRouter returned finish_reason "length". The model exhausted the output token budget (${this.maxTokens} tokens). This commonly happens with higher reasoning effort levels. Increase maxTokens in your config (e.g., maxTokens: 16384) or lower reasoningEffort.`,
+          text,
+          this.maxTokens
+        );
+      }
+      const reasoningTokens = response.usage?.completion_tokens_details?.reasoning_tokens;
+      return {
+        text,
+        usage: response.usage ? {
+          inputTokens: response.usage.prompt_tokens,
+          outputTokens: response.usage.completion_tokens,
+          ...reasoningTokens !== void 0 && { reasoningTokens }
+        } : void 0
+      };
+    } catch (err) {
+      if (err instanceof VisualAITruncationError || err instanceof VisualAIProviderError) {
+        throw err;
+      }
+      throw mapProviderError(err);
+    }
+  }
+};
+
 // src/core/config.ts
 var MODEL_PREFIX_TO_PROVIDER = [
   ["claude-", "anthropic"],
@@ -900,6 +1035,7 @@ var MODEL_PREFIX_TO_PROVIDER = [
 function inferProviderFromModel(model) {
   const known = MODEL_TO_PROVIDER.get(model);
   if (known) return known;
+  if (model.includes("/")) return "openrouter";
   const prefixMatch = MODEL_PREFIX_TO_PROVIDER.find(([prefix]) => model.startsWith(prefix));
   return prefixMatch?.[1];
 }
@@ -912,12 +1048,13 @@ function resolveProvider(config) {
   const apiKeyProviderMap = [
     ["ANTHROPIC_API_KEY", "anthropic"],
     ["OPENAI_API_KEY", "openai"],
-    ["GOOGLE_API_KEY", "google"]
+    ["GOOGLE_API_KEY", "google"],
+    ["OPENROUTER_API_KEY", "openrouter"]
   ];
   const detected = apiKeyProviderMap.find(([key]) => process.env[key]);
   if (detected) return detected[1];
   throw new VisualAIConfigError(
-    "Cannot determine provider. Set a model name (config or VISUAL_AI_MODEL) or an API key env variable (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY)."
+    "Cannot determine provider. Set a model name (config or VISUAL_AI_MODEL) or an API key env variable (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, OPENROUTER_API_KEY)."
   );
 }
 function parseBooleanEnv(envName, value) {
@@ -945,11 +1082,11 @@ function resolveConfig(config) {
   }
   const userSetMaxTokens = config.maxTokens !== void 0;
   let maxTokens = config.maxTokens ?? DEFAULT_MAX_TOKENS;
-  if (!userSetMaxTokens && provider === "openai" && (config.reasoningEffort === "high" || config.reasoningEffort === "xhigh")) {
+  if (!userSetMaxTokens && (provider === "openai" || provider === "openrouter") && (config.reasoningEffort === "high" || config.reasoningEffort === "xhigh")) {
     maxTokens = OPENAI_REASONING_MAX_TOKENS;
     if (debug) {
       process.stderr.write(
-        `[visual-ai-assertions] Auto-increased maxTokens from ${DEFAULT_MAX_TOKENS} to ${OPENAI_REASONING_MAX_TOKENS} for OpenAI with reasoningEffort "${config.reasoningEffort}".
+        `[visual-ai-assertions] Auto-increased maxTokens from ${DEFAULT_MAX_TOKENS} to ${OPENAI_REASONING_MAX_TOKENS} for provider "${provider}" with reasoningEffort "${config.reasoningEffort}".
 `
       );
     }
@@ -1038,10 +1175,18 @@ var PRICING_TABLE = {
     inputPricePerToken: 0.25 / PER_MILLION,
     outputPricePerToken: 2 / PER_MILLION
   },
+  [`${Provider.GOOGLE}:${Model.Google.GEMINI_3_6_FLASH}`]: {
+    inputPricePerToken: 1.5 / PER_MILLION,
+    outputPricePerToken: 7.5 / PER_MILLION
+  },
   [`${Provider.GOOGLE}:${Model.Google.GEMINI_3_5_FLASH}`]: {
     inputPricePerToken: 1.5 / PER_MILLION,
     outputPricePerToken: 9 / PER_MILLION
   },
+  [`${Provider.GOOGLE}:${Model.Google.GEMINI_3_5_FLASH_LITE}`]: {
+    inputPricePerToken: 0.3 / PER_MILLION,
+    outputPricePerToken: 2.5 / PER_MILLION
+  },
   [`${Provider.GOOGLE}:${Model.Google.GEMINI_3_1_PRO_PREVIEW}`]: {
     inputPricePerToken: 2 / PER_MILLION,
     outputPricePerToken: 12 / PER_MILLION
@@ -1053,6 +1198,28 @@ var PRICING_TABLE = {
   [`${Provider.GOOGLE}:${Model.Google.GEMINI_3_FLASH_PREVIEW}`]: {
     inputPricePerToken: 0.5 / PER_MILLION,
     outputPricePerToken: 3 / PER_MILLION
+  },
+  // OpenRouter passes through upstream per-model pricing (verified 2026-07-22
+  // against https://openrouter.ai/api/v1/models).
+  [`${Provider.OPENROUTER}:${Model.OpenRouter.GROK_4_5}`]: {
+    inputPricePerToken: 2 / PER_MILLION,
+    outputPricePerToken: 6 / PER_MILLION
+  },
+  [`${Provider.OPENROUTER}:${Model.OpenRouter.KIMI_K3}`]: {
+    inputPricePerToken: 3 / PER_MILLION,
+    outputPricePerToken: 15 / PER_MILLION
+  },
+  [`${Provider.OPENROUTER}:${Model.OpenRouter.KIMI_K2_7_CODE}`]: {
+    inputPricePerToken: 0.82 / PER_MILLION,
+    outputPricePerToken: 3.75 / PER_MILLION
+  },
+  [`${Provider.OPENROUTER}:${Model.OpenRouter.QWEN_3_7_PLUS}`]: {
+    inputPricePerToken: 0.32 / PER_MILLION,
+    outputPricePerToken: 1.28 / PER_MILLION
+  },
+  [`${Provider.OPENROUTER}:${Model.OpenRouter.QWEN_3_6_FLASH}`]: {
+    inputPricePerToken: 0.1875 / PER_MILLION,
+    outputPricePerToken: 1.125 / PER_MILLION
   }
 };
 function calculateCost(provider, model, inputTokens, outputTokens) {
@@ -1130,7 +1297,8 @@ async function timedSendMessage(driver, images, prompt, options) {
 var import_sharp = __toESM(require("sharp"), 1);
 var DIFF_ALLOWED_MODELS = /* @__PURE__ */ new Set([
   Model.Google.GEMINI_3_FLASH_PREVIEW,
-  Model.Google.GEMINI_3_5_FLASH
+  Model.Google.GEMINI_3_5_FLASH,
+  Model.Google.GEMINI_3_6_FLASH
 ]);
 async function generateAiDiff(imgA, imgB, model, driver) {
   if (!driver.generateImage) {
@@ -1773,7 +1941,57 @@ function isVideoInput(input) {
   }
   return false;
 }
+function isFramesInput(input) {
+  return typeof input === "object" && input !== null && !Buffer.isBuffer(input) && !(input instanceof Uint8Array) && Array.isArray(input.frames);
+}
+function isTimestampedFrameInput(frame) {
+  return typeof frame === "object" && !Buffer.isBuffer(frame) && !(frame instanceof Uint8Array) && "image" in frame;
+}
+async function normalizeFrames(input) {
+  const rawFrames = input.frames;
+  const fps = input.fps ?? DEFAULT_FPS;
+  if (rawFrames.length === 0) {
+    throw new VisualAIVideoError("frames must be a non-empty array of image inputs");
+  }
+  if (rawFrames.length > MAX_FRAMES_HARD_CAP) {
+    throw new VisualAIVideoError(
+      `frames length ${rawFrames.length} exceeds the hard cap of ${MAX_FRAMES_HARD_CAP}. Pass fewer frames or open an issue if you need a larger limit.`
+    );
+  }
+  if (!Number.isFinite(fps) || fps <= 0) {
+    throw new VisualAIVideoError(`Invalid fps: ${fps}. Must be a finite number > 0.`);
+  }
+  const frames = await Promise.all(
+    rawFrames.map(async (raw, index) => {
+      const timestamped = isTimestampedFrameInput(raw);
+      const imageInput = timestamped ? raw.image : raw;
+      const explicit = timestamped ? raw.timestampSeconds : void 0;
+      const timestampSeconds = explicit ?? index / fps;
+      if (!Number.isFinite(timestampSeconds) || timestampSeconds < 0) {
+        throw new VisualAIVideoError(
+          `Invalid timestampSeconds for frame ${index}: ${String(timestampSeconds)}. Must be a finite number >= 0.`
+        );
+      }
+      const image = await normalizeImage(imageInput);
+      return {
+        data: image.data,
+        mimeType: image.mimeType,
+        get base64() {
+          return image.base64;
+        },
+        timestampSeconds,
+        index
+      };
+    })
+  );
+  const durationSeconds = frames.reduce((max, f) => Math.max(max, f.timestampSeconds), 0);
+  await saveDebugFrames(frames);
+  return { kind: "video", frames, durationSeconds };
+}
 async function normalizeMedia(input, videoOptions) {
+  if (isFramesInput(input)) {
+    return normalizeFrames(input);
+  }
   if (isVideoInput(input)) {
     const { path, cleanup } = await resolveVideoToPath(input);
     try {
@@ -1930,7 +2148,8 @@ function toSchemaOptions(schema) {
 var PROVIDER_REGISTRY = {
   anthropic: (config) => new AnthropicDriver(config),
   openai: (config) => new OpenAIDriver(config),
-  google: (config) => new GoogleDriver(config)
+  google: (config) => new GoogleDriver(config),
+  openrouter: (config) => new OpenRouterDriver(config)
 };
 function createDriver(provider, config) {
   return PROVIDER_REGISTRY[provider](config);