visual-ai-assertions 0.22.0 → 0.25.0

package/README.md

@@ -1,6 +1,6 @@
 # visual-ai-assertions
 
-AI-powered visual assertions for E2E tests. Send screenshots — or short video recordings — to Claude, GPT, Gemini — or Grok, Kimi, and Qwen via OpenRouter — and get structured, typed results.
+AI-powered visual assertions for E2E tests. Send screenshots — or short video recordings — to Claude, GPT, Gemini — or Grok, Kimi, Qwen, and GLM via OpenRouter — and get structured, typed results.
 
 ## Installation
 
@@ -11,7 +11,7 @@ npm install visual-ai-assertions
 # Optional: install additional provider SDKs
 npm install @anthropic-ai/sdk    # for Claude
 npm install @google/genai        # for Gemini
-# OpenRouter (Grok, Kimi, Qwen, ...) uses the OpenAI SDK — no extra install
+# OpenRouter (Grok, Kimi, Qwen, GLM, ...) uses the OpenAI SDK — no extra install
 
 # Zod is a peer dependency
 npm install zod
@@ -352,16 +352,22 @@ Oversized images are automatically resized to provider limits.
 
 `ai.check()` and `ai.ask()` also accept short video recordings (`.mp4`, `.webm`, `.mov`, `.mkv`) — useful for asserting on transient UI like toast messages. Accepted shapes are file path, `data:video/...;base64,...` URL, raw base64 string, `Buffer`, and `Uint8Array`. HTTP/HTTPS URLs are not supported for video inputs — fetch the bytes yourself first.
 
+How the video reaches the model depends on the provider. **Google models receive the video itself** (native delivery): Gemini samples and tokenises it server-side and also hears the audio track. **Every other provider gets frames sampled with ffmpeg** and sent as an ordered image timeline. Both paths enforce the same duration cap before any provider call, and both give you per-statement timestamps.
+
 ```typescript
 // Playwright recording on disk
 const result = await ai.check("./trace/video/recording.webm", [
   'A success toast with text "Saved" briefly appears',
 ]);
+console.log(result.statements[0].timestampSeconds); // 3.5
+
+// Google model → the video was sent natively
+console.log(result.video);
+// { durationSeconds: 4.0, fps: 1, mimeType: "video/webm", delivery: "inline" }
 
-// Result includes frame metadata + per-statement timestamps
+// Any other provider → frames were sampled
 console.log(result.frames);
-// { count: 4, timestampsSeconds: [0.5, 1.5, 2.5, 3.5], durationSeconds: 4.0 }
-console.log(result.statements[0].timestampSeconds); // 3.5
+// { count: 3, timestampsSeconds: [0.5, 2.5, 3.5], durationSeconds: 4.0, droppedUnchanged: 1 }
 
 // Override sampling — defaults are 1 fps, max 10 frames, max 10 s of video
 await ai.check("./long-clip.mp4", ["Loader disappears"], {
@@ -369,9 +375,37 @@ await ai.check("./long-clip.mp4", ["Loader disappears"], {
 });
 ```
 
-`maxFrames` is hard-capped at 60 to keep memory bounded. Frames are downscaled so the longer edge fits within 1568 px before being sent to the provider.
+`fps` applies to both paths: it is the ffmpeg sampling rate, and for native delivery it is passed to the provider as its sampling rate. `maxFrames` and `dedupe` apply to frame sampling only; `maxFrames` is hard-capped at 60 to keep memory bounded, and frames are downscaled so the longer edge fits within 1568 px before being sent.
+
+**Choosing the delivery.** `video.mode` is `"auto"` by default: native where the provider supports it, frames elsewhere. Set `"frames"` to sample frames on a Google model too, or `"native"` to insist on native delivery — that throws `VisualAIConfigError` on providers without it.
+
+```typescript
+// Sample frames on Gemini instead of sending the video
+await ai.check("./clip.mp4", ["Loader disappears"], { video: { mode: "frames" } });
+
+// Fail loudly if the configured model cannot take video natively
+await ai.ask("./clip.mp4", "What happens?", { video: { mode: "native" } });
+```
+
+Native delivery is worth preferring where available. Measured on a 15 s recording with a seven-statement `check()` on Gemini 3.8 Flash, native video at 1 fps cost about 60% of the frames path with dedupe on and answered about 1.5 s faster, with identical verdicts and correct timestamps on every run; Gemini tokenises a video frame at roughly 60 tokens against about 1,080 for the same frame sent as an image. Videos over 19 MB are uploaded through the Gemini Files API automatically (`delivery: "file"`) and deleted again afterwards. For `ask()`, native results carry `timestampReferences` (seconds) in place of the frame-indexed `frameReferences`.
 
-How it works: the library samples frames with ffmpeg and sends them to the provider as an ordered timeline. A statement passes when it is true at any sampled frame, unless its wording specifies otherwise (e.g. "throughout"). Template helpers (`accessibility`, `layout`, `pageLoad`, `content`, `elementsVisible`, `elementsHidden`) are image-only — pass video to `check()` or `ask()` instead.
+**Unchanged frames are dropped.** A recording of a mostly static screen would otherwise pay input tokens for every near-identical sample, so by default each sampled frame is compared against the most recently kept frame and dropped when fewer than 0.1% of its pixels changed — roughly a 37x37 px region on a 1568x880 frame. A toast, a loader, or a dialog comfortably clears that; compression noise and a blinking text caret do not. The first frame is always kept, `durationSeconds` still covers the whole clip, and the prompt tells the model how many frames were dropped and that the screen stayed unchanged between the listed timestamps. `result.frames.droppedUnchanged` reports how many were dropped.
+
+```typescript
+// Keep every sampled frame
+await ai.check("./clip.webm", ["A spinner is visible throughout"], {
+  video: { dedupe: false },
+});
+
+// Keep smaller changes (fraction of pixels that must change, in (0, 1])
+await ai.check("./clip.webm", ["The 24 px status dot turns green"], {
+  video: { dedupe: { threshold: 0.0002 } },
+});
+```
+
+A change smaller than the threshold — a lone 24x24 px icon on a full-size frame is about 0.04% — is treated as no change, so lower `threshold` or pass `dedupe: false` when the assertion is about something that small.
+
+How it works: on the frames path the library samples frames with ffmpeg, drops the ones that did not visibly change, and sends the rest to the provider as an ordered timeline; on the native path it probes the duration, then hands the provider the video and a matching prompt. Either way a statement passes when it is true at any point, unless its wording specifies otherwise (e.g. "throughout"). Template helpers (`accessibility`, `layout`, `pageLoad`, `content`, `elementsVisible`, `elementsHidden`) are image-only — pass video to `check()` or `ask()` instead.
 
 **ffmpeg setup.** Video support works out of the box — `fluent-ffmpeg`, `@ffmpeg-installer/ffmpeg`, and `@ffprobe-installer/ffprobe` ship as regular dependencies and bundle platform-specific ffmpeg/ffprobe binaries. If you ran `npm install` you already have everything you need. On platforms where the prebuilt binary is unavailable (or if you've pruned dependencies), `check()` and `ask()` throw `VisualAIVideoError` (import from `visual-ai-assertions` to `instanceof`-narrow it) when called with video input.
 
@@ -384,7 +418,7 @@ const frames = [await page.screenshot(), await page.screenshot()];
 
 const result = await ai.check({ frames }, ['A success toast with text "Saved" appears']);
 console.log(result.frames);
-// { count: 2, timestampsSeconds: [0, 1], durationSeconds: 1 }
+// { count: 2, timestampsSeconds: [0, 1], durationSeconds: 1, droppedUnchanged: 0 }
 
 // Control timestamps: give an explicit fps, or per-frame timestampSeconds
 await ai.ask(
@@ -398,7 +432,7 @@ await ai.ask(
 );
 ```
 
-Timestamps for bare frames are derived as `index / fps` (default `fps` is `1`); a per-frame `timestampSeconds` overrides that. The frame count is subject to the same 60-frame hard cap as video sampling, and the `video` sampling option is ignored for this path.
+Timestamps for bare frames are derived as `index / fps` (default `fps` is `1`); a per-frame `timestampSeconds` overrides that. The frame count is subject to the same 60-frame hard cap as video sampling, and the `video` sampling option is ignored for this path — pre-sampled frames are always sent as frames, even to Google models. Frames that did not visibly change from the preceding kept frame are dropped exactly as for video input; control it with `dedupe` on the `FramesInput` itself, e.g. `{ frames, dedupe: false }` or `{ frames, dedupe: { threshold: 0.0002 } }`.
 
 ### Formatting & Assertion Helpers
 
@@ -511,7 +545,10 @@ import type {
   CheckResult,
   CompareResult,
   Frame,
+  FrameDedupeOptions,
   MediaInput,
+  NativeVideoMetadata,
+  VideoDeliveryMode,
   SupportedMimeType,
   SupportedVideoMimeType,
   VideoFramesMetadata,
@@ -622,9 +659,12 @@ Any [OpenRouter](https://openrouter.ai/models) model slug (always `vendor/model`
 | Qwen3.8 Max    | `qwen/qwen3.8-max`          | $2           | $6            | First Max tier with image input       |
 | Qwen3.7 Plus   | `qwen/qwen3.7-plus`         | $0.32        | $1.28         | Cost-effective, GUI/screen-reading    |
 | Qwen3.6 Flash  | `qwen/qwen3.6-flash`        | $0.19        | $1.13         | **Default** — cheap flash vision tier |
+| GLM 5.3 Flash  | `z-ai/glm-5.3-flash`        | $0.15        | $0.50         | Z.ai flash tier, 1.3M context²        |
 
 ¹ Muse Spark 1.3 is age-gated by OpenRouter: calls return HTTP 403 (`VisualAIAuthError`) until the account completes the 18+ confirmation at [openrouter.ai/settings/preferences](https://openrouter.ai/settings/preferences). It also reasons by default — expect several hundred reasoning tokens per call even with no `reasoningEffort` set.
 
+² GLM 5.3 Flash reasons by default — expect one to two hundred reasoning tokens per call even with no `reasoningEffort` set, billed at the output rate. OpenRouter's own context cap for it is 1,048,576 tokens (Z.ai lists 1,310,720) and its output ceiling is 131,072.
+
 Meta also publishes `meta/muse-spark-1.3-contributor`, the same model at $0.10 / $0.20 per MTok — about 12x cheaper — because Meta uses everything submitted through it for product improvement. It has **no named constant** (`Model.OpenRouter` does not expose it) and never appears by default anywhere in this library, so using it takes a deliberate, explicit choice: pass the slug directly as a plain string, `visualAI({ model: "meta/muse-spark-1.3-contributor" })`. Any OpenRouter slug works this way — see the note above the table — and cost tracking works correctly once you opt in. OpenRouter itself blocks it with HTTP 404 (`paid-model-training-violation-by-account`) until the account's privacy settings allow training endpoints, at [openrouter.ai/settings/privacy](https://openrouter.ai/settings/privacy). Only use it if sending your screenshots to Meta for training is a trade you've deliberately made.
 
 `qwen/qwen3.7-max` and the DeepSeek V4 family (`deepseek/deepseek-v4-pro`, `deepseek/deepseek-v4-flash`, and dated variants such as `deepseek/deepseek-v4-pro-0813`) are not listed because they accept no image input on OpenRouter.