visual-ai-assertions 0.8.0 → 0.10.0

package/README.md

@@ -1,6 +1,6 @@
 # visual-ai-assertions
 
-AI-powered visual assertions for E2E tests. Send screenshots to Claude, GPT, or Gemini and get structured, typed results.
+AI-powered visual assertions for E2E tests. Send screenshots — or short video recordings — to Claude, GPT, or Gemini and get structured, typed results.
 
 ## Installation
 
@@ -112,9 +112,9 @@ const ai = visualAI({
 });
 ```
 
-### `ai.check(image, statements, options?)`
+### `ai.check(input, statements, options?)`
 
-Visual assertion. Returns `pass: true` only if ALL statements are true.
+Visual assertion against a screenshot or short video. Returns `pass: true` only if ALL statements are true. For video inputs, a statement passes when it is true at any sampled frame.
 
 ```typescript
 // Single statement
@@ -130,6 +130,12 @@ const result = await ai.check(screenshot, [
 const result = await ai.check(screenshot, ["The form is submitted"], {
   instructions: ["Ignore loading spinners that appear briefly"],
 });
+
+// Video input — statement is true if it ever happens during the clip
+const result = await ai.check("./recording.webm", [
+  'A success toast with text "Saved" briefly appears',
+]);
+console.log(result.statements[0].timestampSeconds); // e.g. 3.5
 ```
 
 **Returns:** `CheckResult`
@@ -149,9 +155,9 @@ const result = await ai.check(screenshot, ["The form is submitted"], {
 }
 ```
 
-### `ai.ask(image, prompt, options?)`
+### `ai.ask(input, prompt, options?)`
 
-Free-form analysis. Returns structured issues with priority and category.
+Free-form analysis of an image or video. Returns structured issues with priority and category. Video inputs are sampled into a frame timeline; the result includes `frameReferences` indicating which frames the model relied on.
 
 ```typescript
 const result = await ai.ask(screenshot, "Analyze this page for UI issues");
@@ -312,6 +318,33 @@ await ai.check("https://example.com/screenshot.png", "...");
 
 Oversized images are automatically resized to provider limits.
 
+### Video Input
+
+`ai.check()` and `ai.ask()` also accept short video recordings (`.mp4`, `.webm`, `.mov`, `.mkv`) — useful for asserting on transient UI like toast messages. Accepted shapes are file path, `data:video/...;base64,...` URL, raw base64 string, `Buffer`, and `Uint8Array`. HTTP/HTTPS URLs are not supported for video inputs — fetch the bytes yourself first.
+
+```typescript
+// Playwright recording on disk
+const result = await ai.check("./trace/video/recording.webm", [
+  'A success toast with text "Saved" briefly appears',
+]);
+
+// Result includes frame metadata + per-statement timestamps
+console.log(result.frames);
+// { count: 4, timestampsSeconds: [0.5, 1.5, 2.5, 3.5], durationSeconds: 4.0 }
+console.log(result.statements[0].timestampSeconds); // 3.5
+
+// Override sampling — defaults are 1 fps, max 10 frames, max 10 s of video
+await ai.check("./long-clip.mp4", ["Loader disappears"], {
+  video: { fps: 2, maxFrames: 20, maxDurationSeconds: 15 },
+});
+```
+
+`maxFrames` is hard-capped at 60 to keep memory bounded. Frames are downscaled so the longer edge fits within 1568 px before being sent to the provider.
+
+How it works: the library samples frames with ffmpeg and sends them to the provider as an ordered timeline. A statement passes when it is true at any sampled frame, unless its wording specifies otherwise (e.g. "throughout"). Template helpers (`accessibility`, `layout`, `pageLoad`, `content`, `elementsVisible`, `elementsHidden`) are image-only — pass video to `check()` or `ask()` instead.
+
+**ffmpeg setup.** Video support works out of the box — `fluent-ffmpeg`, `@ffmpeg-installer/ffmpeg`, and `@ffprobe-installer/ffprobe` ship as regular dependencies and bundle platform-specific ffmpeg/ffprobe binaries. If you ran `npm install` you already have everything you need. On platforms where the prebuilt binary is unavailable (or if you've pruned dependencies), `check()` and `ask()` throw `VisualAIVideoError` (import from `visual-ai-assertions` to `instanceof`-narrow it) when called with video input.
+
 ### Formatting & Assertion Helpers
 
 ```typescript
@@ -356,6 +389,9 @@ try {
       case "IMAGE_INVALID":
         // Invalid image: corrupt, unsupported format, etc.
         break;
+      case "VIDEO_INVALID":
+        // Invalid video: missing ffmpeg deps, oversized clip, decode failure, etc.
+        break;
       case "RESPONSE_PARSE_FAILED":
         // AI returned unparseable response — error.rawResponse has raw text
         break;
@@ -387,13 +423,15 @@ The `VisualAIKnownError` union and `isVisualAIKnownError()` helper are useful wh
 
 ### Optional Configuration
 
-| Variable                   | Description                                                                                                    |
-| -------------------------- | -------------------------------------------------------------------------------------------------------------- |
-| `VISUAL_AI_MODEL`          | Default model when `model` is not set in config. Overrides the provider's default model.                       |
-| `VISUAL_AI_DEBUG`          | Enable error diagnostic logging to stderr. Does **not** enable prompt/response logging. Use `"true"` or `"1"`. |
-| `VISUAL_AI_DEBUG_PROMPT`   | Enable prompt-only debug logging to stderr. Use `"true"` or `"1"`.                                             |
-| `VISUAL_AI_DEBUG_RESPONSE` | Enable response-only debug logging to stderr. Use `"true"` or `"1"`.                                           |
-| `VISUAL_AI_TRACK_USAGE`    | Enable usage tracking (token counts and cost) to stderr. Use `"true"` or `"1"`.                                |
+| Variable                     | Description                                                                                                                                                                                                                        |
+| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `VISUAL_AI_MODEL`            | Default model when `model` is not set in config. Overrides the provider's default model.                                                                                                                                           |
+| `VISUAL_AI_DEBUG`            | Enable error diagnostic logging to stderr. Does **not** enable prompt/response logging. Use `"true"` or `"1"`.                                                                                                                     |
+| `VISUAL_AI_DEBUG_PROMPT`     | Enable prompt-only debug logging to stderr. Use `"true"` or `"1"`.                                                                                                                                                                 |
+| `VISUAL_AI_DEBUG_RESPONSE`   | Enable response-only debug logging to stderr. Use `"true"` or `"1"`.                                                                                                                                                               |
+| `VISUAL_AI_DEBUG_FRAMES`     | Persist sampled video frames to disk for offline inspection. Use `"true"` or `"1"`. Frames are written to `./visual-ai-debug-frames/<timestamp>-<id>/` (override path with the next variable). Has no effect on image-only inputs. |
+| `VISUAL_AI_DEBUG_FRAMES_DIR` | Override the base directory for `VISUAL_AI_DEBUG_FRAMES`. Each call still gets its own timestamped subdirectory inside it.                                                                                                         |
+| `VISUAL_AI_TRACK_USAGE`      | Enable usage tracking (token counts and cost) to stderr. Use `"true"` or `"1"`.                                                                                                                                                    |
 
 ## Configuration
 
@@ -415,7 +453,12 @@ import type {
   AskResult,
   CheckResult,
   CompareResult,
+  Frame,
+  MediaInput,
   SupportedMimeType,
+  SupportedVideoMimeType,
+  VideoFramesMetadata,
+  VideoSamplingOptions,
   VisualAIConfig,
   VisualAIErrorCode,
 } from "visual-ai-assertions";