visual-ai-assertions 0.26.0 → 0.27.0

package/README.md

@@ -2,6 +2,8 @@
 
 AI-powered visual assertions for E2E tests. Send screenshots — or short video recordings — to Claude, GPT, Gemini — or Grok, Kimi, Qwen, and GLM via OpenRouter — and get structured, typed results.
 
+Every supported model is benchmarked against hand-labelled screenshots — **[see how they compare](https://nullp2ike.github.io/visual-reasoning/)** ([methodology](#benchmarks)).
+
 ## Installation
 
 ```bash
@@ -589,14 +591,14 @@ Set `VISUAL_AI_REASONING_EFFORT` to apply one without touching code; an explicit
 
 When omitted, each provider uses its default behavior. The `"xhigh"` level enables maximum reasoning depth. `"minimal"` is **not portable** — several OpenAI models reject it with HTTP 400 (including the `gpt-6.1-sol` default), and Google/OpenRouter clamp it to `low` rather than send it; use `"low"` as the floor unless you know the model accepts it.
 
-| Provider                                                  | Native Parameter                                      | `"xhigh"` maps to    |
-| --------------------------------------------------------- | ----------------------------------------------------- | -------------------- |
-| Anthropic (Fable 5.1/5, Opus 5.5/5/4.8/4.7, Sonnet 5.5/5) | `thinking.type: "adaptive"` + `output_config.effort`  | `effort: "xhigh"`    |
-| Anthropic (Opus 4.6, Sonnet 4.6)                          | `thinking.type: "adaptive"` + `output_config.effort`  | `effort: "max"`      |
-| Anthropic (Haiku 4.5)                                     | budget-based extended thinking (token budget)         | 16384-token budget   |
-| OpenAI                                                    | `reasoning.effort` (Responses API)                    | `effort: "xhigh"`    |
-| Google                                                    | `thinkingConfig.thinkingLevel` (1:1: low/medium/high) | `"high"` (max level) |
-| OpenRouter                                                | `reasoning.effort` (normalized low/medium/high)       | `effort: "high"`     |
+| Provider                                                             | Native Parameter                                      | `"xhigh"` maps to    |
+| -------------------------------------------------------------------- | ----------------------------------------------------- | -------------------- |
+| Anthropic (Fable 5.1/5, Opus 5.5/5/4.8/4.7, Sonnet 5.5/5, Haiku 5.5) | `thinking.type: "adaptive"` + `output_config.effort`  | `effort: "xhigh"`    |
+| Anthropic (Opus 4.6, Sonnet 4.6)                                     | `thinking.type: "adaptive"` + `output_config.effort`  | `effort: "max"`      |
+| Anthropic (Haiku 4.5)                                                | budget-based extended thinking (token budget)         | 16384-token budget   |
+| OpenAI                                                               | `reasoning.effort` (Responses API)                    | `effort: "xhigh"`    |
+| Google                                                               | `thinkingConfig.thinkingLevel` (1:1: low/medium/high) | `"high"` (max level) |
+| OpenRouter                                                           | `reasoning.effort` (normalized low/medium/high)       | `effort: "high"`     |
 
 ## Supported Models
 
@@ -604,19 +606,22 @@ All listed models support image/vision input. Pass any model ID to the `model` c
 
 ### Anthropic
 
-| Model             | Model ID            | Input $/MTok | Output $/MTok | Notes                                         |
-| ----------------- | ------------------- | ------------ | ------------- | --------------------------------------------- |
-| Claude Fable 5.1  | `claude-fable-5-1`  | $10          | $50           | Most capable; long-horizon agentic work       |
-| Claude Fable 5    | `claude-fable-5`    | $10          | $50           | Predecessor to Fable 5.1, same price          |
-| Claude Opus 5.5   | `claude-opus-5-5`   | $4           | $20           | Newest Opus; >30% faster output than Opus 5   |
-| Claude Opus 5     | `claude-opus-5`     | $5           | $25           | Previous Opus flagship; supports `xhigh`      |
-| Claude Opus 4.8   | `claude-opus-4-8`   | $5           | $25           | Prior Opus tier; supports `xhigh`             |
-| Claude Opus 4.7   | `claude-opus-4-7`   | $5           | $25           | Previous Opus; supports `xhigh` effort tier   |
-| Claude Opus 4.6   | `claude-opus-4-6`   | $5           | $25           | Previous flagship, 128K max output            |
-| Claude Sonnet 5.5 | `claude-sonnet-5-5` | $2           | $10           | **Default** — newest Sonnet; supports `xhigh` |
-| Claude Sonnet 5   | `claude-sonnet-5`   | $3           | $15           | Near-Opus quality on coding/agentic work      |
-| Claude Sonnet 4.6 | `claude-sonnet-4-6` | $3           | $15           | Prior default; best value in its generation   |
-| Claude Haiku 4.5  | `claude-haiku-4-5`  | $1           | $5            | Fastest, budget-friendly                      |
+| Model             | Model ID            | Input $/MTok | Output $/MTok | Notes                                             |
+| ----------------- | ------------------- | ------------ | ------------- | ------------------------------------------------- |
+| Claude Fable 5.1  | `claude-fable-5-1`  | $10          | $50           | Most capable; long-horizon agentic work           |
+| Claude Fable 5    | `claude-fable-5`    | $10          | $50           | Predecessor to Fable 5.1, same price              |
+| Claude Opus 5.5   | `claude-opus-5-5`   | $4           | $20           | Newest Opus; >30% faster output than Opus 5       |
+| Claude Opus 5     | `claude-opus-5`     | $5           | $25           | Previous Opus flagship; supports `xhigh`          |
+| Claude Opus 4.8   | `claude-opus-4-8`   | $5           | $25           | Prior Opus tier; supports `xhigh`                 |
+| Claude Opus 4.7   | `claude-opus-4-7`   | $5           | $25           | Previous Opus; supports `xhigh` effort tier       |
+| Claude Opus 4.6   | `claude-opus-4-6`   | $5           | $25           | Previous flagship, 128K max output                |
+| Claude Sonnet 5.5 | `claude-sonnet-5-5` | $2           | $10           | **Default** — newest Sonnet; supports `xhigh`     |
+| Claude Sonnet 5   | `claude-sonnet-5`   | $3           | $15           | Near-Opus quality on coding/agentic work          |
+| Claude Sonnet 4.6 | `claude-sonnet-4-6` | $3           | $15           | Prior default; best value in its generation       |
+| Claude Haiku 5.5  | `claude-haiku-5-5`  | $0.10        | $0.50         | Newest Haiku; cheapest Claude; supports `xhigh` † |
+| Claude Haiku 4.5  | `claude-haiku-4-5`  | $1           | $5            | Fastest, budget-friendly                          |
+
+† Haiku 5.5 prices apply to prompts up to 100K tokens; longer prompts bill at $0.50 / $2.50, far beyond screenshot-sized calls. Unlike Haiku 4.5 it uses adaptive thinking like the other current Claude models, and it thinks even when `reasoningEffort` is not set.
 
 ### OpenAI
 
@@ -689,6 +694,51 @@ Meta also publishes `meta/muse-spark-1.3-contributor`, the same model at $0.10 /
 
 `qwen/qwen3.7-max` and the DeepSeek V4 family (`deepseek/deepseek-v4-pro`, `deepseek/deepseek-v4-flash`, and dated variants such as `deepseek/deepseek-v4-pro-0813`) are not listed because they accept no image input on OpenRouter.
 
+## Benchmarks
+
+Which model should you actually pick? This repo benchmarks every supported model
+against hand-labelled screenshots and publishes the results:
+
+**📊 [Live results — nullp2ike.github.io/visual-reasoning](https://nullp2ike.github.io/visual-reasoning/)**
+
+The site carries the defect-discovery leaderboard, a screenshot × model matrix you
+can drill into for any individual answer, and the same runs graded independently by
+five different judges so you can see where the grading itself is contested.
+
+The benchmark asks each model one open question — "What looks visually broken on
+this page?" — with no hints, and an LLM judge matches what it reports against the
+defects seeded into each screenshot. The headline is recall of those defects,
+weighed against the extras a model invents per run: a missed defect is a bug that
+ships, and an invented one is noise someone has to triage.
+
+### The `golden` dataset
+
+18 screenshots — 17 with exactly one deliberately seeded defect, plus a clean
+control where anything reported counts as a false positive. The prompt names a
+short list of out-of-scope non-defects (edge-clipped carousel items, content cut
+off by the viewport bottom) so models are not penalised for reporting framing as
+breakage.
+
+Current coverage, at `medium` effort with 5 repeats per cell:
+
+- 38 model/effort/fidelity series over 3,420 graded runs
+
+A few results worth knowing before you choose a default:
+
+| Model                                     | Discovery recall | Extras / run | Cost / run |
+| ----------------------------------------- | ---------------- | ------------ | ---------- |
+| `claude-opus-5-5`                         | 100%             | 0.53         | $0.023     |
+| `claude-sonnet-5-5` _(Anthropic default)_ | 94%              | 2.81         | $0.011     |
+| `gpt-6.1-sol` _(OpenAI default)_          | 93%              | 0.02         | $0.0055    |
+| `gemini-3.8-flash` _(Google default)_     | 87%              | 0.34         | $0.0050    |
+
+Recall is not the whole picture. `claude-sonnet-5-5` matches the Fable tier on
+recall but reports 2.81 extras per run against `gpt-6.1-sol`'s 0.02 — for
+open-ended discovery that is a lot of noise to triage.
+
+Full methodology, how to run a sweep, and how to add your own dataset:
+[`bench/README.md`](bench/README.md).
+
 ## License
 
 MIT

package/dist/index.cjs

@@ -497,6 +497,7 @@ var Model = {
     SONNET_5_5: "claude-sonnet-5-5",
     SONNET_5: "claude-sonnet-5",
     SONNET_4_6: "claude-sonnet-4-6",
+    HAIKU_5_5: "claude-haiku-5-5",
     HAIKU_4_5: "claude-haiku-4-5"
   },
   OpenAI: {
@@ -716,7 +717,8 @@ var XHIGH_CAPABLE_MODELS = /* @__PURE__ */ new Set([
   Model.Anthropic.OPUS_4_8,
   Model.Anthropic.OPUS_4_7,
   Model.Anthropic.SONNET_5_5,
-  Model.Anthropic.SONNET_5
+  Model.Anthropic.SONNET_5,
+  Model.Anthropic.HAIKU_5_5
 ]);
 function mapEffort(level, model) {
   if (level === "minimal") return "low";
@@ -1428,6 +1430,12 @@ var PRICING_TABLE = {
     inputPricePerToken: 3 / PER_MILLION,
     outputPricePerToken: 15 / PER_MILLION
   },
+  // Prompts above 100K tokens bill at $0.50/$2.50, far beyond screenshot-sized
+  // calls. Cached input is $0.01/MTok and cache writes $0.125/MTok (not modelled).
+  [`${Provider.ANTHROPIC}:${Model.Anthropic.HAIKU_5_5}`]: {
+    inputPricePerToken: 0.1 / PER_MILLION,
+    outputPricePerToken: 0.5 / PER_MILLION
+  },
   [`${Provider.ANTHROPIC}:${Model.Anthropic.HAIKU_4_5}`]: {
     inputPricePerToken: 1 / PER_MILLION,
     outputPricePerToken: 5 / PER_MILLION