snifftest 0.1.0

package/bench/panel.yaml

@@ -0,0 +1,136 @@
+# The panel `snifftest bench` asks the same questions as the judgment rules.
+#
+# Nothing here is a claim that a model exists. Each entry carries a pattern and
+# a preference; on the day of the run every entry is matched against the
+# provider's own model list and the exact slug, its published prices and
+# whether it takes a JSON mode are written into the raw output. A model the
+# provider does not list becomes a row that says "not available on <date>" and
+# is never swapped for a near neighbour.
+#
+# Most of the panel is routed through OpenRouter so that one key reproduces the
+# whole bench and every row carries the same proxy hop. The last entry calls one
+# of those models directly, which is what makes that hop measurable rather than
+# an asterisk.
+#
+# Jev is in the rotation, as the last row. Its accuracy still comes from
+# `snifftest eval`, where arm C is scored against arms A and B over the same
+# seeded corpus and joined into the headline table with `--eval`; what the row
+# below adds is a latency measured in the same interleaved rotation, with the
+# same repeats, as every other row. Two latency numbers in one column measured
+# two different ways is not a comparison.
+#
+# A row may also name `alternates`. A preferred slug the provider has stopped
+# listing then falls to a substitute the panel file named, and the tables print
+# which one was taken; a row with no alternates says "not available" instead of
+# quietly measuring a neighbour.
+#
+# A row may also declare `max_tokens` and `reasoning`. Those two settings are
+# the only way the rows differ, they are there because a reasoning model spends
+# its internal tokens out of the same completion budget as its answer, and both
+# are printed under the tables so the difference is declared rather than hidden.
+
+version: 1
+
+prices:
+  # OpenRouter publishes a price per model, so only the direct row needs a file.
+  anthropic: prices/anthropic-2026-09-17.yaml
+  # No dated published price has been recorded for the judgment service, so no
+  # file is named here and its row prints its cost as unknown. A constant in
+  # this repo's source is not a source. When a dated one exists it goes in
+  # `prices/` beside the Anthropic file and this line points at it.
+
+models:
+  - id: haiku
+    label: Claude Haiku 4.5
+    tier: fast
+    provider: openrouter
+    match: "^anthropic/claude-haiku-4"
+    prefer:
+      - anthropic/claude-haiku-4.5
+
+  - id: sonnet
+    label: Claude Sonnet 5
+    tier: mid
+    provider: openrouter
+    match: "^anthropic/claude-sonnet-5"
+    prefer:
+      - anthropic/claude-sonnet-5
+
+  - id: opus
+    label: Claude Opus 5
+    tier: deep
+    provider: openrouter
+    match: "^anthropic/claude-opus-5"
+    prefer:
+      - anthropic/claude-opus-5
+
+  - id: openai-fast
+    label: OpenAI, fast tier
+    tier: fast
+    provider: openrouter
+    match: "^openai/gpt-5[^/]*mini$"
+    prefer:
+      - openai/gpt-5-mini
+
+  - id: gemini-flash
+    label: Gemini Flash
+    tier: fast
+    provider: openrouter
+    match: "^google/gemini-[0-9.]+-flash$"
+    prefer:
+      - google/gemini-2.5-flash
+
+  - id: deepseek
+    label: DeepSeek, chat tier
+    tier: fast
+    provider: openrouter
+    match: "^deepseek/deepseek-chat"
+    prefer:
+      - deepseek/deepseek-chat-v3
+    note: >
+      DeepSeek publishes no model called Flash; this is its chat tier, which is
+      the cheap-and-quick row the panel wanted.
+
+  - id: openai-deep
+    label: OpenAI, deep tier
+    tier: deep
+    provider: openrouter
+    match: "^openai/gpt-5(\\.[0-9]+)?$"
+    prefer:
+      - openai/gpt-5
+    # This row reasons, and on OpenRouter the reasoning tokens come out of the
+    # same completion budget as the answer. On the default budget it can spend
+    # the whole of it thinking and stop before it writes the JSON object, which
+    # lands in the table as a parse failure and reads as a verdict on the model
+    # rather than on the request. So the budget is raised and the effort is
+    # declared: low keeps the row affordable, and the footnote prints both, so
+    # nobody reads this row as the model's best.
+    max_tokens: 4000
+    reasoning:
+      effort: low
+
+  - id: sonnet-direct
+    label: Claude Sonnet 5, direct
+    tier: control
+    provider: anthropic
+    match: "^claude-sonnet-5"
+    prefer:
+      - claude-sonnet-5
+    note: >
+      The overhead control. Same model as the sonnet row, called without the
+      OpenRouter hop, so the pair measures the hop.
+
+  - id: jev
+    label: Jev, the judgment arm
+    tier: judgment
+    provider: jev
+    match: "^jev-"
+    prefer:
+      - jev-latest
+    note: >
+      Asked its own way, one request per paragraph carrying every rule as a
+      separate question, which is the shape the service documents and the shape
+      arm C uses. The request count matches the other rows; the request shape
+      does not. It returns a probability and no boolean, so its flag column is
+      that probability against the shipped threshold rather than a decision the
+      service made.

package/bench/prices/anthropic-2026-09-17.yaml

@@ -0,0 +1,33 @@
+# Published list prices for the Anthropic direct row, in US dollars per million
+# tokens.
+#
+# This file exists because the Messages API returns token counts and no prices,
+# and because a price written into source code is a number with no provenance
+# and no expiry. Every table that uses these numbers cites this file, its
+# source and the date below.
+#
+# `match` is a substring of the served model id, so a dated id such as
+# `claude-sonnet-5-20260101` lands on its family. A served model that matches no
+# row here has no cost at all in the report. It is never a zero, which would
+# read as "this call was free".
+#
+# Re-verify on the run date and write a new dated file rather than editing this
+# one; the run that used the old numbers should keep pointing at them.
+
+version: 1
+source: >-
+  Anthropic published list prices, transcribed on 2026-09-17; re-verify against
+  the published price list on the run date before quoting these figures.
+verified_on: 2026-09-17
+currency: usd_per_million_tokens
+
+models:
+  - match: haiku-4-5
+    input: 1
+    output: 5
+  - match: sonnet-5
+    input: 2
+    output: 10
+  - match: opus-5
+    input: 5
+    output: 25

package/bench/prices/openai-2026-09-17.yaml

@@ -0,0 +1,27 @@
+# Published list prices for the OpenAI direct rows, in US dollars per million
+# tokens.
+#
+# This file exists because the models endpoint returns ids and nothing else: no
+# prices, and no list of the parameters each model takes. A price written into
+# source code is a number with no provenance and no expiry, so it lives here
+# instead, and every table that uses these numbers cites this file, its source
+# and the date below.
+#
+# `match` is a substring of the served model id, so a dated id lands on its
+# family. A served model that matches no row here has no cost at all in the
+# report. It is never a zero, which would read as "this call was free".
+#
+# Re-verify on the run date and write a new dated file rather than editing this
+# one; the run that used the old numbers should keep pointing at them.
+
+version: 1
+source: >-
+  OpenAI published list prices (developers.openai.com/api/docs/pricing),
+  transcribed on 2026-09-17
+verified_on: 2026-09-17
+currency: usd_per_million_tokens
+
+models:
+  - match: gpt-5.6-sol
+    input: 4
+    output: 20