snifftest 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dan Willoughby
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,372 @@
+<p align="center">
+  <img src="assets/nose/concepts/e-sketch.svg" alt="A pencil sketch of a nose in profile" width="160">
+</p>
+
+# Sniff Test
+
+Does the draft pass the sniff test?
+
+Sniff Test reads Markdown and plain text and checks it against your house
+rules. It prints one line per flag with the file, the line, the rule that
+tripped and how sure it is.
+
+There are two kinds of rule. **Countable rules** are regular expressions.
+Long dashes, a paragraph with three colons, a banned word, every sentence the
+same length. They run on your machine, cost nothing and send nothing anywhere.
+**Judgment rules** need something read rather than counted. A closing sentence
+that only restates the paragraph, a claim hedged three times, a line that talks
+the reader out of the offer. For those it sends one paragraph at a time to a
+small hosted judgment model and gets back one probability per rule. It only
+does that after you have said yes.
+
+Two things it is not. There is no general model in the loop; the judgment
+model returns one probability per question and nothing else. And it will not
+rewrite your text. A flag is a sentence for you to fix.
+
+```sh
+npm install --global snifftest
+snifftest check --dry-run draft.md
+```
+
+That runs the countable rules and nothing leaves your machine. To run the
+judgment rules too, put a TypeSafe key in `TYPESAFE_API_KEY` and drop
+`--dry-run`. The first time, it prints exactly what it is about to send and
+where, and waits for your answer.
+
+It exits 0 when nothing tripped, 1 when at least one rule did, 2 when the tool
+could not do its job, and 3 when the judgment rules needed a yes and did not
+get one.
+
+The nose is the mascot. It is one hand-drawn SVG with six expressions, in
+`assets/nose/`, and `snifftest serve` opens a local page where you type and the
+nose reacts.
+
+## Reading a flag
+
+```
+draft.md:14 dash_present 1.00 A long dash. Give the sentence a full stop instead.
+draft.md:31 restating_closer 0.84 The last sentence says the paragraph again. End one sentence earlier.
+```
+
+The number is how sure the checker is, between 0 and 1, and it says nothing
+about how bad the problem is. A countable rule always scores 1.00. A judgment
+rule counts as a flag at or above the threshold, which is 0.7 unless the
+ruleset or `--threshold` says otherwise. A reading between 0.4 and 0.6 is
+reported as no judgment and never becomes a flag or a pass. That band exists
+because the judgment model answers about 0.5 on text it cannot read, and a
+flat middle number is the one failure that looks like a clean draft.
+
+`--format json` prints the same flags for a machine, with every reading
+including the ones under the threshold. `snifftest rules` prints the ruleset in
+force and which file each rule came from, which is the quickest answer to "why
+did that flag".
+
+## The rules
+
+The default ruleset ships in `rules/default.yaml`. Fifteen rules, and the file
+itself is the documentation. Ten apply to most prose and five are conventions
+of copy meant to sell.
+
+| Rule | Kind | What it catches |
+| --- | --- | --- |
+| `dash_present` | countable | An en dash or an em dash. A hyphen is fine, and so is an en dash between numerals. |
+| `colon_heavy` | countable | Three or more colons in one paragraph. URLs and clock times do not count. |
+| `sentence_rhythm` | countable | Every sentence the same length, in paragraphs of four sentences and sixty words or more. |
+| `slop_vocab` | countable | Twelve words that turn up far more often in generated text than in prose a person wrote. |
+| `banned_words` | countable | Your own list. Empty until you fill it. |
+| `not_x_but_y` | judgment | A sentence whose whole job is to swap one label for another. |
+| `tricolon` | judgment | Three items for the cadence. |
+| `stacked_hedging` | judgment | Two or more softeners on one claim. |
+| `rhetorical_opener` | judgment | A paragraph that opens on a question nobody asked. |
+| `restating_closer` | judgment | A last sentence that says the paragraph again. |
+| `self_undercutting` | judgment, marketing | Copy that talks the reader out of the thing on offer. |
+| `first_x_that` | judgment, marketing | A claim to be the first. |
+| `naked_cost_figure` | judgment, marketing | A cost with no customer price and no alternative beside it. |
+| `jobs_claim` | judgment, marketing | A claim about employment. |
+| `pullquote_fragment` | judgment, marketing | A display quote with no verb. |
+
+The five rules tagged `marketing` sit out an ordinary run. Ask for them on a
+landing page with `--only marketing`, which runs those five and nothing else.
+To run everything, write a project file that extends the default and sets
+`off_by_default: []`.
+
+Fenced code is dropped before either kind of rule sees it. Headings, tables,
+front matter, link definitions and HTML comments are never sent to the
+judgment rules, and the countable rules only read the ones a rule names.
+
+### Writing your own
+
+A `.snifftest.yaml` in your project root is picked up on every run. It can
+extend the default ruleset and change the rules it disagrees with, matched by
+id, so a house that wants five colons instead of three writes one rule and
+inherits the other fourteen.
+
+```yaml
+version: 1
+extends: default
+threshold: 0.7
+
+rules:
+  # A countable rule of your own is a pattern and a message.
+  - id: no_utilize
+    kind: regex
+    pattern: "\\butili[sz]e"
+    flags: i
+    message: "Use. The word is use."
+
+  # Override a shipped rule by reusing its id.
+  - id: colon_heavy
+    kind: regex
+    builtin: colon_count
+    min: 5
+    message: "Five colons in one paragraph."
+
+  # A judgment rule is a description, its near misses, and two criteria.
+  - id: passive_apology
+    kind: judgment
+    what: |
+      A sentence that apologises for the document itself: for its length,
+      its lateness, or the writer's lack of expertise.
+    not_for: |
+      An apology to a person for a thing that happened.
+      A stated limit of scope.
+    examples:
+      - "Sorry this is so long."
+      - "I am no expert, but here goes."
+    criteria:
+      true: "At least one sentence apologises for the document or the writer."
+      false: "No sentence apologises for the document or the writer."
+    message: "The draft is apologising for itself. Cut the line."
+```
+
+A countable rule names either a `builtin` or a `pattern`, never both. The
+built-ins are `dash_present`, `colon_count`, `sentence_rhythm`, `slop_vocab`
+and `banned_words`, and each takes the tuning keys the default file shows. A
+rule may carry `tags`, and a countable rule may name the kinds of block it
+applies to with `chunks`. A judgment rule's `not_for` is what keeps it honest.
+A rule with a `what` and no `not_for` will flag things you did not mean.
+
+A pattern with a nested quantifier is refused when the ruleset loads, and
+every pattern runs against a capped slice of text. Neither is a proof that a
+pattern cannot run for a long time. See `SECURITY.md`.
+
+To propose a rule for the shipped set, open a rule proposal issue.
+`CONTRIBUTING.md` says what one needs.
+
+## Consent, and what leaves the machine
+
+Nothing is sent before the answer to one question is yes. The countable rules
+run first, on your machine, whatever happens next. Then, if the ruleset has
+judgment rules and you did not pass `--dry-run`, the checker looks for a key in
+`TYPESAFE_API_KEY`. Without one it prints the countable verdict, says the
+judgment rules could not run, and exits 2. With one, it prints this and waits:
+
+```
+Sniff Test is about to use the judgment rules, which run on a model.
+
+  What leaves this machine: one paragraph of your text at a time, from 3 files,
+    together with the wording of these rules: not_x_but_y, tricolon, stacked_hedging, rhetorical_opener, restating_closer.
+  Where it goes: https://api.typesafe.ai/v1/systemone (TypeSafe), over HTTPS, with your TYPESAFE_API_KEY.
+  What comes back: one probability per rule per paragraph.
+  What is never sent: file names, file paths, anything outside the text you pointed at.
+  There is no telemetry, no analytics, and nothing is stored by this tool.
+
+  Countable rules never leave the machine. Run with --dry-run to use only those.
+
+Send paragraphs to TypeSafe? [y/N]
+```
+
+A yes is remembered under `~/.config/snifftest/consent.json`, or under
+`$XDG_CONFIG_HOME` when that is set. It is never written into the directory
+being checked, because a consent file committed to a repository would answer
+for everyone who clones it. `--yes` answers and remembers without the prompt.
+`SNIFFTEST_SEND=1` answers for one run in CI and writes nothing down. With no
+terminal to ask in and no answer in the environment, the checker exits 3 and
+sends nothing.
+
+Each request carries one paragraph and every judgment rule's wording. It goes
+over HTTPS with your key as a bearer token, and the reply is a probability per
+rule. The wire format and the guards around it are in `src/jev.ts`, which is
+the only file in the tool that opens a network connection for `check`.
+
+Answers are cached on disk so that a run interrupted by an outage does not pay
+for every paragraph again. The cache lives under `~/.cache/snifftest`, or
+`$XDG_CACHE_HOME`, or `SNIFFTEST_CACHE_DIR`. Each entry is named by a hash of
+the paragraph, the exact wording of the questions asked about it and the model
+they were sent to, and it holds rule ids, probabilities, the model name and a
+date. The paragraph itself is never
+written to disk. Entries expire after a fortnight. `--no-cache` skips the cache
+for a run, and `SNIFFTEST_CACHE_DIR=off` turns it off for good.
+
+`SECURITY.md` has the full account, including the limits.
+
+## Before a commit
+
+One command, from the root of your repository:
+
+```sh
+curl -fsSL https://raw.githubusercontent.com/DanRWilloughby/snifftest/v0.1.0/hooks/pre-commit \
+  -o .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit
+```
+
+Read it before you trust it. The hook checks the Markdown and text files you
+staged, as you staged them, with the countable rules only. It sends nothing
+unless `SNIFFTEST_SEND=1` and `TYPESAFE_API_KEY` are both in your environment.
+It blocks a commit on a flag. When the checker itself is missing, broken or
+could not be fetched, it says so and steps aside, because a failed download is
+not evidence about your prose. `SNIFFTEST_STRICT=1` reverses that.
+
+Skip it once with `SNIFFTEST_SKIP=1 git commit` or `git commit --no-verify`.
+
+The same script works under husky, and there are two entries for the
+pre-commit framework, `snifftest` for the free rules and `snifftest-send` under
+the manual stage for the rest. `docs/husky.md` has all three.
+
+## In CI
+
+```yaml
+name: Prose
+on: pull_request
+
+permissions: {}
+
+jobs:
+  snifftest:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.0.0
+      - uses: DanRWilloughby/snifftest@v0.1.0
+        with:
+          paths: docs
+```
+
+The Action runs the countable rules and nothing else unless you set `send` and
+pass a key in as `api-key`. A pull request from a fork always gets the
+countable rules only, whatever the workflow says. The Action refuses to run on
+any event whose name ends in `_target`, refuses a floating `version`, and
+fetches the exact version named from the npm registry into a scratch directory
+outside the checkout, so the repository being checked cannot hand it a checker
+by committing one. It needs no token of its own. With `comment: true` and a `github-token` it posts
+the checker's output on the pull request, and no line of the draft is copied
+into that comment.
+
+Pin the Action by commit SHA if you want the stronger guarantee. The inputs and
+the exit codes are tabled in `docs/husky.md`.
+
+## In Claude Code
+
+```
+/plugin marketplace add DanRWilloughby/snifftest
+/plugin install snifftest@snifftest
+```
+
+Then `/snifftest draft.md`, or hand Claude a draft and ask whether it passes.
+The skill runs the countable rules by default. The judgment pass is the user's
+call and the skill never answers the sending question on their behalf: it adds
+no flag that skips the prompt and never touches the key. `docs/claude-code.md`
+has the two other ways to install it.
+
+## Measuring it
+
+`snifftest eval` plants one known fault per rule into copies of your own clean
+paragraphs, runs the corpus three ways, and reports what each way caught and
+what it flagged on the clean originals. Arm A is no tool. Arm B is the
+countable rules. Arm C is the countable rules plus the judgment rules, and it
+is the only arm that costs anything. Each run writes a Markdown report and a
+`scores.json`, and it writes the clean and seeded paragraphs beside them with
+a `.gitignore`, because they carry your prose in full.
+
+`snifftest bench` asks a panel of general models the same questions over the
+same corpus, through OpenRouter and one direct Anthropic call, and puts cost,
+speed and accuracy side by side. An OpenAI row can be called directly too, on
+OpenAI's own API with `OPENAI_API_KEY`, so a model can be measured without an
+OpenRouter account and its latency is the provider's rather than a proxy's.
+`bench/panel-direct.yaml` is a panel of nothing but direct rows. It asks for
+consent per destination, and a yes given for TypeSafe is never a yes for anyone
+else.
+
+The run committed under `bench/results/2026-09-17/` is the eval over the
+packaged corpus with eight seeds per rule, and beside it the bench over the
+same 166 paragraphs with `bench/panel-direct.yaml`: four general models, each
+on its own provider's API, on the same day.
+
+<!-- numbers:start -->
+| Arm | Judgment faults caught, of 80 | Clean paragraphs flagged, of 54 | Cost per 100 paragraphs | Median per paragraph |
+| --- | --- | --- | --- | --- |
+| Sniff Test, judgment model (Jev, measured) | 63 | 1 | $0.0129 | 182 ms |
+| Claude Haiku 4.5 (measured) | 66 | 37 | $0.43 | 1,971 ms |
+| Claude Sonnet 5 (measured) | 72 | 2 | $1.34 | 6,083 ms |
+| Claude Opus 5 (measured) | 77 | 0 | $3.08 | 6,532 ms |
+| OpenAI gpt-5.6-sol (measured) | 73 | 0 | $1.64 | 4,428 ms |
+<!-- numbers:end -->
+
+Every row is one paragraph per request. The Sniff Test row is the eval's arm
+C, flags counted at 0.7, cost and latency from provider-reported usage. The
+model rows are `snifftest bench` on 2026-09-17, one repeat, cache off, and
+they count each model's own yes, which is the decision the model made; their
+cost is the returned token usage at the list prices recorded under
+`bench/prices/` with no prompt caching, and their latency is the wall clock at
+this end. Haiku, Sonnet and Opus were given 2,000 completion tokens, because
+at the default 900 Sonnet 5 ran out of room on 20 of its 166 replies on an
+earlier pass of the same run and a cut reply counts as unanswered; gpt-5.6-sol
+was given 4,000 at low reasoning effort. `bench-tables.md` in the results
+folder carries p95, the per-rule detail, every failure and the budget each row
+was sent. Jev ran in the same rotation as the models, where it read 59 of 80,
+2 of 54 and 198 ms, with one rule missing from its reply on 13 paragraphs;
+across the runs of this corpus on 2026-09-17 the judgment model read between
+59 and 64 of 80, and that swing is the size to read every row against.
+
+What that supports. On this corpus the judgment model catches about as many
+faults as the cheapest general model and fewer than the mid and top tiers,
+raises far fewer false alarms than the cheapest one, and does it at about a
+thirtieth of that model's cost and a tenth of its latency, with no general
+model in the loop. The top models catch more, at twenty to thirty-five times
+the latency and a hundred to two hundred and forty times the cost.
+
+Three things to read beside the table. Eight seeds per rule is a small sample,
+so the per-rule figures are direction rather than measurement. The eval
+measures every rule in the file, including the five marketing rules an
+ordinary check sits out. And two of the ten judgment rules were reworded after
+a first run on these seeds and measured again on the same seeds;
+`docs/eval-notes.md` carries both numbers and the reasoning.
+
+`snifftest eval --twins examples/adversarial` measures what a sentence written
+to the checker does to the readings around it. `examples/CORPUS.md` describes
+the corpus and where every paragraph came from.
+
+## Configuration
+
+| Option | What it does |
+| --- | --- |
+| `--rules <path>` | Use this ruleset instead of `.snifftest.yaml` or the built-in one. |
+| `--root <dir>` | The tree being checked, when it is not the directory you are in. |
+| `--threshold <0-1>` | The probability at or above which a judgment counts as a flag. |
+| `--format text\|json` | How to print the flags. |
+| `--only <tags>` | Run only the rules carrying one of these tags. |
+| `--skip <tags>` | Never run a rule carrying one of these tags. |
+| `--dry-run` | Make no network request of any kind. For `check`, the countable rules and nothing else. |
+| `--no-cache` | Ask about every paragraph again instead of reusing an answer from the last fortnight. |
+| `--yes`, `-y` | Answer the send question for this run and remember the answer. |
+
+| Variable | What it does |
+| --- | --- |
+| `TYPESAFE_API_KEY` | The key the judgment rules are sent with. Read from the environment and nowhere else. |
+| `SNIFFTEST_SEND` | Answer the send question in CI without remembering it. `1` means TypeSafe, which is where `check` sends and nowhere else. |
+| `SNIFFTEST_CACHE_DIR` | Where cached answers live. `off` for none. |
+| `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` | Used by `bench` only. Each one is the key its own rows are called with, and a panel row whose key is missing says so rather than running. |
+
+`snifftest --help` lists the options for `eval` and `bench` as well.
+
+The package has no runtime dependencies and needs Node 20 or newer. Installing
+from the npm registry needs nothing else. Installing from a git URL needs Bun
+on the machine, because the executable is built on install.
+
+## Contributing
+
+`CONTRIBUTING.md` has the ground rules, the test setup and what a rule proposal
+needs. Security problems go to a private advisory, never to a public issue.
+`SECURITY.md` says how.
+
+## Licence
+
+MIT. See `LICENSE`.

package/SECURITY.md

@@ -0,0 +1,165 @@
+# Security
+
+What leaves your machine, when, and what to do if you find a problem.
+
+## Supported versions
+
+| Version | Supported |
+| --- | --- |
+| 0.1.x | Yes |
+
+## What leaves the machine
+
+The countable rules are regular expressions. They run on your machine and
+never open a network connection.
+
+The judgment rules send text. For each paragraph, one request goes to
+`https://api.typesafe.ai/v1/systemone` over HTTPS with your `TYPESAFE_API_KEY`
+as a bearer token. The request carries that one paragraph and the wording of
+every judgment rule in the ruleset in force. The reply is one probability per
+rule. That endpoint is the only place `check` sends anything, and the address
+cannot be changed by a ruleset, a flag or an environment variable.
+
+What is never sent: file names, file paths, and anything outside the text you
+pointed the tool at. Fenced code is dropped before either kind of rule reads
+it. Headings, tables, front matter, link definitions and HTML comments are
+never sent. A file that is not UTF-8 text is skipped and reported as skipped.
+Text that carries a data URI or a long unbroken run of encoded characters is
+refused before a request is built, because the judgment model answers HTTP 200
+with meaningless probabilities on such input. A paragraph over 24,000
+characters is split before it is sent.
+
+There is no telemetry and no analytics.
+
+`bench` is different. It sends the same paragraphs to general models at other
+companies. Each destination is its own key and its own endpoint, and a row runs
+only when its key is set:
+
+- OpenRouter, which routes most of the panel: the paragraphs and the rule
+  wording go to `https://openrouter.ai/api/v1/chat/completions` with
+  `OPENROUTER_API_KEY`.
+- Anthropic, called directly: the same text to
+  `https://api.anthropic.com/v1/messages` with `ANTHROPIC_API_KEY`.
+- OpenAI, called directly: the same text to
+  `https://api.openai.com/v1/chat/completions` with `OPENAI_API_KEY`.
+
+It asks for consent naming each destination, and a yes given for TypeSafe never
+covers any of them. Before the first model call each provider's model list is
+read, which is an authenticated request with that key, so it sits behind the
+same question.
+
+## Consent
+
+Nothing is sent before the answer to the send question is yes.
+
+- The key is checked first. Without `TYPESAFE_API_KEY` the judgment rules
+  cannot run, the countable verdict is printed, and the run exits 2. The
+  question is never asked for a request the tool could not make.
+- With a key, the first run prints what will be sent, where, with which key,
+  and what comes back, then waits for `y`. Any other answer sends nothing.
+- A yes is stored in `$XDG_CONFIG_HOME/snifftest/consent.json`, or
+  `~/.config/snifftest/consent.json`, with mode 0600 in a 0700 directory. It is
+  never stored in the directory being checked. The stored answer names the
+  destinations it covers, and a new destination asks again.
+- `--yes` answers without the prompt and stores the answer.
+- `SNIFFTEST_SEND` answers for one run and stores nothing. Its value names the
+  destinations it answers for, comma separated. `1` is shorthand for TypeSafe,
+  the only place `check` sends, so a CI job that sets it for `check` has not
+  answered for `bench`.
+- With no terminal to ask in and no answer in the environment, the run exits 3
+  and sends nothing. Exit 3 is the question, not an error to work around.
+- `--dry-run` makes no network request of any kind, on every command.
+
+The key is read from `TYPESAFE_API_KEY` and from nowhere else. There is no
+config file for it and no flag. Every error message that could quote a
+provider's reply is passed through a scrubber that removes the key, whole or
+cut at either end, before it is printed.
+
+## The hook and the Action
+
+The pre-commit hook in `hooks/pre-commit` runs `check --dry-run` on the staged
+Markdown and text files. It sends nothing unless `SNIFFTEST_SEND` is set and
+`TYPESAFE_API_KEY` is present, both at once. It refuses a `snifftest` on `PATH`
+that resolves inside the repository being committed to, and it fetches the
+pinned version from a scratch directory rather than from inside the checkout,
+so the repository cannot supply the checker by committing one. A tool failure
+never blocks a commit unless `SNIFFTEST_STRICT=1`, but a countable flag that
+was printed before the failure still does.
+
+The two entries in `.pre-commit-hooks.yaml` behave the same way. `snifftest`
+never sends. `snifftest-send` sits under the manual stage and needs both the
+key and `SNIFFTEST_SEND=1`.
+
+The GitHub Action in `action.yml`:
+
+- Runs the countable rules only, unless `send` is `true` and a key is passed in
+  as `api-key`. It reads no secret of its own.
+- Forces the countable rules on a pull request from a fork, whatever the
+  workflow asked for, and on any event it has no fork guard for.
+- Refuses to run on any event whose name ends in `_target`.
+- Refuses a floating `version`. It fetches the exact version named from the npm
+  registry, from a scratch directory, on every run.
+- Puts the key in the environment of the step that runs the checker only when
+  sending is on, and never in the step that decides.
+- Posts a pull request comment only when `comment` is `true` and a token is
+  passed in as `github-token`. The comment carries the checker's output, which
+  is file, line, rule, probability and the rule's message. No line of the draft
+  is copied into it.
+
+The Claude Code skill runs the countable rules by default, adds no flag that
+skips the consent prompt, and never reads or prints the key.
+
+## The answer cache
+
+Judgment answers are cached on disk so a run interrupted by an outage does not
+pay for every paragraph twice. The cache is under `SNIFFTEST_CACHE_DIR`, else
+`$XDG_CACHE_HOME/snifftest`, else `~/.cache/snifftest`. Each file is named by a
+SHA-256 of the paragraph, the exact wording of the questions and the model
+alias, and it holds rule ids, probabilities, the served model name and a date.
+The paragraph is never written to disk. Files are written 0600 inside a 0700
+directory, and a relative `SNIFFTEST_CACHE_DIR` is resolved against the home
+directory so the cache can never land inside the tree being checked. Entries
+expire after fourteen days.
+`--no-cache` skips the cache for a run and `SNIFFTEST_CACHE_DIR=off` disables
+it. The cache is opened only after the key check and the consent gate, and
+never on `--dry-run`.
+
+## Known limits
+
+- A hash is not the text, but anyone who can read the cache and already has a
+  paragraph can confirm from it that the paragraph was checked.
+- Cache entries are keyed on the model alias `jev-latest`. A run's first live
+  answer names the model actually served, and entries from an older model are
+  dropped from then on. A run answered entirely from the cache has no live
+  answer to compare against, so for up to fourteen days after the service
+  moves the alias such a run can report the previous version's answers.
+- A cache directory that other users or other jobs can write to is a cache they
+  can poison. Keep it private. A poisoned entry can change a probability and
+  nothing else, because the reader drops any entry that is not a well-formed
+  set of numbers between 0 and 1.
+- A pattern rule is refused when it applies a quantifier to a group that
+  already contains one, or when two branches of an alternation can start on
+  the same character. Behind those shape rules sits a timing probe with a
+  small budget. The probe is a measurement, not a proof: a shape nobody has
+  written down yet can pass it on a fast machine and run long on a slow one.
+- The hook and the Action fetch the checker from the npm registry at the
+  pinned version. What arrives is whatever the registry serves for that name
+  and version, so a tag protection rule on this repository and a commit pin on
+  the Action are the readers' controls, and owning the name on the registry is
+  the maintainer's.
+- The scrubber removes the key's value, and prefixes and suffixes of it. It
+  cannot remove a re-encoding of the value, such as base64 or a hash, and does
+  not try.
+- A `.snifftest.yaml` found in the working directory may extend only files
+  under its own directory tree, or the shipped default, and is refused when it
+  is a symbolic link. A file named on `--rules` is followed wherever it points,
+  because naming it is the decision.
+
+## Reporting a vulnerability
+
+Open a private security advisory on the repository:
+
+https://github.com/DanRWilloughby/snifftest/security/advisories/new
+
+Do not open a public issue. Say what you found, how to reproduce it, and which
+version. You will get a reply on the advisory.

package/bench/panel-direct.yaml

@@ -0,0 +1,97 @@
+# The direct panel: every row on its provider's own API.
+#
+# `bench/panel.yaml` routes most of its rows through OpenRouter, which buys one
+# key for the whole bench at the cost of a proxy hop in every latency number.
+# This file is the other trade. Each row here is called on the API of the
+# company that serves it, with that company's own key, so the latencies are the
+# providers' own rather than a proxy's, and a row's cost is the published price
+# of the model rather than a reseller's.
+#
+# What it costs to read this way: more keys, and rows that cannot be compared
+# with the ones in the routed panel on latency, because they did not make the
+# same journey. Rows within this file are comparable with each other.
+#
+# Everything else is the same contract. Nothing here claims a model exists: each
+# row carries a pattern and a preference, every row is matched against the
+# provider's own model list on the day of the run, and a model the provider does
+# not list becomes a row that says "not available on <date>" rather than being
+# swapped for a near neighbour.
+
+version: 1
+
+prices:
+  # Neither the Messages API nor the OpenAI models endpoint publishes a price,
+  # so both direct providers are priced from a dated file and every table that
+  # uses those numbers cites the file, its source and the date it was checked.
+  anthropic: prices/anthropic-2026-09-17.yaml
+  openai: prices/openai-2026-09-17.yaml
+  # No dated published price has been recorded for the judgment service, so no
+  # file is named here and its row prints its cost as unknown. A constant in
+  # this repo's source is not a source.
+
+models:
+  # The three Claude rows share one completion budget, larger than the default.
+  # On the first direct run, at the default 900 tokens, Sonnet 5 ran out of room
+  # on 20 of its 152 replies and Opus 5 on 2: the JSON was cut mid-object and
+  # the paragraph counted as unanswered. A cut reply is a budget failure, not a
+  # model verdict, so every Claude row is given room to finish, and the same
+  # room, so their cost columns stay a fair comparison.
+  - id: haiku
+    label: Claude Haiku 4.5
+    tier: fast
+    provider: anthropic
+    match: "^claude-haiku-4-5"
+    prefer:
+      - claude-haiku-4-5-20251001
+    max_tokens: 2000
+
+  - id: sonnet
+    label: Claude Sonnet 5
+    tier: mid
+    provider: anthropic
+    match: "^claude-sonnet-5"
+    prefer:
+      - claude-sonnet-5
+    max_tokens: 2000
+
+  - id: opus
+    label: Claude Opus 5
+    tier: deep
+    provider: anthropic
+    match: "^claude-opus-5"
+    prefer:
+      - claude-opus-5
+    max_tokens: 2000
+
+  - id: sol
+    label: gpt-5.6-sol
+    tier: mid
+    provider: openai
+    match: "^gpt-5\\.6-sol$"
+    prefer:
+      - gpt-5.6-sol
+    # This row reasons, and on this API the reasoning tokens come out of the
+    # same completion budget as the answer. On the default budget it can spend
+    # the whole of it thinking and stop before it writes the JSON object, which
+    # lands in the table as a parse failure and reads as a verdict on the model
+    # rather than on the request. So the budget is raised and the effort is
+    # declared: low keeps the row affordable, and the footnote prints both, so
+    # nobody reads this row as the model's best.
+    max_tokens: 4000
+    reasoning:
+      effort: low
+
+  - id: jev
+    label: Jev, the judgment arm
+    tier: judgment
+    provider: jev
+    match: "^jev-"
+    prefer:
+      - jev-latest
+    note: >
+      Asked its own way, one request per paragraph carrying every rule as a
+      separate question, which is the shape the service documents and the shape
+      arm C uses. The request count matches the other rows; the request shape
+      does not. It returns a probability and no boolean, so its flag column is
+      that probability against the shipped threshold rather than a decision the
+      service made.