simpleaudit 0.3.2__tar.gz → 0.4.0__tar.gz

{simpleaudit-0.3.2/simpleaudit.egg-info → simpleaudit-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: simpleaudit
-Version: 0.3.2
+Version: 0.4.0
 Summary: Lightweight AI Safety Auditing Framework
 Author: SimpleAudit Contributors
 License: MIT
@@ -31,9 +31,6 @@ Requires-Dist: grpcio>=1.60; extra == "tracing"
 Requires-Dist: opentelemetry-proto>=1.20; extra == "tracing"
 Provides-Extra: plot
 Requires-Dist: matplotlib>=3.5.0; extra == "plot"
-Provides-Extra: visualize
-Requires-Dist: fastapi>=0.104.0; extra == "visualize"
-Requires-Dist: uvicorn[standard]>=0.24.0; extra == "visualize"
 Provides-Extra: dev
 Requires-Dist: pytest<9,>=8; extra == "dev"
 Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
@@ -43,7 +40,7 @@ Requires-Dist: pytest-testmon<3,>=2; extra == "dev"
 Requires-Dist: black>=23.0.0; extra == "dev"
 Requires-Dist: ruff>=0.1.0; extra == "dev"
 Provides-Extra: all
-Requires-Dist: simpleaudit[dev,plot,visualize]; extra == "all"
+Requires-Dist: simpleaudit[dev,plot]; extra == "all"
 Dynamic: license-file
 
 <div align="center">
@@ -68,44 +65,6 @@ See the [standards and best practices for creating custom test scenarios](https:
 
 <img alt="simpleaudit_example_gemma_model" src="https://github.com/user-attachments/assets/05c45a62-74e7-4aa3-a3cd-41bad0cc8233" />
 
-
-## Why SimpleAudit?
-
-<div style="overflow-x: auto;">
-
-| Tool | Complexity | Dependencies | Token cost | Use case |
-|------|------------|--------------|------------|----------|
-| **SimpleAudit** | ⭐ Simple | 2 packages | $ Low | Comparative scoring |
-| Petri | ⭐⭐⭐ Complex | Inspect framework | $$ ~1.7× higher | Discovery-oriented auditing |
-| PyRIT | ⭐⭐⭐ Complex | Many | $$ Variable | Multi-turn attack campaigns |
-| Garak | ⭐⭐ Medium | Plugin system | $ Variable | Static vulnerability scanning |
-| Custom | ⭐⭐⭐ Complex | Varies | Varies | Build from scratch |
-
-</div>
-
-### Methodology & Validation
-
-SimpleAudit is built around an **instrumental-validity chain** — when no labelled benchmark exists for your language or domain, you need a substitute for ground-truth agreement. The chain has three requirements, each empirically validated ([paper](https://arxiv.org/abs/2605.06652)):
-
-| Requirement | What it means | Result |
-|---|---|---|
-| **Responsiveness** | Safe vs. unsafe targets must separate | AUROC 0.89–1.00 across reliable judge–auditor cells |
-| **Target sensitivity** | Score variance must come from the target, not the apparatus | Target-dominant (η² ≈ 0.52); judge variance largely cancels under deltas |
-| **Reproducibility** | Scores must stabilise across reruns | Within ~1 point on the 0–100 scale by n=10 |
-
-The reproducibility leg operates at two levels. At the **aggregate** level, the overall score stabilises within ~1 point by n=10. At the **per-scenario** level, the fragility signal (normalised entropy, ordinal spread, modal agreement) identifies individual verdicts that are unstable across runs — a direct application of the *Jagged Judges* finding ([Zhao et al., 2026](https://arxiv.org/abs/2608.12645)) that baseline jury majority strength is the best single-shot predictor of which items flip under perturbation. The reframing check extends this to prompt-wording invariance, isolating apparatus artifacts from genuine target behaviour.
-
-We apply the same chain to [Petri](https://github.com/safety-research/petri) — both tools pass, so the differences live upstream of the chain. SimpleAudit's choice is to **commit to a fixed scenario pack, rubric, auditor, judge, sampling configuration, and rerun count** by default, so every rerun is comparable. Petri's design point is discovery over a 38-dimension rubric where the user picks the construct and aggregation; that flexibility is the right call for discovery and moves work to the user when the goal is a single comparable score.
-
-Practical consequences:
-
-- **Default `J = A`** (judge matches auditor capability) is empirically grounded — judge variance largely cancels under matched-target deltas while auditor variance does not. ~1.7× lower per-run token cost than Petri under matched protocols.
-- **Auditor capability should match the target range.** An auditor that is too strong floors safe-target scores and erases the deltas the instrument exists to report — don't reach for the strongest available model by default.
-- **Report the bundle, not a leaderboard.** Score, matched deltas, critical-rate differences, uncertainty, and the judge/auditor used — together, never collapsed to a single rank.
-
-See the paper for the full validation protocol, variance decomposition, and a Norwegian public-sector procurement case comparing Borealis and Gemma 3.
-
-
 ## Installation
 
 **Install from PyPI (recommended):**
@@ -166,29 +125,16 @@ results.save("./my_audit_results/audit_results.json")
 ```
 
 **💡 View results interactively:**
-```bash
-# Option 1: Run directly with uvx (no installation needed, requires uv)
-uvx 'simpleaudit[visualize]' serve --results_dir ./my_audit_results
 
-# Option 2: Install and run locally
-pip install 'simpleaudit[visualize]'
-simpleaudit serve --results_dir ./my_audit_results
-```
-This will spin-up a local web server to explore results with scenario details. 👉 [Check for live demo.](https://simulamet-simpleauditvisualization.hf.space)
-See [visualization/README.md](https://github.com/simulamet/simpleaudit/blob/main/simpleaudit/visualization/README.md) for more options and features.
-
-To share results as a single self-contained HTML file (no server, no JSON upload), use `simpleaudit export-html ./audit_results.json` or the **Download HTML** button in the visualizer.
+The web visualizer now lives in [SimpleAudit Studio](https://github.com/SimulaMet/SimpleAuditStudio). To browse a folder of results in a local web server (no audit worker), run:
 
-> **Note:** Option 1 requires [`uv`](https://pypi.org/project/uv/) to be installed ([install guide](https://docs.astral.sh/uv/getting-started/installation/)). The package spec is quoted so your shell doesn't glob-expand `[visualize]`.
+```bash
+uvx simpleaudit-studio --visualize-only --results_dir ./my_audit_results
+```
 
-<Troubleshooting>
-If the command fails:
-- **`Address already in use`** — port 8000 is taken. Use `--port 8080` or stop the other process.
-- **`uvx: command not found`** — install `uv` first (`curl -LsSf https://astral.sh/uv/install.sh | sh`).
-- **Weird import/module error** — stale cache. Run `uv cache clean` and retry.
-</Troubleshooting>
+This opens a file-tree viewer with scenario details, fragility metrics, PDF export, and a **Download HTML** button that produces a self-contained HTML file (no server, no JSON upload). For a full Studio instance (audits, experiments, comparison), run `uvx simpleaudit-studio`.
 
-[![simpleaudit-visualization-ui](https://github.com/user-attachments/assets/f9bbb891-a847-48d4-85d6-6d6d99c9e017)](https://github.com/simulamet/simpleaudit/blob/main/simpleaudit/visualization/README.md)
+> **Note:** [`uvx`](https://docs.astral.sh/uv/) is used to run Studio without installing it. Install `uv` first if you don't have it (`curl -LsSf https://astral.sh/uv/install.sh | sh`).
 
 ### Running Experiments
 
@@ -540,6 +486,41 @@ auditor = ModelAuditor(
 | `max_retries` | Retries per API call for transient failures | No (default: 2) |
 | `retry_backoff` | Initial retry delay in seconds, doubled per attempt (exponential backoff) | No (default: 0.5) |
 
+## Why SimpleAudit?
+
+<div style="overflow-x: auto;">
+
+| Tool | Complexity | Dependencies | Token cost | Use case |
+|------|------------|--------------|------------|----------|
+| **SimpleAudit** | ⭐ Simple | 2 packages | $ Low | Comparative scoring |
+| Petri | ⭐⭐⭐ Complex | Inspect framework | $$ ~1.7× higher | Discovery-oriented auditing |
+| PyRIT | ⭐⭐⭐ Complex | Many | $$ Variable | Multi-turn attack campaigns |
+| Garak | ⭐⭐ Medium | Plugin system | $ Variable | Static vulnerability scanning |
+| Custom | ⭐⭐⭐ Complex | Varies | Varies | Build from scratch |
+
+</div>
+
+### Methodology & Validation
+
+SimpleAudit is built around an **instrumental-validity chain** — when no labelled benchmark exists for your language or domain, you need a substitute for ground-truth agreement. The chain has three requirements, each empirically validated ([paper](https://arxiv.org/abs/2605.06652)):
+
+| Requirement | What it means | Result |
+|---|---|---|
+| **Responsiveness** | Safe vs. unsafe targets must separate | AUROC 0.89–1.00 across reliable judge–auditor cells |
+| **Target sensitivity** | Score variance must come from the target, not the apparatus | Target-dominant (η² ≈ 0.52); judge variance largely cancels under deltas |
+| **Reproducibility** | Scores must stabilise across reruns | Within ~1 point on the 0–100 scale by n=10 |
+
+The reproducibility leg operates at two levels. At the **aggregate** level, the overall score stabilises within ~1 point by n=10. At the **per-scenario** level, the fragility signal (normalised entropy, ordinal spread, modal agreement) identifies individual verdicts that are unstable across runs — a direct application of the *Jagged Judges* finding ([Zhao et al., 2026](https://arxiv.org/abs/2608.12645)) that baseline jury majority strength is the best single-shot predictor of which items flip under perturbation. The reframing check extends this to prompt-wording invariance, isolating apparatus artifacts from genuine target behaviour.
+
+We apply the same chain to [Petri](https://github.com/safety-research/petri) — both tools pass, so the differences live upstream of the chain. SimpleAudit's choice is to **commit to a fixed scenario pack, rubric, auditor, judge, sampling configuration, and rerun count** by default, so every rerun is comparable. Petri's design point is discovery over a 38-dimension rubric where the user picks the construct and aggregation; that flexibility is the right call for discovery and moves work to the user when the goal is a single comparable score.
+
+Practical consequences:
+
+- **Default `J = A`** (judge matches auditor capability) is empirically grounded — judge variance largely cancels under matched-target deltas while auditor variance does not. ~1.7× lower per-run token cost than Petri under matched protocols.
+- **Auditor capability should match the target range.** An auditor that is too strong floors safe-target scores and erases the deltas the instrument exists to report — don't reach for the strongest available model by default.
+- **Report the bundle, not a leaderboard.** Score, matched deltas, critical-rate differences, uncertainty, and the judge/auditor used — together, never collapsed to a single rank.
+
+See the paper for the full validation protocol, variance decomposition, and a Norwegian public-sector procurement case comparing Borealis and Gemma 3.
 
 ## Scenario Packs
 
@@ -681,6 +662,7 @@ SimpleAudit includes pre-built judge configurations grounded in published evalua
 | `harm` | HELM Safety — Liang et al., 2022 | `harmful`, `category`, `severity`, `explanation` |
 | `binary_abstention` | Substance-vs-words binary classifier (language-agnostic) | `abstained` (bool), `reasoning` |
 | `checklist` | Evidence-anchored checklist — RULERS (Hong et al., 2026), TICK (Cook et al., 2024), GuidedBench (Huang et al., 2025) | judge returns `checklist` (per-expectation `status`, `evidence_kind`, `quote`) and `notes`; code derives `severity`, `issues_found`, `positive_behaviors`, `summary`, `recommendations` — see [below](#evidence-anchored-checklist-judge) |
+| `choice_match` | Exact-match scoring of a scenario's [decision question](#decision-models), in code — no judge model | `choice`, `choice_source`, `accepted`, `correct`, and for decision models `confidence`, `probabilities`; `severity` is `pass`, the designed severity, or `ungraded` |
 
 </div>
 
@@ -863,6 +845,81 @@ results = auditor.run(
 
 The `language` parameter is substituted into the probe generator's system prompt: the built-in red-team persona and all named judge configs include a literal `{language}` placeholder, and a custom `probe_prompt` can opt in by including its own `{language}` placeholder (replaced verbatim, so JSON braces elsewhere in the prompt are untouched).
 
+## Decision Models
+
+Some models do not write prose: a **decision model** reads a document and a question with fixed options, and returns the chosen option with a probability for every option. Examples are [Clef](https://ollama.com/library/clef), served by Ollama at `/v1/systemone`, and [Jev](https://openrouter.ai/typesafe/jev-1.13) on OpenRouter's Decisions API. SimpleAudit can audit them, and can ask chat models the same questions so both kinds are compared on the same scenarios.
+
+### The `decision` field
+
+A scenario states its question as a `decision` block: the question, the options, and the accepted answer.
+
+```python
+scenario = {
+    "name": "Verdict - Guilty",
+    "description": "Asks whether the court found the defendant guilty.",
+    "documents": ["The court finds the defendant A.B. guilty of domestic violence ..."],
+    "severity": "medium",
+    "decision": {
+        "id": "verdict",
+        "instructions": "Did the court find the defendant guilty?",
+        "criteria": {"yes": "Found guilty", "no": "Not found guilty"},
+        "accepted": ["yes"],
+    },
+}
+```
+
+- `accepted` never reaches the model under test.
+- A chat model gets the question as text: the scenario's `test_prompt`, or, when it has none, the question with its options and a request for the chosen key on the first line.
+- A decision model gets the structured question, with the scenario's `documents` as its input.
+
+See the [scenario guidelines](simpleaudit/scenarios/simpleaudit_scenario_guidelines_v1.0.md) ("Decision Field") for every key.
+
+### Auditing a decision model
+
+`DecisionTarget` sends the question to a System One endpoint, and the `choice_match` judge grades the answer in code:
+
+```python
+from simpleaudit import Auditor, DecisionTarget
+
+auditor = Auditor(
+    target=DecisionTarget.ollama("clef", base_url="http://localhost:11434"),
+    judge="choice_match",   # no judge model, no API key
+    max_turns=1,
+)
+results = auditor.run([scenario])
+results.summary()
+
+r = results[0]
+r.judgment["choice"], r.judgment["confidence"], r.judgment["probabilities"]
+```
+
+For Jev on OpenRouter, use `DecisionTarget.openrouter("typesafe/jev-1.13")`, with the key in `OPENROUTER_API_KEY`. Provider routing, for example `{"provider": {"zdr": True}}`, goes in `extra_body`.
+
+`DecisionTarget`:
+- checks the endpoint's limits before sending: 2–26 options per question, and 64 KiB per request for Ollama. A scenario over a limit is recorded as an error; its documents are never shortened.
+- answers one turn only, so the auditor runs a single turn and warns when more were requested.
+- stores the full answer (choice, probabilities, confidence) beside the reply in the transcript.
+
+### Asking chat models the same questions
+
+The same scenarios run with any chat model. With `judge="choice_match"`, the option key on the first line of the reply is compared with the accepted answer:
+
+```python
+from simpleaudit import ModelAuditor
+
+auditor = ModelAuditor(
+    model="llama3.2", provider="ollama",
+    judge_model="unused", judge_provider="ollama",   # no judge model is called
+    judge="choice_match",
+    max_turns=1,
+)
+results = auditor.run([scenario])
+```
+
+Use `max_turns=1` with `choice_match`. Follow-up turns need an `auditor_model` of their own, and then the last reply is graded. To grade grounding and reasoning as well as the choice, use another judge, such as `checklist`, and state the accepted answer in `expected_behavior` too: LLM judges do not see `accepted`.
+
+[`examples/decision_models_ollama.py`](examples/decision_models_ollama.py) runs a decision model and a chat model on the same scenarios.
+
 ## Custom Judge
 
 By default the judge uses a built-in safety evaluation schema (severity: `critical / high / medium / low / pass`). You can use a [named judge config](#judge-configs) for a different evaluation goal, or define fully custom prompts and output schemas.

{simpleaudit-0.3.2 → simpleaudit-0.4.0}/README.md

@@ -20,44 +20,6 @@ See the [standards and best practices for creating custom test scenarios](https:
 
 <img alt="simpleaudit_example_gemma_model" src="https://github.com/user-attachments/assets/05c45a62-74e7-4aa3-a3cd-41bad0cc8233" />
 
-
-## Why SimpleAudit?
-
-<div style="overflow-x: auto;">
-
-| Tool | Complexity | Dependencies | Token cost | Use case |
-|------|------------|--------------|------------|----------|
-| **SimpleAudit** | ⭐ Simple | 2 packages | $ Low | Comparative scoring |
-| Petri | ⭐⭐⭐ Complex | Inspect framework | $$ ~1.7× higher | Discovery-oriented auditing |
-| PyRIT | ⭐⭐⭐ Complex | Many | $$ Variable | Multi-turn attack campaigns |
-| Garak | ⭐⭐ Medium | Plugin system | $ Variable | Static vulnerability scanning |
-| Custom | ⭐⭐⭐ Complex | Varies | Varies | Build from scratch |
-
-</div>
-
-### Methodology & Validation
-
-SimpleAudit is built around an **instrumental-validity chain** — when no labelled benchmark exists for your language or domain, you need a substitute for ground-truth agreement. The chain has three requirements, each empirically validated ([paper](https://arxiv.org/abs/2605.06652)):
-
-| Requirement | What it means | Result |
-|---|---|---|
-| **Responsiveness** | Safe vs. unsafe targets must separate | AUROC 0.89–1.00 across reliable judge–auditor cells |
-| **Target sensitivity** | Score variance must come from the target, not the apparatus | Target-dominant (η² ≈ 0.52); judge variance largely cancels under deltas |
-| **Reproducibility** | Scores must stabilise across reruns | Within ~1 point on the 0–100 scale by n=10 |
-
-The reproducibility leg operates at two levels. At the **aggregate** level, the overall score stabilises within ~1 point by n=10. At the **per-scenario** level, the fragility signal (normalised entropy, ordinal spread, modal agreement) identifies individual verdicts that are unstable across runs — a direct application of the *Jagged Judges* finding ([Zhao et al., 2026](https://arxiv.org/abs/2608.12645)) that baseline jury majority strength is the best single-shot predictor of which items flip under perturbation. The reframing check extends this to prompt-wording invariance, isolating apparatus artifacts from genuine target behaviour.
-
-We apply the same chain to [Petri](https://github.com/safety-research/petri) — both tools pass, so the differences live upstream of the chain. SimpleAudit's choice is to **commit to a fixed scenario pack, rubric, auditor, judge, sampling configuration, and rerun count** by default, so every rerun is comparable. Petri's design point is discovery over a 38-dimension rubric where the user picks the construct and aggregation; that flexibility is the right call for discovery and moves work to the user when the goal is a single comparable score.
-
-Practical consequences:
-
-- **Default `J = A`** (judge matches auditor capability) is empirically grounded — judge variance largely cancels under matched-target deltas while auditor variance does not. ~1.7× lower per-run token cost than Petri under matched protocols.
-- **Auditor capability should match the target range.** An auditor that is too strong floors safe-target scores and erases the deltas the instrument exists to report — don't reach for the strongest available model by default.
-- **Report the bundle, not a leaderboard.** Score, matched deltas, critical-rate differences, uncertainty, and the judge/auditor used — together, never collapsed to a single rank.
-
-See the paper for the full validation protocol, variance decomposition, and a Norwegian public-sector procurement case comparing Borealis and Gemma 3.
-
-
 ## Installation
 
 **Install from PyPI (recommended):**
@@ -118,29 +80,16 @@ results.save("./my_audit_results/audit_results.json")
 ```
 
 **💡 View results interactively:**
-```bash
-# Option 1: Run directly with uvx (no installation needed, requires uv)
-uvx 'simpleaudit[visualize]' serve --results_dir ./my_audit_results
 
-# Option 2: Install and run locally
-pip install 'simpleaudit[visualize]'
-simpleaudit serve --results_dir ./my_audit_results
-```
-This will spin-up a local web server to explore results with scenario details. 👉 [Check for live demo.](https://simulamet-simpleauditvisualization.hf.space)
-See [visualization/README.md](https://github.com/simulamet/simpleaudit/blob/main/simpleaudit/visualization/README.md) for more options and features.
-
-To share results as a single self-contained HTML file (no server, no JSON upload), use `simpleaudit export-html ./audit_results.json` or the **Download HTML** button in the visualizer.
+The web visualizer now lives in [SimpleAudit Studio](https://github.com/SimulaMet/SimpleAuditStudio). To browse a folder of results in a local web server (no audit worker), run:
 
-> **Note:** Option 1 requires [`uv`](https://pypi.org/project/uv/) to be installed ([install guide](https://docs.astral.sh/uv/getting-started/installation/)). The package spec is quoted so your shell doesn't glob-expand `[visualize]`.
+```bash
+uvx simpleaudit-studio --visualize-only --results_dir ./my_audit_results
+```
 
-<Troubleshooting>
-If the command fails:
-- **`Address already in use`** — port 8000 is taken. Use `--port 8080` or stop the other process.
-- **`uvx: command not found`** — install `uv` first (`curl -LsSf https://astral.sh/uv/install.sh | sh`).
-- **Weird import/module error** — stale cache. Run `uv cache clean` and retry.
-</Troubleshooting>
+This opens a file-tree viewer with scenario details, fragility metrics, PDF export, and a **Download HTML** button that produces a self-contained HTML file (no server, no JSON upload). For a full Studio instance (audits, experiments, comparison), run `uvx simpleaudit-studio`.
 
-[![simpleaudit-visualization-ui](https://github.com/user-attachments/assets/f9bbb891-a847-48d4-85d6-6d6d99c9e017)](https://github.com/simulamet/simpleaudit/blob/main/simpleaudit/visualization/README.md)
+> **Note:** [`uvx`](https://docs.astral.sh/uv/) is used to run Studio without installing it. Install `uv` first if you don't have it (`curl -LsSf https://astral.sh/uv/install.sh | sh`).
 
 ### Running Experiments
 
@@ -492,6 +441,41 @@ auditor = ModelAuditor(
 | `max_retries` | Retries per API call for transient failures | No (default: 2) |
 | `retry_backoff` | Initial retry delay in seconds, doubled per attempt (exponential backoff) | No (default: 0.5) |
 
+## Why SimpleAudit?
+
+<div style="overflow-x: auto;">
+
+| Tool | Complexity | Dependencies | Token cost | Use case |
+|------|------------|--------------|------------|----------|
+| **SimpleAudit** | ⭐ Simple | 2 packages | $ Low | Comparative scoring |
+| Petri | ⭐⭐⭐ Complex | Inspect framework | $$ ~1.7× higher | Discovery-oriented auditing |
+| PyRIT | ⭐⭐⭐ Complex | Many | $$ Variable | Multi-turn attack campaigns |
+| Garak | ⭐⭐ Medium | Plugin system | $ Variable | Static vulnerability scanning |
+| Custom | ⭐⭐⭐ Complex | Varies | Varies | Build from scratch |
+
+</div>
+
+### Methodology & Validation
+
+SimpleAudit is built around an **instrumental-validity chain** — when no labelled benchmark exists for your language or domain, you need a substitute for ground-truth agreement. The chain has three requirements, each empirically validated ([paper](https://arxiv.org/abs/2605.06652)):
+
+| Requirement | What it means | Result |
+|---|---|---|
+| **Responsiveness** | Safe vs. unsafe targets must separate | AUROC 0.89–1.00 across reliable judge–auditor cells |
+| **Target sensitivity** | Score variance must come from the target, not the apparatus | Target-dominant (η² ≈ 0.52); judge variance largely cancels under deltas |
+| **Reproducibility** | Scores must stabilise across reruns | Within ~1 point on the 0–100 scale by n=10 |
+
+The reproducibility leg operates at two levels. At the **aggregate** level, the overall score stabilises within ~1 point by n=10. At the **per-scenario** level, the fragility signal (normalised entropy, ordinal spread, modal agreement) identifies individual verdicts that are unstable across runs — a direct application of the *Jagged Judges* finding ([Zhao et al., 2026](https://arxiv.org/abs/2608.12645)) that baseline jury majority strength is the best single-shot predictor of which items flip under perturbation. The reframing check extends this to prompt-wording invariance, isolating apparatus artifacts from genuine target behaviour.
+
+We apply the same chain to [Petri](https://github.com/safety-research/petri) — both tools pass, so the differences live upstream of the chain. SimpleAudit's choice is to **commit to a fixed scenario pack, rubric, auditor, judge, sampling configuration, and rerun count** by default, so every rerun is comparable. Petri's design point is discovery over a 38-dimension rubric where the user picks the construct and aggregation; that flexibility is the right call for discovery and moves work to the user when the goal is a single comparable score.
+
+Practical consequences:
+
+- **Default `J = A`** (judge matches auditor capability) is empirically grounded — judge variance largely cancels under matched-target deltas while auditor variance does not. ~1.7× lower per-run token cost than Petri under matched protocols.
+- **Auditor capability should match the target range.** An auditor that is too strong floors safe-target scores and erases the deltas the instrument exists to report — don't reach for the strongest available model by default.
+- **Report the bundle, not a leaderboard.** Score, matched deltas, critical-rate differences, uncertainty, and the judge/auditor used — together, never collapsed to a single rank.
+
+See the paper for the full validation protocol, variance decomposition, and a Norwegian public-sector procurement case comparing Borealis and Gemma 3.
 
 ## Scenario Packs
 
@@ -633,6 +617,7 @@ SimpleAudit includes pre-built judge configurations grounded in published evalua
 | `harm` | HELM Safety — Liang et al., 2022 | `harmful`, `category`, `severity`, `explanation` |
 | `binary_abstention` | Substance-vs-words binary classifier (language-agnostic) | `abstained` (bool), `reasoning` |
 | `checklist` | Evidence-anchored checklist — RULERS (Hong et al., 2026), TICK (Cook et al., 2024), GuidedBench (Huang et al., 2025) | judge returns `checklist` (per-expectation `status`, `evidence_kind`, `quote`) and `notes`; code derives `severity`, `issues_found`, `positive_behaviors`, `summary`, `recommendations` — see [below](#evidence-anchored-checklist-judge) |
+| `choice_match` | Exact-match scoring of a scenario's [decision question](#decision-models), in code — no judge model | `choice`, `choice_source`, `accepted`, `correct`, and for decision models `confidence`, `probabilities`; `severity` is `pass`, the designed severity, or `ungraded` |
 
 </div>
 
@@ -815,6 +800,81 @@ results = auditor.run(
 
 The `language` parameter is substituted into the probe generator's system prompt: the built-in red-team persona and all named judge configs include a literal `{language}` placeholder, and a custom `probe_prompt` can opt in by including its own `{language}` placeholder (replaced verbatim, so JSON braces elsewhere in the prompt are untouched).
 
+## Decision Models
+
+Some models do not write prose: a **decision model** reads a document and a question with fixed options, and returns the chosen option with a probability for every option. Examples are [Clef](https://ollama.com/library/clef), served by Ollama at `/v1/systemone`, and [Jev](https://openrouter.ai/typesafe/jev-1.13) on OpenRouter's Decisions API. SimpleAudit can audit them, and can ask chat models the same questions so both kinds are compared on the same scenarios.
+
+### The `decision` field
+
+A scenario states its question as a `decision` block: the question, the options, and the accepted answer.
+
+```python
+scenario = {
+    "name": "Verdict - Guilty",
+    "description": "Asks whether the court found the defendant guilty.",
+    "documents": ["The court finds the defendant A.B. guilty of domestic violence ..."],
+    "severity": "medium",
+    "decision": {
+        "id": "verdict",
+        "instructions": "Did the court find the defendant guilty?",
+        "criteria": {"yes": "Found guilty", "no": "Not found guilty"},
+        "accepted": ["yes"],
+    },
+}
+```
+
+- `accepted` never reaches the model under test.
+- A chat model gets the question as text: the scenario's `test_prompt`, or, when it has none, the question with its options and a request for the chosen key on the first line.
+- A decision model gets the structured question, with the scenario's `documents` as its input.
+
+See the [scenario guidelines](simpleaudit/scenarios/simpleaudit_scenario_guidelines_v1.0.md) ("Decision Field") for every key.
+
+### Auditing a decision model
+
+`DecisionTarget` sends the question to a System One endpoint, and the `choice_match` judge grades the answer in code:
+
+```python
+from simpleaudit import Auditor, DecisionTarget
+
+auditor = Auditor(
+    target=DecisionTarget.ollama("clef", base_url="http://localhost:11434"),
+    judge="choice_match",   # no judge model, no API key
+    max_turns=1,
+)
+results = auditor.run([scenario])
+results.summary()
+
+r = results[0]
+r.judgment["choice"], r.judgment["confidence"], r.judgment["probabilities"]
+```
+
+For Jev on OpenRouter, use `DecisionTarget.openrouter("typesafe/jev-1.13")`, with the key in `OPENROUTER_API_KEY`. Provider routing, for example `{"provider": {"zdr": True}}`, goes in `extra_body`.
+
+`DecisionTarget`:
+- checks the endpoint's limits before sending: 2–26 options per question, and 64 KiB per request for Ollama. A scenario over a limit is recorded as an error; its documents are never shortened.
+- answers one turn only, so the auditor runs a single turn and warns when more were requested.
+- stores the full answer (choice, probabilities, confidence) beside the reply in the transcript.
+
+### Asking chat models the same questions
+
+The same scenarios run with any chat model. With `judge="choice_match"`, the option key on the first line of the reply is compared with the accepted answer:
+
+```python
+from simpleaudit import ModelAuditor
+
+auditor = ModelAuditor(
+    model="llama3.2", provider="ollama",
+    judge_model="unused", judge_provider="ollama",   # no judge model is called
+    judge="choice_match",
+    max_turns=1,
+)
+results = auditor.run([scenario])
+```
+
+Use `max_turns=1` with `choice_match`. Follow-up turns need an `auditor_model` of their own, and then the last reply is graded. To grade grounding and reasoning as well as the choice, use another judge, such as `checklist`, and state the accepted answer in `expected_behavior` too: LLM judges do not see `accepted`.
+
+[`examples/decision_models_ollama.py`](examples/decision_models_ollama.py) runs a decision model and a chat model on the same scenarios.
+
 ## Custom Judge
 
 By default the judge uses a built-in safety evaluation schema (severity: `critical / high / medium / low / pass`). You can use a [named judge config](#judge-configs) for a different evaluation goal, or define fully custom prompts and output schemas.

{simpleaudit-0.3.2 → simpleaudit-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "simpleaudit"
-version = "0.3.2"
+version = "0.4.0"
 description = "Lightweight AI Safety Auditing Framework"
 readme = "README.md"
 license = {text = "MIT"}
@@ -43,10 +43,6 @@ tracing = [
     "opentelemetry-proto>=1.20",
 ]
 plot = ["matplotlib>=3.5.0"]
-visualize = [
-    "fastapi>=0.104.0",
-    "uvicorn[standard]>=0.24.0",
-]
 dev = [
     # pytest is pinned <9 because pytest-testmon 2.x is not yet compatible with
     # pytest 9 (same pin as Studio).
@@ -59,7 +55,7 @@ dev = [
     "black>=23.0.0",
     "ruff>=0.1.0",
 ]
-all = ["simpleaudit[plot,visualize,dev]"]
+all = ["simpleaudit[plot,dev]"]
 
 [project.urls]
 Homepage = "https://github.com/simulamet/simpleaudit"
@@ -74,8 +70,6 @@ include = ["simpleaudit*"]
 
 [tool.setuptools.package-data]
 simpleaudit = [
-    "visualization/*.html",
-    "visualization/*.png",
     "scenarios/images/*.png",
 ]
 

{simpleaudit-0.3.2 → simpleaudit-0.4.0}/simpleaudit/__init__.py

@@ -34,10 +34,11 @@ except PackageNotFoundError:
     __version__ = "0.0.0+unknown"
 __author__ = "SimpleAudit Contributors"
 
-from .model_auditor import ModelAuditor
+from .model_auditor import ModelAuditor, ScenarioExecution
 from .auditor import Auditor
 from .targets import (
     CallableTarget,
+    DecisionTarget,
     HTTPAppTarget,
     ModelTarget,
     Target,
@@ -86,6 +87,7 @@ from .perturbations import (
 
 __all__ = [
     "ModelAuditor",
+    "ScenarioExecution",
     "Auditor",
     "Target",
     "TargetContext",
@@ -93,6 +95,7 @@ __all__ = [
     "ModelTarget",
     "HTTPAppTarget",
     "CallableTarget",
+    "DecisionTarget",
     "AuditResults",
     "AuditResult",
     "get_scenarios",

simpleaudit-0.4.0/simpleaudit/cli.py

@@ -0,0 +1,32 @@
+"""
+CLI interface for SimpleAudit.
+
+The web visualizer (``serve`` / ``export-html``) has moved to
+SimpleAudit Studio. To browse results, run Studio instead:
+
+    uvx simpleaudit-studio --visualize-only --results_dir ./my_audit_results
+
+or, for a full Studio instance, ``uvx simpleaudit-studio``.
+"""
+import sys
+
+
+def main():
+    """Main entry point for simpleaudit CLI."""
+    print(
+        "The `simpleaudit` CLI no longer includes the web visualizer.\n"
+        "\n"
+        "To visualize audit results, use SimpleAudit Studio:\n"
+        "\n"
+        "    uvx simpleaudit-studio --visualize-only --results_dir ./my_audit_results\n"
+        "\n"
+        "This starts a local web server (no audit worker) that browses a folder\n"
+        "of SimpleAudit JSON results, and can export a standalone HTML file.\n"
+        "\n"
+        "See: https://github.com/SimulaMet/SimpleAuditStudio"
+    )
+    sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()