sys1bench 0.3.0__tar.gz → 0.3.2__tar.gz

sys1bench-0.3.2/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: sys1bench
+Version: 0.3.2
+Summary: Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness.
+Author-email: Rahul Sharma <rsharma@rptu.de>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/rssr25/system-one-bench
+Project-URL: Documentation, https://github.com/rssr25/system-one-bench/blob/main/docs/SPEC_v2.md
+Project-URL: Issues, https://github.com/rssr25/system-one-bench/issues
+Project-URL: Changelog, https://github.com/rssr25/system-one-bench/blob/main/CHANGELOG.md
+Keywords: benchmark,calibration,system-one,decision-models,jev,laya,evaluation,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Operating System :: OS Independent
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: scipy>=1.11
+Requires-Dist: pandas>=2.0
+Requires-Dist: pydantic>=2.5
+Requires-Dist: httpx>=0.27
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typer>=0.12
+Provides-Extra: plots
+Requires-Dist: matplotlib>=3.8; extra == "plots"
+Provides-Extra: local
+Requires-Dist: torch>=2.4; extra == "local"
+Requires-Dist: transformers>=4.45; extra == "local"
+Requires-Dist: sentence-transformers>=3.0; extra == "local"
+Provides-Extra: laya
+Requires-Dist: laya>=0.3; extra == "laya"
+Provides-Extra: llm
+Requires-Dist: vllm>=0.6; extra == "llm"
+Requires-Dist: outlines>=0.1; extra == "llm"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: matplotlib>=3.8; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+<p align="center">
+  <img src="docs/assets/logo.svg" alt="sys1bench" width="480">
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
+</p>
+
+<p align="center"><b>A benchmark for typed System One decision models</b><br>
+Jev · Laya · Kev · and whatever comes next</p>
+
+---
+
+System One models read a block of state plus typed questions and return a probability distribution over the answers you defined — one of N options, a level on an ordinal scale, or yes/no — in a single forward pass. No text is generated, so nothing has to be parsed. **sys1bench** measures what that buys: whether the probabilities are calibrated (against a noise floor), how much the answer depends on how the question is worded, how confidence behaves out of scope, how accuracy scales with options and state length, whether batched questions interfere, and what it all costs per decision. Headline numbers come from generated items whose labels follow a stated policy, so they cannot be memorised.
+
+```bash
+pip install sys1bench
+sys1bench suite all results/my-model --config my_model.yaml        # A–I on generated, policy-labelled data
+sys1bench report results --html results/dashboard.html            # tables + interactive dashboard
+```
+
+## Results at a glance
+
+<p align="center">
+  <img src="docs/figures/framing_tickets_priority.png" alt="Accuracy across question wordings on the policy-following priority score" width="49%">
+  <img src="docs/figures/reliability_tickets_priority.png" alt="Reliability diagram, priority score, all models" width="41%">
+</p>
+<p align="center">
+  <img src="docs/figures/cardinality_accuracy.png" alt="Exact accuracy versus number of options" width="45%">
+  <img src="docs/figures/latency_vs_questions.png" alt="Median latency versus number of questions per request" width="45%">
+</p>
+
+<p align="center"><sub>Jev 1.13.0 (hosted), Laya 0.3.4 and Kev 0.8B / 4B / 9B (local, GB10) on identical generated manifests, n = 500, September 2026. Left to right: accuracy across five paraphrases and criteria variants of the same question (red = canonical wording); reliability on the same question; accuracy as the option count grows from 2 to 255; median latency as more questions share one request. Hosted latency includes the network path and is not comparable to local compute time.</sub></p>
+
+## Why this benchmark
+
+| | |
+|---|---|
+| **Policy-labelled data** | Labels follow a rule stated in the question (e.g. *urgency + 1 if angry + 1 if premium tier, cap 4*). A regex cannot solve it; neither can memorised public sets. Control arms are paired item-for-item: unknowable, label-noise, distractor, none-of-the-above. |
+| **Calibration relative to noise** | Every ECE is divided by the ECE a perfectly calibrated model would show on that sample. Brier decomposition, clipped NLL, per-primitive temperature refit, quantisation report. |
+| **Wording as a factor** | Accuracy is the median over five paraphrases and three criteria variants, with the range shown; adversarial wordings reported as a worst case. |
+| **Selective prediction and cost** | Risk–coverage, coverage at 5 % risk, abstention with and without an explicit option, and realised cost per 10 k decisions under argmax, Bayes and escalate policies from per-question cost matrices. |
+| **Ordinal, not categorical** | MAE, weighted kappa, ranked probability score, monotonicity along controlled severity ladders, 3/5/10-level scale invariance. |
+| **Honest infrastructure** | Hosted and local latency never share an axis; the serving device is recorded on every row; a local model that lands on CPU when CUDA was requested aborts the run; every response is cached and re-derivable offline. |
+
+## Add your model
+
+A hosted model with a conventional JSON decisions API needs a config file, no code:
+
+```bash
+sys1bench configs example_future_vendor > my_model.yaml   # fill in url, auth env var, field names
+```
+
+Anything else subclasses `BaseAdapter` (declare capabilities, implement `decide`) and registers with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. Kev, which speaks the TypeSafe API, runs with `--config kev_4b` and no code at all.
+
+## Documentation
+
+- [Specification](docs/SPEC_v2.md) — contract, data tiers, suites A–I, statistics, audits, rules for future models
+- [Design review](docs/REVIEW_v1.md) — how the design was derived from the public state of the art
+- [Changelog](CHANGELOG.md) · [Issues](https://github.com/rssr25/system-one-bench/issues)
+
+<p align="center"><sub>Apache 2.0 · Rahul Sharma · numbers change with model versions, so every row carries the version string the provider returned</sub></p>

sys1bench-0.3.2/README.md

@@ -0,0 +1,65 @@
+<p align="center">
+  <img src="docs/assets/logo.svg" alt="sys1bench" width="480">
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
+</p>
+
+<p align="center"><b>A benchmark for typed System One decision models</b><br>
+Jev · Laya · Kev · and whatever comes next</p>
+
+---
+
+System One models read a block of state plus typed questions and return a probability distribution over the answers you defined — one of N options, a level on an ordinal scale, or yes/no — in a single forward pass. No text is generated, so nothing has to be parsed. **sys1bench** measures what that buys: whether the probabilities are calibrated (against a noise floor), how much the answer depends on how the question is worded, how confidence behaves out of scope, how accuracy scales with options and state length, whether batched questions interfere, and what it all costs per decision. Headline numbers come from generated items whose labels follow a stated policy, so they cannot be memorised.
+
+```bash
+pip install sys1bench
+sys1bench suite all results/my-model --config my_model.yaml        # A–I on generated, policy-labelled data
+sys1bench report results --html results/dashboard.html            # tables + interactive dashboard
+```
+
+## Results at a glance
+
+<p align="center">
+  <img src="docs/figures/framing_tickets_priority.png" alt="Accuracy across question wordings on the policy-following priority score" width="49%">
+  <img src="docs/figures/reliability_tickets_priority.png" alt="Reliability diagram, priority score, all models" width="41%">
+</p>
+<p align="center">
+  <img src="docs/figures/cardinality_accuracy.png" alt="Exact accuracy versus number of options" width="45%">
+  <img src="docs/figures/latency_vs_questions.png" alt="Median latency versus number of questions per request" width="45%">
+</p>
+
+<p align="center"><sub>Jev 1.13.0 (hosted), Laya 0.3.4 and Kev 0.8B / 4B / 9B (local, GB10) on identical generated manifests, n = 500, September 2026. Left to right: accuracy across five paraphrases and criteria variants of the same question (red = canonical wording); reliability on the same question; accuracy as the option count grows from 2 to 255; median latency as more questions share one request. Hosted latency includes the network path and is not comparable to local compute time.</sub></p>
+
+## Why this benchmark
+
+| | |
+|---|---|
+| **Policy-labelled data** | Labels follow a rule stated in the question (e.g. *urgency + 1 if angry + 1 if premium tier, cap 4*). A regex cannot solve it; neither can memorised public sets. Control arms are paired item-for-item: unknowable, label-noise, distractor, none-of-the-above. |
+| **Calibration relative to noise** | Every ECE is divided by the ECE a perfectly calibrated model would show on that sample. Brier decomposition, clipped NLL, per-primitive temperature refit, quantisation report. |
+| **Wording as a factor** | Accuracy is the median over five paraphrases and three criteria variants, with the range shown; adversarial wordings reported as a worst case. |
+| **Selective prediction and cost** | Risk–coverage, coverage at 5 % risk, abstention with and without an explicit option, and realised cost per 10 k decisions under argmax, Bayes and escalate policies from per-question cost matrices. |
+| **Ordinal, not categorical** | MAE, weighted kappa, ranked probability score, monotonicity along controlled severity ladders, 3/5/10-level scale invariance. |
+| **Honest infrastructure** | Hosted and local latency never share an axis; the serving device is recorded on every row; a local model that lands on CPU when CUDA was requested aborts the run; every response is cached and re-derivable offline. |
+
+## Add your model
+
+A hosted model with a conventional JSON decisions API needs a config file, no code:
+
+```bash
+sys1bench configs example_future_vendor > my_model.yaml   # fill in url, auth env var, field names
+```
+
+Anything else subclasses `BaseAdapter` (declare capabilities, implement `decide`) and registers with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. Kev, which speaks the TypeSafe API, runs with `--config kev_4b` and no code at all.
+
+## Documentation
+
+- [Specification](docs/SPEC_v2.md) — contract, data tiers, suites A–I, statistics, audits, rules for future models
+- [Design review](docs/REVIEW_v1.md) — how the design was derived from the public state of the art
+- [Changelog](CHANGELOG.md) · [Issues](https://github.com/rssr25/system-one-bench/issues)
+
+<p align="center"><sub>Apache 2.0 · Rahul Sharma · numbers change with model versions, so every row carries the version string the provider returned</sub></p>

{sys1bench-0.3.0 → sys1bench-0.3.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sys1bench"
-version = "0.3.0"
+version = "0.3.2"
 description = "Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -41,7 +41,6 @@ dev = ["pytest>=8.0", "ruff>=0.6", "matplotlib>=3.8", "build>=1.2", "twine>=5.0"
 [project.urls]
 Homepage = "https://github.com/rssr25/system-one-bench"
 Documentation = "https://github.com/rssr25/system-one-bench/blob/main/docs/SPEC_v2.md"
-Results = "https://github.com/rssr25/system-one-bench/blob/main/docs/RESULTS_2026-09-22_narrative.md"
 Issues = "https://github.com/rssr25/system-one-bench/issues"
 Changelog = "https://github.com/rssr25/system-one-bench/blob/main/CHANGELOG.md"
 
@@ -59,7 +58,7 @@ build-backend = "setuptools.build_meta"
 where = ["src"]
 
 [tool.setuptools.package-data]
-"sys1bench.data" = ["framings/*.yaml", "canary/*.jsonl"]
+"sys1bench.data" = ["framings/*.yaml", "canary/*.jsonl", "configs/*.yaml"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

{sys1bench-0.3.0 → sys1bench-0.3.2}/src/sys1bench/__init__.py

@@ -1,3 +1,3 @@
 """sys1bench: benchmark harness for typed System One decision models."""
 
-__version__ = "0.3.0"
+__version__ = "0.3.2"

{sys1bench-0.3.0 → sys1bench-0.3.2}/src/sys1bench/adapters/jev_typesafe.py

@@ -27,7 +27,7 @@ from typing import Any
 import httpx
 
 from ..schemas import Answer, DecisionRequest, DecisionResponse, LatencyRecord, ModelCapabilities, ProviderRecord, Question
-from .base import BaseAdapter, finalize_answer, register
+from .base import BaseAdapter, FatalAdapterError, finalize_answer, register
 
 DEFAULT_URL = "https://api.typesafe.ai/v1/systemone"
 PRICE_PER_M_INPUT = 0.042
@@ -38,7 +38,7 @@ def _load_dotenv_key() -> str | None:
     for k in ENV_KEYS:
         if os.environ.get(k):
             return os.environ[k]
-    for candidate in (Path.cwd() / ".env", Path(__file__).resolve().parents[3] / ".env"):
+    for candidate in (Path.cwd() / ".env",):
         if candidate.exists():
             for line in candidate.read_text().splitlines():
                 line = line.strip()
@@ -89,17 +89,46 @@ def parse_answer(q: Question, payload: dict[str, Any]) -> Answer:
 @register("jev_typesafe")
 class JevTypeSafeAdapter(BaseAdapter):
     def __init__(self, model_id: str = "jev-1.13.0", url: str = DEFAULT_URL, api_key: str | None = None,
-                 timeout_s: float = 30.0, max_retries: int = 6, allow_alias: bool = False, **tunables) -> None:
+                 timeout_s: float = 30.0, max_retries: int = 6, allow_alias: bool = False, version_note: str | None = None,
+                 probe_models: bool = False, deployment: str = "hosted", hardware: str | None = None, **tunables) -> None:
+        """Also serves any TypeSafe-API-compatible server (e.g. Kev at http://127.0.0.1:8009/v1/systemone): set `url`,
+        `api_key: local`, `allow_alias: true`, `deployment: local`, and `probe_models: true` to record what the server
+        reports at /v1/models (backend, dtype, revision) as the version hash."""
         super().__init__(model_id, **tunables)
         if (model_id.endswith("latest") or model_id.endswith("preview")) and not allow_alias:
             raise ValueError("pin a versioned Jev id (e.g. jev-1.13.0); aliases move silently. Pass allow_alias=True to override.")
         self.url, self.timeout_s, self.max_retries = url, timeout_s, max_retries
+        self.deployment, self.hardware = deployment, hardware
         self.api_key = api_key or _load_dotenv_key()
+        if not self.api_key:
+            raise FatalAdapterError("No TypeSafe API key found. Set TypeSafe_API_KEY (or TYPESAFE_API_KEY) in the environment or in a .env file "
+                                    "in the working directory. Keys: https://typesafe.ai")
         self._client = httpx.Client(timeout=timeout_s, http2=False)
+        self.version_note = version_note
+        self.model_label: str | None = None
+        if probe_models:
+            try:
+                base = url.split("/v1/")[0]
+                r = self._client.get(f"{base}/v1/models", headers={"Authorization": f"Bearer {self.api_key}"})
+                r.raise_for_status()
+                data = r.json()
+                models = data.get("models") or data.get("data") or []
+                mine = next((m for m in models if m.get("name") == model_id or m.get("id") == model_id), models[0] if models else {})
+                bits = [str(mine.get(k)) for k in ("name", "id", "revision", "run", "backend", "dtype", "release_date") if mine.get(k)]
+                bits += [f"{k}={v}" for k, v in data.items() if k not in ("models", "data") and isinstance(v, (str, int, float))]
+                self.version_note = " ".join(bits)[:200] or version_note
+                self.tunables["server_models"] = self.version_note
+                # a server that only exposes an alias (Kev: "kev-latest") is labelled by the run it reports, so that
+                # several checkpoints behind the same alias do not merge in reports
+                run = mine.get("run") or mine.get("revision")
+                if run and (model_id.endswith("latest") or model_id.endswith("preview")):
+                    self.model_label = str(run).split("/")[-1]
+            except Exception as e:  # probing is best effort
+                self.tunables["server_models_error"] = str(e)[:120]
 
     @property
     def capabilities(self) -> ModelCapabilities:
-        return ModelCapabilities(name=self.model_id, deployment="hosted", max_options=255, max_score_levels=10,
+        return ModelCapabilities(name=self.model_id, deployment=self.deployment, max_options=255, max_score_levels=10,  # type: ignore[arg-type]
                                  max_state_tokens=32_000, emits_confidence=True, supports_native_abstain=False,
                                  supports_batching=True, languages={"en"})
 
@@ -121,6 +150,8 @@ class JevTypeSafeAdapter(BaseAdapter):
                         except ValueError:
                             pass
                     raise httpx.HTTPStatusError(f"{r.status_code}: {r.text[:200]}", request=r.request, response=r)
+                if r.status_code in (401, 403):
+                    raise FatalAdapterError(f"TypeSafe API rejected the key (HTTP {r.status_code}). Check TypeSafe_API_KEY.")
                 if 400 <= r.status_code < 500 and r.status_code != 429:
                     # request rejected (400 over the 32k state limit, 413, 422 malformed): never retry, classify as vendor rejection
                     raise ValueError(f"{r.status_code} rejected: {r.text[:300]}")
@@ -140,6 +171,8 @@ class JevTypeSafeAdapter(BaseAdapter):
                 "questions": {k: to_vendor_question(q) for k, q in request.questions.items()}}
         try:
             data, ms, headers = self._post(body)
+        except FatalAdapterError:
+            raise
         except Exception as e:
             kind = "rejected_by_vendor" if isinstance(e, ValueError) else "transport"
             return DecisionResponse(
@@ -156,9 +189,11 @@ class JevTypeSafeAdapter(BaseAdapter):
         return DecisionResponse(
             answers=answers,
             latency=LatencyRecord(client_ms=ms, route="typesafe", timestamp=time.time()),
-            provider=ProviderRecord(adapter_id=self.adapter_id, model_id_requested=self.model_id, model_id_returned=data.get("model"),
-                                    version_hash=str(data.get("model") or self.model_id),
-                                    generation_id=headers.get("x-request-id") or headers.get("request-id"), route="typesafe",
+            provider=ProviderRecord(adapter_id=self.adapter_id, model_id_requested=self.model_id,
+                                    model_id_returned=self.model_label or data.get("model"),
+                                    version_hash=(str(data.get("model") or self.model_id) + (f" | {self.version_note}" if self.version_note else "")),
+                                    generation_id=headers.get("x-request-id") or headers.get("x-typesafe-request-id") or headers.get("request-id"),
+                                    route="typesafe" if self.url == DEFAULT_URL else self.url, hardware=self.hardware,
                                     billed_input_tokens=in_tok, billed_output_tokens=usage.get("output_tokens"),
                                     cost_usd=(in_tok / 1e6 * PRICE_PER_M_INPUT) if in_tok is not None else None),
             raw=data,

{sys1bench-0.3.0 → sys1bench-0.3.2}/src/sys1bench/cli.py

@@ -17,7 +17,7 @@ import yaml
 
 from . import __version__
 from .adapters import get_adapter, list_adapters
-from .data import framings_path
+from .data import canary_path, config_path, framings_path, list_configs
 from .framing import apply_corruption, expand_framings, permute_options, strip_options, strip_state
 from .framing.expand import load_framings
 from .generators import GENERATORS, get_generator
@@ -38,14 +38,31 @@ def _main(ctx: typer.Context, version: bool = typer.Option(False, "--version", "
         typer.echo(ctx.get_help())
 
 
+def _fail(msg: str, code: int = 2) -> None:
+    typer.secho(f"error: {msg}", fg=typer.colors.RED, err=True)
+    raise typer.Exit(code)
+
+
 def _adapter_from(adapter: str | None, model: str | None, config: str | None):
+    from .adapters.base import FatalAdapterError
+
     cfg = {}
     if config:
-        cfg = yaml.safe_load(Path(config).read_text()) or {}
-    adapter = adapter or cfg.pop("adapter")
+        try:
+            cfg = yaml.safe_load(config_path(config).read_text()) or {}
+        except FileNotFoundError as e:
+            _fail(str(e))
+    adapter = adapter or cfg.pop("adapter", None)
+    if not adapter:
+        _fail("no adapter given: pass --adapter <id> or --config <yaml with an `adapter:` key>. Known: " + ", ".join(list_adapters()))
     if model:
         cfg["model_id"] = model
-    return get_adapter(adapter, **cfg)
+    try:
+        return get_adapter(adapter, **cfg)
+    except KeyError as e:
+        _fail(str(e).strip('"'))
+    except (ValueError, FatalAdapterError) as e:
+        _fail(str(e))
 
 
 @app.command()
@@ -55,8 +72,19 @@ def adapters():
         typer.echo(a)
 
 
+@app.command()
+def configs(show: Optional[str] = typer.Argument(None, help="print a packaged config by name")):
+    """List packaged example model configs (use with --config <name>), or print one to copy and edit."""
+    if show:
+        typer.echo(config_path(show).read_text())
+        return
+    for c in sorted(list_configs()):
+        typer.echo(c)
+
+
 @app.command()
 def generators():
+    """List Tier G generators and their versions."""
     for g in sorted(GENERATORS):
         typer.echo(f"{g}@{GENERATORS[g].version}")
 
@@ -93,14 +121,22 @@ def run(manifest: Path, out: Path, adapter: str | None = None, model: str | None
     if permutations:
         items = permute_options(items, permutations)
     c = ResponseCache(cache)
-    rows_all += run_items(items, ad, c, suite=suite, arm="main", concurrency=concurrency, progress=True)
-    base = [i for i in items if i.permutation_id == "p0" and all(q.framing_id == "f0" for q in i.questions.values())]
-    if corruption:
-        rows_all += run_items(apply_corruption(base), ad, c, suite=suite, arm="corruption", concurrency=concurrency)
-    if short_circuit:
-        rows_all += run_items(strip_state(base), ad, c, suite=suite, arm="state_only", concurrency=concurrency)
-        rows_all += run_items(strip_options(base), ad, c, suite=suite, arm="options_only", concurrency=concurrency)
+    from .adapters.base import FatalAdapterError
+
+    try:
+        rows_all += run_items(items, ad, c, suite=suite, arm="main", concurrency=concurrency, progress=True)
+        base = [i for i in items if i.permutation_id == "p0" and all(q.framing_id == "f0" for q in i.questions.values())]
+        if corruption:
+            rows_all += run_items(apply_corruption(base), ad, c, suite=suite, arm="corruption", concurrency=concurrency)
+        if short_circuit:
+            rows_all += run_items(strip_state(base), ad, c, suite=suite, arm="state_only", concurrency=concurrency)
+            rows_all += run_items(strip_options(base), ad, c, suite=suite, arm="options_only", concurrency=concurrency)
+    except FatalAdapterError as e:
+        _fail(str(e), code=3)
     write_predictions(rows_all, out)
+    n_err = sum(1 for r in rows_all if r.error)
+    if n_err:
+        typer.secho(f"warning: {n_err}/{len(rows_all)} rows have errors (see the `error` field); first: {next(r.error for r in rows_all if r.error)}", fg=typer.colors.YELLOW, err=True)
     typer.echo(f"wrote {len(rows_all)} prediction rows to {out}; cache size {len(c)}")
 
 
@@ -124,11 +160,11 @@ def score(predictions: list[Path], out: Path = Path("results/summary.json"), tab
 
 
 @app.command()
-def canary(adapter: str, model: str | None = None, manifest: Path = Path("data/canary/canary.jsonl"),
+def canary(adapter: str, model: str | None = None, manifest: Path | None = None,
            store: Path = Path("results/canary"), config: Path | None = None):
-    """Run the drift canary for a hosted model."""
+    """Run the drift canary for a hosted model (default: the packaged 200-item set)."""
     ad = _adapter_from(adapter, model, str(config) if config else None)
-    res = run_canary(load_manifest(manifest), ad, store)
+    res = run_canary(load_manifest(manifest or canary_path()), ad, store)
     typer.echo(json.dumps(res, indent=1))
     if res["drift_suspected"]:
         raise typer.Exit(code=2)
@@ -259,7 +295,8 @@ def robustness(out: Path, adapter: Optional[str] = None, model: Optional[str] =
 
 
 @app.command()
-def plots(results_dir: Path, out: Optional[Path] = None, questions: str = "tickets.queue,tickets.is_angry,tickets.priority,phish.is_phishing,phish.attack_class,phish.urgency"):
+def plots(results_dir: Path, out: Optional[Path] = None, questions: str = "tickets.queue,tickets.is_angry,tickets.priority,phish.is_phishing,phish.attack_class,phish.urgency",
+          include: Optional[str] = typer.Option(None, help="comma-separated substrings; only model ids matching one of them are drawn")):
     """Render reliability diagrams, risk-coverage curves and framing-range charts for every model dir under results_dir."""
     from .report.plots import framing_range_plot, reliability_diagram, risk_coverage_plot
     from .report.scorecard import framing_scorecard
@@ -279,6 +316,8 @@ def plots(results_dir: Path, out: Optional[Path] = None, questions: str = "ticke
             if not rows:
                 continue
             label = rows[0].model_id
+            if include and not any(tok.strip().lower() in label.lower() for tok in include.split(",")):
+                continue
             by_model[label] = [r for r in rows if r.permutation_id == "p0" and r.framing_id == "f0"]
             acc_by_model[label] = framing_scorecard(rows)["accuracy_by_framing"]
         if not by_model:

{sys1bench-0.3.0 → sys1bench-0.3.2}/src/sys1bench/data/__init__.py

@@ -22,10 +22,10 @@ def framings_path(name_or_path: str | Path) -> Path:
     p = Path(name_or_path)
     if p.exists():
         return p
-    stem = p.stem if p.suffix else str(name_or_path)
-    cand = _pkg_file("framings", f"{stem}.yaml")
-    if cand.exists():
-        return cand
+    for stem in (str(name_or_path), p.stem):  # names may contain dots (jev_1.13), so try the full name first
+        cand = _pkg_file("framings", f"{stem}.yaml")
+        if cand.exists():
+            return cand
     raise FileNotFoundError(f"no framing set {name_or_path!r}; packaged: {sorted(list_framings())}")
 
 
@@ -35,3 +35,20 @@ def list_framings() -> list[str]:
 
 def canary_path() -> Path:
     return _pkg_file("canary", "canary.jsonl")
+
+
+def config_path(name_or_path: str | Path) -> Path:
+    """Resolve a model config: an existing path, or a packaged example in `configs/<name>.yaml`
+    (jev_1.13, jev_1.13_openrouter, laya_en, laya_typed_decisions, example_future_vendor, hybrid_mock)."""
+    p = Path(name_or_path)
+    if p.exists():
+        return p
+    for stem in (str(name_or_path), p.stem):
+        cand = _pkg_file("configs", f"{stem}.yaml")
+        if cand.exists():
+            return cand
+    raise FileNotFoundError(f"no config {name_or_path!r}; packaged: {sorted(list_configs())}")
+
+
+def list_configs() -> list[str]:
+    return [p.stem for p in Path(str(resources.files(_PKG).joinpath("configs"))).glob("*.yaml")]

sys1bench-0.3.2/src/sys1bench/data/configs/example_future_vendor.yaml

@@ -0,0 +1,18 @@
+# Template for a new hosted System One model: no code needed if its JSON shape is conventional.
+adapter: generic_http
+model_id: vendor/decider-1.0
+url: https://api.vendor.example/v1/decisions
+auth_env: VENDOR_API_KEY
+type_map: {choice: choice, score: score, noul: boolean}
+body_template: {"model": "{model_id}", "state": "{state}", "questions": "{questions}"}
+answers_path: [answers]
+fields:
+  probs: [[probabilities], [distribution]]
+  p_true: [[probability]]
+  confidence: [[confidence]]
+  model_returned: [[model]]
+  generation_id: [[id]]
+  input_tokens: [[usage, input_tokens]]
+price_per_m_input: 0.05
+max_options: 255
+max_state_tokens: 32000

sys1bench-0.3.2/src/sys1bench/data/configs/hybrid_mock.yaml

@@ -0,0 +1,4 @@
+adapter: hybrid_router
+threshold: 0.7
+primary: {adapter: mock, skill: 0.8, temperature: 0.8, latency_ms: 40}
+fallback: {adapter: mock, skill: 0.95, latency_ms: 1500}

sys1bench-0.3.2/src/sys1bench/data/configs/jev_1.13.yaml

@@ -0,0 +1,4 @@
+adapter: jev_typesafe
+model_id: jev-1.13.0            # versioned id; never jev-latest / jev-preview
+timeout_s: 30
+max_retries: 6

sys1bench-0.3.2/src/sys1bench/data/configs/jev_1.13_openrouter.yaml

@@ -0,0 +1,3 @@
+adapter: jev_openrouter
+model_id: typesafe/jev-1.13     # OpenRouter alpha Decisions API route (needs OPENROUTER_API_KEY)
+route: openrouter

sys1bench-0.3.2/src/sys1bench/data/configs/kev_0.8b.yaml

@@ -0,0 +1,12 @@
+# Kev (github.com/jaredpalmer/kev): a Jev-compatible decision model served locally.
+#   uv run --extra serve python -m kev.serve --run jaredpalmer/kev-0.8b --port 8009
+adapter: jev_typesafe
+model_id: kev-latest          # Kev's only served name; the revision is recorded from /v1/models
+url: http://127.0.0.1:8009/v1/systemone
+api_key: local
+allow_alias: true
+probe_models: true
+deployment: local
+version_note: jaredpalmer/kev-0.8b
+timeout_s: 120
+max_retries: 3

sys1bench-0.3.2/src/sys1bench/data/configs/kev_4b.yaml

@@ -0,0 +1,12 @@
+# Kev (github.com/jaredpalmer/kev): a Jev-compatible decision model served locally.
+#   uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009
+adapter: jev_typesafe
+model_id: kev-latest          # Kev's only served name; the revision is recorded from /v1/models
+url: http://127.0.0.1:8009/v1/systemone
+api_key: local
+allow_alias: true
+probe_models: true
+deployment: local
+version_note: jaredpalmer/kev-4b
+timeout_s: 120
+max_retries: 3

sys1bench-0.3.2/src/sys1bench/data/configs/kev_9b.yaml

@@ -0,0 +1,12 @@
+# Kev (github.com/jaredpalmer/kev): a Jev-compatible decision model served locally.
+#   uv run --extra serve python -m kev.serve --run jaredpalmer/kev-9b --port 8009
+adapter: jev_typesafe
+model_id: kev-latest          # Kev's only served name; the revision is recorded from /v1/models
+url: http://127.0.0.1:8009/v1/systemone
+api_key: local
+allow_alias: true
+probe_models: true
+deployment: local
+version_note: jaredpalmer/kev-9b
+timeout_s: 120
+max_retries: 3

sys1bench-0.3.2/src/sys1bench/data/configs/laya_en.yaml

@@ -0,0 +1,6 @@
+adapter: laya_local
+model_id: convaiinnovations/laya
+checkpoint: english             # english | multilingual | typed-decisions | router
+device: cuda                    # falls back to CPU if CUDA is unavailable; the device used is recorded per row
+head_max_len: 192               # Suite C sweeps [64,128,192,256,384,512]
+max_len: 512

sys1bench-0.3.2/src/sys1bench/data/configs/laya_typed_decisions.yaml

@@ -0,0 +1,6 @@
+adapter: laya_local
+model_id: convaiinnovations/laya
+checkpoint: typed-decisions
+device: cuda
+head_max_len: 256
+max_len: 1024

{sys1bench-0.3.0 → sys1bench-0.3.2}/src/sys1bench/report/plots.py

@@ -92,7 +92,7 @@ def framing_range_plot(acc_by_model_framing: dict[str, dict[str, float]], out: P
             ax.plot(v, i, marker="o" if f == "f0" else ".", color="#c33" if f == "f0" else "#333", ms=7 if f == "f0" else 5)
     ax.set_yticks(range(len(models)))
     ax.set_yticklabels(models, fontsize=8)
-    ax.set_xlabel("accuracy (red = canonical framing, dots = paraphrases / criteria variants)")
+    ax.set_xlabel("accuracy  (red: canonical wording, dots: other framings)", fontsize=8)
     ax.set_title(title, fontsize=10)
     ax.set_xlim(0, 1.02)
     fig.tight_layout()
@@ -141,3 +141,28 @@ def interference_heatmap(res: dict[str, Any], out: Path, metric: str = "mean_jsd
     fig.savefig(out, dpi=160)
     plt.close(fig)
     return out
+
+
+def multi_line_plot(series: dict[str, list[tuple[float, float]]], out: Path, xlabel: str, ylabel: str, title: str = "",
+                    xlog: bool = False, ylim: tuple[float, float] | None = None) -> Path:
+    """Several named series on one axis; used for cross-model sweep comparisons in the README."""
+    plt = _plt()
+    fig, ax = plt.subplots(figsize=(5.2, 3.4))
+    for name, pts in series.items():
+        pts = sorted(p for p in pts if p[1] == p[1])
+        if pts:
+            ax.plot([p[0] for p in pts], [p[1] for p in pts], marker="o", ms=4, lw=1.6, label=name)
+    if xlog:
+        ax.set_xscale("log")
+    if ylim:
+        ax.set_ylim(*ylim)
+    ax.set_xlabel(xlabel)
+    ax.set_ylabel(ylabel)
+    ax.set_title(title, fontsize=10)
+    ax.grid(alpha=0.25)
+    ax.legend(fontsize=7, frameon=False)
+    fig.tight_layout()
+    out.parent.mkdir(parents=True, exist_ok=True)
+    fig.savefig(out, dpi=170)
+    plt.close(fig)
+    return out