sys1bench 0.3.1__tar.gz → 0.3.2__tar.gz

sys1bench-0.3.2/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: sys1bench
+Version: 0.3.2
+Summary: Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness.
+Author-email: Rahul Sharma <rsharma@rptu.de>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/rssr25/system-one-bench
+Project-URL: Documentation, https://github.com/rssr25/system-one-bench/blob/main/docs/SPEC_v2.md
+Project-URL: Issues, https://github.com/rssr25/system-one-bench/issues
+Project-URL: Changelog, https://github.com/rssr25/system-one-bench/blob/main/CHANGELOG.md
+Keywords: benchmark,calibration,system-one,decision-models,jev,laya,evaluation,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Operating System :: OS Independent
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: scipy>=1.11
+Requires-Dist: pandas>=2.0
+Requires-Dist: pydantic>=2.5
+Requires-Dist: httpx>=0.27
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typer>=0.12
+Provides-Extra: plots
+Requires-Dist: matplotlib>=3.8; extra == "plots"
+Provides-Extra: local
+Requires-Dist: torch>=2.4; extra == "local"
+Requires-Dist: transformers>=4.45; extra == "local"
+Requires-Dist: sentence-transformers>=3.0; extra == "local"
+Provides-Extra: laya
+Requires-Dist: laya>=0.3; extra == "laya"
+Provides-Extra: llm
+Requires-Dist: vllm>=0.6; extra == "llm"
+Requires-Dist: outlines>=0.1; extra == "llm"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: matplotlib>=3.8; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+<p align="center">
+  <img src="docs/assets/logo.svg" alt="sys1bench" width="480">
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
+</p>
+
+<p align="center"><b>A benchmark for typed System One decision models</b><br>
+Jev · Laya · Kev · and whatever comes next</p>
+
+---
+
+System One models read a block of state plus typed questions and return a probability distribution over the answers you defined — one of N options, a level on an ordinal scale, or yes/no — in a single forward pass. No text is generated, so nothing has to be parsed. **sys1bench** measures what that buys: whether the probabilities are calibrated (against a noise floor), how much the answer depends on how the question is worded, how confidence behaves out of scope, how accuracy scales with options and state length, whether batched questions interfere, and what it all costs per decision. Headline numbers come from generated items whose labels follow a stated policy, so they cannot be memorised.
+
+```bash
+pip install sys1bench
+sys1bench suite all results/my-model --config my_model.yaml        # A–I on generated, policy-labelled data
+sys1bench report results --html results/dashboard.html            # tables + interactive dashboard
+```
+
+## Results at a glance
+
+<p align="center">
+  <img src="docs/figures/framing_tickets_priority.png" alt="Accuracy across question wordings on the policy-following priority score" width="49%">
+  <img src="docs/figures/reliability_tickets_priority.png" alt="Reliability diagram, priority score, all models" width="41%">
+</p>
+<p align="center">
+  <img src="docs/figures/cardinality_accuracy.png" alt="Exact accuracy versus number of options" width="45%">
+  <img src="docs/figures/latency_vs_questions.png" alt="Median latency versus number of questions per request" width="45%">
+</p>
+
+<p align="center"><sub>Jev 1.13.0 (hosted), Laya 0.3.4 and Kev 0.8B / 4B / 9B (local, GB10) on identical generated manifests, n = 500, September 2026. Left to right: accuracy across five paraphrases and criteria variants of the same question (red = canonical wording); reliability on the same question; accuracy as the option count grows from 2 to 255; median latency as more questions share one request. Hosted latency includes the network path and is not comparable to local compute time.</sub></p>
+
+## Why this benchmark
+
+| | |
+|---|---|
+| **Policy-labelled data** | Labels follow a rule stated in the question (e.g. *urgency + 1 if angry + 1 if premium tier, cap 4*). A regex cannot solve it; neither can memorised public sets. Control arms are paired item-for-item: unknowable, label-noise, distractor, none-of-the-above. |
+| **Calibration relative to noise** | Every ECE is divided by the ECE a perfectly calibrated model would show on that sample. Brier decomposition, clipped NLL, per-primitive temperature refit, quantisation report. |
+| **Wording as a factor** | Accuracy is the median over five paraphrases and three criteria variants, with the range shown; adversarial wordings reported as a worst case. |
+| **Selective prediction and cost** | Risk–coverage, coverage at 5 % risk, abstention with and without an explicit option, and realised cost per 10 k decisions under argmax, Bayes and escalate policies from per-question cost matrices. |
+| **Ordinal, not categorical** | MAE, weighted kappa, ranked probability score, monotonicity along controlled severity ladders, 3/5/10-level scale invariance. |
+| **Honest infrastructure** | Hosted and local latency never share an axis; the serving device is recorded on every row; a local model that lands on CPU when CUDA was requested aborts the run; every response is cached and re-derivable offline. |
+
+## Add your model
+
+A hosted model with a conventional JSON decisions API needs a config file, no code:
+
+```bash
+sys1bench configs example_future_vendor > my_model.yaml   # fill in url, auth env var, field names
+```
+
+Anything else subclasses `BaseAdapter` (declare capabilities, implement `decide`) and registers with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. Kev, which speaks the TypeSafe API, runs with `--config kev_4b` and no code at all.
+
+## Documentation
+
+- [Specification](docs/SPEC_v2.md) — contract, data tiers, suites A–I, statistics, audits, rules for future models
+- [Design review](docs/REVIEW_v1.md) — how the design was derived from the public state of the art
+- [Changelog](CHANGELOG.md) · [Issues](https://github.com/rssr25/system-one-bench/issues)
+
+<p align="center"><sub>Apache 2.0 · Rahul Sharma · numbers change with model versions, so every row carries the version string the provider returned</sub></p>

sys1bench-0.3.2/README.md

@@ -0,0 +1,65 @@
+<p align="center">
+  <img src="docs/assets/logo.svg" alt="sys1bench" width="480">
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
+</p>
+
+<p align="center"><b>A benchmark for typed System One decision models</b><br>
+Jev · Laya · Kev · and whatever comes next</p>
+
+---
+
+System One models read a block of state plus typed questions and return a probability distribution over the answers you defined — one of N options, a level on an ordinal scale, or yes/no — in a single forward pass. No text is generated, so nothing has to be parsed. **sys1bench** measures what that buys: whether the probabilities are calibrated (against a noise floor), how much the answer depends on how the question is worded, how confidence behaves out of scope, how accuracy scales with options and state length, whether batched questions interfere, and what it all costs per decision. Headline numbers come from generated items whose labels follow a stated policy, so they cannot be memorised.
+
+```bash
+pip install sys1bench
+sys1bench suite all results/my-model --config my_model.yaml        # A–I on generated, policy-labelled data
+sys1bench report results --html results/dashboard.html            # tables + interactive dashboard
+```
+
+## Results at a glance
+
+<p align="center">
+  <img src="docs/figures/framing_tickets_priority.png" alt="Accuracy across question wordings on the policy-following priority score" width="49%">
+  <img src="docs/figures/reliability_tickets_priority.png" alt="Reliability diagram, priority score, all models" width="41%">
+</p>
+<p align="center">
+  <img src="docs/figures/cardinality_accuracy.png" alt="Exact accuracy versus number of options" width="45%">
+  <img src="docs/figures/latency_vs_questions.png" alt="Median latency versus number of questions per request" width="45%">
+</p>
+
+<p align="center"><sub>Jev 1.13.0 (hosted), Laya 0.3.4 and Kev 0.8B / 4B / 9B (local, GB10) on identical generated manifests, n = 500, September 2026. Left to right: accuracy across five paraphrases and criteria variants of the same question (red = canonical wording); reliability on the same question; accuracy as the option count grows from 2 to 255; median latency as more questions share one request. Hosted latency includes the network path and is not comparable to local compute time.</sub></p>
+
+## Why this benchmark
+
+| | |
+|---|---|
+| **Policy-labelled data** | Labels follow a rule stated in the question (e.g. *urgency + 1 if angry + 1 if premium tier, cap 4*). A regex cannot solve it; neither can memorised public sets. Control arms are paired item-for-item: unknowable, label-noise, distractor, none-of-the-above. |
+| **Calibration relative to noise** | Every ECE is divided by the ECE a perfectly calibrated model would show on that sample. Brier decomposition, clipped NLL, per-primitive temperature refit, quantisation report. |
+| **Wording as a factor** | Accuracy is the median over five paraphrases and three criteria variants, with the range shown; adversarial wordings reported as a worst case. |
+| **Selective prediction and cost** | Risk–coverage, coverage at 5 % risk, abstention with and without an explicit option, and realised cost per 10 k decisions under argmax, Bayes and escalate policies from per-question cost matrices. |
+| **Ordinal, not categorical** | MAE, weighted kappa, ranked probability score, monotonicity along controlled severity ladders, 3/5/10-level scale invariance. |
+| **Honest infrastructure** | Hosted and local latency never share an axis; the serving device is recorded on every row; a local model that lands on CPU when CUDA was requested aborts the run; every response is cached and re-derivable offline. |
+
+## Add your model
+
+A hosted model with a conventional JSON decisions API needs a config file, no code:
+
+```bash
+sys1bench configs example_future_vendor > my_model.yaml   # fill in url, auth env var, field names
+```
+
+Anything else subclasses `BaseAdapter` (declare capabilities, implement `decide`) and registers with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. Kev, which speaks the TypeSafe API, runs with `--config kev_4b` and no code at all.
+
+## Documentation
+
+- [Specification](docs/SPEC_v2.md) — contract, data tiers, suites A–I, statistics, audits, rules for future models
+- [Design review](docs/REVIEW_v1.md) — how the design was derived from the public state of the art
+- [Changelog](CHANGELOG.md) · [Issues](https://github.com/rssr25/system-one-bench/issues)
+
+<p align="center"><sub>Apache 2.0 · Rahul Sharma · numbers change with model versions, so every row carries the version string the provider returned</sub></p>

{sys1bench-0.3.1 → sys1bench-0.3.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sys1bench"
-version = "0.3.1"
+version = "0.3.2"
 description = "Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -41,7 +41,6 @@ dev = ["pytest>=8.0", "ruff>=0.6", "matplotlib>=3.8", "build>=1.2", "twine>=5.0"
 [project.urls]
 Homepage = "https://github.com/rssr25/system-one-bench"
 Documentation = "https://github.com/rssr25/system-one-bench/blob/main/docs/SPEC_v2.md"
-Results = "https://github.com/rssr25/system-one-bench/blob/main/docs/RESULTS_2026-09-22_narrative.md"
 Issues = "https://github.com/rssr25/system-one-bench/issues"
 Changelog = "https://github.com/rssr25/system-one-bench/blob/main/CHANGELOG.md"
 

{sys1bench-0.3.1 → sys1bench-0.3.2}/src/sys1bench/__init__.py

@@ -1,3 +1,3 @@
 """sys1bench: benchmark harness for typed System One decision models."""
 
-__version__ = "0.3.1"
+__version__ = "0.3.2"

{sys1bench-0.3.1 → sys1bench-0.3.2}/src/sys1bench/adapters/jev_typesafe.py

@@ -89,20 +89,46 @@ def parse_answer(q: Question, payload: dict[str, Any]) -> Answer:
 @register("jev_typesafe")
 class JevTypeSafeAdapter(BaseAdapter):
     def __init__(self, model_id: str = "jev-1.13.0", url: str = DEFAULT_URL, api_key: str | None = None,
-                 timeout_s: float = 30.0, max_retries: int = 6, allow_alias: bool = False, **tunables) -> None:
+                 timeout_s: float = 30.0, max_retries: int = 6, allow_alias: bool = False, version_note: str | None = None,
+                 probe_models: bool = False, deployment: str = "hosted", hardware: str | None = None, **tunables) -> None:
+        """Also serves any TypeSafe-API-compatible server (e.g. Kev at http://127.0.0.1:8009/v1/systemone): set `url`,
+        `api_key: local`, `allow_alias: true`, `deployment: local`, and `probe_models: true` to record what the server
+        reports at /v1/models (backend, dtype, revision) as the version hash."""
         super().__init__(model_id, **tunables)
         if (model_id.endswith("latest") or model_id.endswith("preview")) and not allow_alias:
             raise ValueError("pin a versioned Jev id (e.g. jev-1.13.0); aliases move silently. Pass allow_alias=True to override.")
         self.url, self.timeout_s, self.max_retries = url, timeout_s, max_retries
+        self.deployment, self.hardware = deployment, hardware
         self.api_key = api_key or _load_dotenv_key()
         if not self.api_key:
             raise FatalAdapterError("No TypeSafe API key found. Set TypeSafe_API_KEY (or TYPESAFE_API_KEY) in the environment or in a .env file "
                                     "in the working directory. Keys: https://typesafe.ai")
         self._client = httpx.Client(timeout=timeout_s, http2=False)
+        self.version_note = version_note
+        self.model_label: str | None = None
+        if probe_models:
+            try:
+                base = url.split("/v1/")[0]
+                r = self._client.get(f"{base}/v1/models", headers={"Authorization": f"Bearer {self.api_key}"})
+                r.raise_for_status()
+                data = r.json()
+                models = data.get("models") or data.get("data") or []
+                mine = next((m for m in models if m.get("name") == model_id or m.get("id") == model_id), models[0] if models else {})
+                bits = [str(mine.get(k)) for k in ("name", "id", "revision", "run", "backend", "dtype", "release_date") if mine.get(k)]
+                bits += [f"{k}={v}" for k, v in data.items() if k not in ("models", "data") and isinstance(v, (str, int, float))]
+                self.version_note = " ".join(bits)[:200] or version_note
+                self.tunables["server_models"] = self.version_note
+                # a server that only exposes an alias (Kev: "kev-latest") is labelled by the run it reports, so that
+                # several checkpoints behind the same alias do not merge in reports
+                run = mine.get("run") or mine.get("revision")
+                if run and (model_id.endswith("latest") or model_id.endswith("preview")):
+                    self.model_label = str(run).split("/")[-1]
+            except Exception as e:  # probing is best effort
+                self.tunables["server_models_error"] = str(e)[:120]
 
     @property
     def capabilities(self) -> ModelCapabilities:
-        return ModelCapabilities(name=self.model_id, deployment="hosted", max_options=255, max_score_levels=10,
+        return ModelCapabilities(name=self.model_id, deployment=self.deployment, max_options=255, max_score_levels=10,  # type: ignore[arg-type]
                                  max_state_tokens=32_000, emits_confidence=True, supports_native_abstain=False,
                                  supports_batching=True, languages={"en"})
 
@@ -163,9 +189,11 @@ class JevTypeSafeAdapter(BaseAdapter):
         return DecisionResponse(
             answers=answers,
             latency=LatencyRecord(client_ms=ms, route="typesafe", timestamp=time.time()),
-            provider=ProviderRecord(adapter_id=self.adapter_id, model_id_requested=self.model_id, model_id_returned=data.get("model"),
-                                    version_hash=str(data.get("model") or self.model_id),
-                                    generation_id=headers.get("x-request-id") or headers.get("request-id"), route="typesafe",
+            provider=ProviderRecord(adapter_id=self.adapter_id, model_id_requested=self.model_id,
+                                    model_id_returned=self.model_label or data.get("model"),
+                                    version_hash=(str(data.get("model") or self.model_id) + (f" | {self.version_note}" if self.version_note else "")),
+                                    generation_id=headers.get("x-request-id") or headers.get("x-typesafe-request-id") or headers.get("request-id"),
+                                    route="typesafe" if self.url == DEFAULT_URL else self.url, hardware=self.hardware,
                                     billed_input_tokens=in_tok, billed_output_tokens=usage.get("output_tokens"),
                                     cost_usd=(in_tok / 1e6 * PRICE_PER_M_INPUT) if in_tok is not None else None),
             raw=data,

{sys1bench-0.3.1 → sys1bench-0.3.2}/src/sys1bench/cli.py

@@ -295,7 +295,8 @@ def robustness(out: Path, adapter: Optional[str] = None, model: Optional[str] =
 
 
 @app.command()
-def plots(results_dir: Path, out: Optional[Path] = None, questions: str = "tickets.queue,tickets.is_angry,tickets.priority,phish.is_phishing,phish.attack_class,phish.urgency"):
+def plots(results_dir: Path, out: Optional[Path] = None, questions: str = "tickets.queue,tickets.is_angry,tickets.priority,phish.is_phishing,phish.attack_class,phish.urgency",
+          include: Optional[str] = typer.Option(None, help="comma-separated substrings; only model ids matching one of them are drawn")):
     """Render reliability diagrams, risk-coverage curves and framing-range charts for every model dir under results_dir."""
     from .report.plots import framing_range_plot, reliability_diagram, risk_coverage_plot
     from .report.scorecard import framing_scorecard
@@ -315,6 +316,8 @@ def plots(results_dir: Path, out: Optional[Path] = None, questions: str = "ticke
             if not rows:
                 continue
             label = rows[0].model_id
+            if include and not any(tok.strip().lower() in label.lower() for tok in include.split(",")):
+                continue
             by_model[label] = [r for r in rows if r.permutation_id == "p0" and r.framing_id == "f0"]
             acc_by_model[label] = framing_scorecard(rows)["accuracy_by_framing"]
         if not by_model:

sys1bench-0.3.2/src/sys1bench/data/configs/kev_0.8b.yaml

@@ -0,0 +1,12 @@
+# Kev (github.com/jaredpalmer/kev): a Jev-compatible decision model served locally.
+#   uv run --extra serve python -m kev.serve --run jaredpalmer/kev-0.8b --port 8009
+adapter: jev_typesafe
+model_id: kev-latest          # Kev's only served name; the revision is recorded from /v1/models
+url: http://127.0.0.1:8009/v1/systemone
+api_key: local
+allow_alias: true
+probe_models: true
+deployment: local
+version_note: jaredpalmer/kev-0.8b
+timeout_s: 120
+max_retries: 3

sys1bench-0.3.2/src/sys1bench/data/configs/kev_4b.yaml

@@ -0,0 +1,12 @@
+# Kev (github.com/jaredpalmer/kev): a Jev-compatible decision model served locally.
+#   uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009
+adapter: jev_typesafe
+model_id: kev-latest          # Kev's only served name; the revision is recorded from /v1/models
+url: http://127.0.0.1:8009/v1/systemone
+api_key: local
+allow_alias: true
+probe_models: true
+deployment: local
+version_note: jaredpalmer/kev-4b
+timeout_s: 120
+max_retries: 3

sys1bench-0.3.2/src/sys1bench/data/configs/kev_9b.yaml

@@ -0,0 +1,12 @@
+# Kev (github.com/jaredpalmer/kev): a Jev-compatible decision model served locally.
+#   uv run --extra serve python -m kev.serve --run jaredpalmer/kev-9b --port 8009
+adapter: jev_typesafe
+model_id: kev-latest          # Kev's only served name; the revision is recorded from /v1/models
+url: http://127.0.0.1:8009/v1/systemone
+api_key: local
+allow_alias: true
+probe_models: true
+deployment: local
+version_note: jaredpalmer/kev-9b
+timeout_s: 120
+max_retries: 3

{sys1bench-0.3.1 → sys1bench-0.3.2}/src/sys1bench/report/plots.py

@@ -92,7 +92,7 @@ def framing_range_plot(acc_by_model_framing: dict[str, dict[str, float]], out: P
             ax.plot(v, i, marker="o" if f == "f0" else ".", color="#c33" if f == "f0" else "#333", ms=7 if f == "f0" else 5)
     ax.set_yticks(range(len(models)))
     ax.set_yticklabels(models, fontsize=8)
-    ax.set_xlabel("accuracy (red = canonical framing, dots = paraphrases / criteria variants)")
+    ax.set_xlabel("accuracy  (red: canonical wording, dots: other framings)", fontsize=8)
     ax.set_title(title, fontsize=10)
     ax.set_xlim(0, 1.02)
     fig.tight_layout()
@@ -141,3 +141,28 @@ def interference_heatmap(res: dict[str, Any], out: Path, metric: str = "mean_jsd
     fig.savefig(out, dpi=160)
     plt.close(fig)
     return out
+
+
+def multi_line_plot(series: dict[str, list[tuple[float, float]]], out: Path, xlabel: str, ylabel: str, title: str = "",
+                    xlog: bool = False, ylim: tuple[float, float] | None = None) -> Path:
+    """Several named series on one axis; used for cross-model sweep comparisons in the README."""
+    plt = _plt()
+    fig, ax = plt.subplots(figsize=(5.2, 3.4))
+    for name, pts in series.items():
+        pts = sorted(p for p in pts if p[1] == p[1])
+        if pts:
+            ax.plot([p[0] for p in pts], [p[1] for p in pts], marker="o", ms=4, lw=1.6, label=name)
+    if xlog:
+        ax.set_xscale("log")
+    if ylim:
+        ax.set_ylim(*ylim)
+    ax.set_xlabel(xlabel)
+    ax.set_ylabel(ylabel)
+    ax.set_title(title, fontsize=10)
+    ax.grid(alpha=0.25)
+    ax.legend(fontsize=7, frameon=False)
+    fig.tight_layout()
+    out.parent.mkdir(parents=True, exist_ok=True)
+    fig.savefig(out, dpi=170)
+    plt.close(fig)
+    return out

sys1bench-0.3.2/src/sys1bench.egg-info/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: sys1bench
+Version: 0.3.2
+Summary: Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness.
+Author-email: Rahul Sharma <rsharma@rptu.de>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/rssr25/system-one-bench
+Project-URL: Documentation, https://github.com/rssr25/system-one-bench/blob/main/docs/SPEC_v2.md
+Project-URL: Issues, https://github.com/rssr25/system-one-bench/issues
+Project-URL: Changelog, https://github.com/rssr25/system-one-bench/blob/main/CHANGELOG.md
+Keywords: benchmark,calibration,system-one,decision-models,jev,laya,evaluation,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Operating System :: OS Independent
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: scipy>=1.11
+Requires-Dist: pandas>=2.0
+Requires-Dist: pydantic>=2.5
+Requires-Dist: httpx>=0.27
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typer>=0.12
+Provides-Extra: plots
+Requires-Dist: matplotlib>=3.8; extra == "plots"
+Provides-Extra: local
+Requires-Dist: torch>=2.4; extra == "local"
+Requires-Dist: transformers>=4.45; extra == "local"
+Requires-Dist: sentence-transformers>=3.0; extra == "local"
+Provides-Extra: laya
+Requires-Dist: laya>=0.3; extra == "laya"
+Provides-Extra: llm
+Requires-Dist: vllm>=0.6; extra == "llm"
+Requires-Dist: outlines>=0.1; extra == "llm"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: matplotlib>=3.8; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+<p align="center">
+  <img src="docs/assets/logo.svg" alt="sys1bench" width="480">
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
+</p>
+
+<p align="center"><b>A benchmark for typed System One decision models</b><br>
+Jev · Laya · Kev · and whatever comes next</p>
+
+---
+
+System One models read a block of state plus typed questions and return a probability distribution over the answers you defined — one of N options, a level on an ordinal scale, or yes/no — in a single forward pass. No text is generated, so nothing has to be parsed. **sys1bench** measures what that buys: whether the probabilities are calibrated (against a noise floor), how much the answer depends on how the question is worded, how confidence behaves out of scope, how accuracy scales with options and state length, whether batched questions interfere, and what it all costs per decision. Headline numbers come from generated items whose labels follow a stated policy, so they cannot be memorised.
+
+```bash
+pip install sys1bench
+sys1bench suite all results/my-model --config my_model.yaml        # A–I on generated, policy-labelled data
+sys1bench report results --html results/dashboard.html            # tables + interactive dashboard
+```
+
+## Results at a glance
+
+<p align="center">
+  <img src="docs/figures/framing_tickets_priority.png" alt="Accuracy across question wordings on the policy-following priority score" width="49%">
+  <img src="docs/figures/reliability_tickets_priority.png" alt="Reliability diagram, priority score, all models" width="41%">
+</p>
+<p align="center">
+  <img src="docs/figures/cardinality_accuracy.png" alt="Exact accuracy versus number of options" width="45%">
+  <img src="docs/figures/latency_vs_questions.png" alt="Median latency versus number of questions per request" width="45%">
+</p>
+
+<p align="center"><sub>Jev 1.13.0 (hosted), Laya 0.3.4 and Kev 0.8B / 4B / 9B (local, GB10) on identical generated manifests, n = 500, September 2026. Left to right: accuracy across five paraphrases and criteria variants of the same question (red = canonical wording); reliability on the same question; accuracy as the option count grows from 2 to 255; median latency as more questions share one request. Hosted latency includes the network path and is not comparable to local compute time.</sub></p>
+
+## Why this benchmark
+
+| | |
+|---|---|
+| **Policy-labelled data** | Labels follow a rule stated in the question (e.g. *urgency + 1 if angry + 1 if premium tier, cap 4*). A regex cannot solve it; neither can memorised public sets. Control arms are paired item-for-item: unknowable, label-noise, distractor, none-of-the-above. |
+| **Calibration relative to noise** | Every ECE is divided by the ECE a perfectly calibrated model would show on that sample. Brier decomposition, clipped NLL, per-primitive temperature refit, quantisation report. |
+| **Wording as a factor** | Accuracy is the median over five paraphrases and three criteria variants, with the range shown; adversarial wordings reported as a worst case. |
+| **Selective prediction and cost** | Risk–coverage, coverage at 5 % risk, abstention with and without an explicit option, and realised cost per 10 k decisions under argmax, Bayes and escalate policies from per-question cost matrices. |
+| **Ordinal, not categorical** | MAE, weighted kappa, ranked probability score, monotonicity along controlled severity ladders, 3/5/10-level scale invariance. |
+| **Honest infrastructure** | Hosted and local latency never share an axis; the serving device is recorded on every row; a local model that lands on CPU when CUDA was requested aborts the run; every response is cached and re-derivable offline. |
+
+## Add your model
+
+A hosted model with a conventional JSON decisions API needs a config file, no code:
+
+```bash
+sys1bench configs example_future_vendor > my_model.yaml   # fill in url, auth env var, field names
+```
+
+Anything else subclasses `BaseAdapter` (declare capabilities, implement `decide`) and registers with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. Kev, which speaks the TypeSafe API, runs with `--config kev_4b` and no code at all.
+
+## Documentation
+
+- [Specification](docs/SPEC_v2.md) — contract, data tiers, suites A–I, statistics, audits, rules for future models
+- [Design review](docs/REVIEW_v1.md) — how the design was derived from the public state of the art
+- [Changelog](CHANGELOG.md) · [Issues](https://github.com/rssr25/system-one-bench/issues)
+
+<p align="center"><sub>Apache 2.0 · Rahul Sharma · numbers change with model versions, so every row carries the version string the provider returned</sub></p>

{sys1bench-0.3.1 → sys1bench-0.3.2}/src/sys1bench.egg-info/SOURCES.txt

@@ -36,6 +36,9 @@ src/sys1bench/data/configs/example_future_vendor.yaml
 src/sys1bench/data/configs/hybrid_mock.yaml
 src/sys1bench/data/configs/jev_1.13.yaml
 src/sys1bench/data/configs/jev_1.13_openrouter.yaml
+src/sys1bench/data/configs/kev_0.8b.yaml
+src/sys1bench/data/configs/kev_4b.yaml
+src/sys1bench/data/configs/kev_9b.yaml
 src/sys1bench/data/configs/laya_en.yaml
 src/sys1bench/data/configs/laya_typed_decisions.yaml
 src/sys1bench/data/framings/guardrail_intent.yaml