sys1bench 0.3.0__tar.gz → 0.3.1__tar.gz

{sys1bench-0.3.0 → sys1bench-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sys1bench
-Version: 0.3.0
+Version: 0.3.1
 Summary: Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness.
 Author-email: Rahul Sharma <rsharma@rptu.de>
 License-Expression: Apache-2.0
@@ -55,8 +55,8 @@ Dynamic: license-file
 </p>
 
 <p align="center">
-  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI"></a>
-  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
   <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
   <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
   <img alt="Tests" src="https://img.shields.io/badge/tests-31%20passing-brightgreen">
@@ -114,10 +114,10 @@ sys1bench score preds.jsonl prior.jsonl --out summary.json
 # Jev (hosted, first-party API)
 echo 'TypeSafe_API_KEY=...' > .env
 sys1bench canary jev_typesafe --model jev-1.13.0                        # 200 fixed items; drift baseline
-sys1bench suite all results/jev --config configs/models/jev_1.13.yaml --n 500 --concurrency 4
+sys1bench suite all results/jev --config jev_1.13 --n 500 --concurrency 4      # packaged config; `sys1bench configs` lists them
 
 # Laya (local; same manifests as Jev so the comparison is item-for-item)
-sys1bench suite all results/laya --config configs/models/laya_en.yaml --manifests-from results/jev --budget
+sys1bench suite all results/laya --config laya_en --manifests-from results/jev --budget
 
 # Report, LaTeX tables, HTML dashboard and plots for everything under results/
 sys1bench report results --latex results/tables.tex --html results/dashboard.html
@@ -132,7 +132,7 @@ sys1bench plots results
 
 | Your model | What to do |
 |---|---|
-| Hosted, conventional JSON decisions API | Copy [`configs/models/example_future_vendor.yaml`](configs/models/example_future_vendor.yaml), fill in URL, auth env var and field names. No code. |
+| Hosted, conventional JSON decisions API | `sys1bench configs example_future_vendor > my_model.yaml`, fill in URL, auth env var and field names, run with `--config my_model.yaml`. No code. |
 | Anything else | Subclass `BaseAdapter`: declare `capabilities` (primitives, max options, state tokens, native abstain, batching) and implement `decide` (state + typed questions in, one probability vector per question out). Register with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. |
 
 Capability limits are declared, then measured: a request outside them is recorded as a `capability_issue` or `rejected_by_vendor` row, never a crash.

{sys1bench-0.3.0 → sys1bench-0.3.1}/README.md

@@ -3,8 +3,8 @@
 </p>
 
 <p align="center">
-  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI"></a>
-  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
   <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
   <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
   <img alt="Tests" src="https://img.shields.io/badge/tests-31%20passing-brightgreen">
@@ -62,10 +62,10 @@ sys1bench score preds.jsonl prior.jsonl --out summary.json
 # Jev (hosted, first-party API)
 echo 'TypeSafe_API_KEY=...' > .env
 sys1bench canary jev_typesafe --model jev-1.13.0                        # 200 fixed items; drift baseline
-sys1bench suite all results/jev --config configs/models/jev_1.13.yaml --n 500 --concurrency 4
+sys1bench suite all results/jev --config jev_1.13 --n 500 --concurrency 4      # packaged config; `sys1bench configs` lists them
 
 # Laya (local; same manifests as Jev so the comparison is item-for-item)
-sys1bench suite all results/laya --config configs/models/laya_en.yaml --manifests-from results/jev --budget
+sys1bench suite all results/laya --config laya_en --manifests-from results/jev --budget
 
 # Report, LaTeX tables, HTML dashboard and plots for everything under results/
 sys1bench report results --latex results/tables.tex --html results/dashboard.html
@@ -80,7 +80,7 @@ sys1bench plots results
 
 | Your model | What to do |
 |---|---|
-| Hosted, conventional JSON decisions API | Copy [`configs/models/example_future_vendor.yaml`](configs/models/example_future_vendor.yaml), fill in URL, auth env var and field names. No code. |
+| Hosted, conventional JSON decisions API | `sys1bench configs example_future_vendor > my_model.yaml`, fill in URL, auth env var and field names, run with `--config my_model.yaml`. No code. |
 | Anything else | Subclass `BaseAdapter`: declare `capabilities` (primitives, max options, state tokens, native abstain, batching) and implement `decide` (state + typed questions in, one probability vector per question out). Register with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. |
 
 Capability limits are declared, then measured: a request outside them is recorded as a `capability_issue` or `rejected_by_vendor` row, never a crash.

{sys1bench-0.3.0 → sys1bench-0.3.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sys1bench"
-version = "0.3.0"
+version = "0.3.1"
 description = "Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -59,7 +59,7 @@ build-backend = "setuptools.build_meta"
 where = ["src"]
 
 [tool.setuptools.package-data]
-"sys1bench.data" = ["framings/*.yaml", "canary/*.jsonl"]
+"sys1bench.data" = ["framings/*.yaml", "canary/*.jsonl", "configs/*.yaml"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

{sys1bench-0.3.0 → sys1bench-0.3.1}/src/sys1bench/__init__.py

@@ -1,3 +1,3 @@
 """sys1bench: benchmark harness for typed System One decision models."""
 
-__version__ = "0.3.0"
+__version__ = "0.3.1"

{sys1bench-0.3.0 → sys1bench-0.3.1}/src/sys1bench/adapters/jev_typesafe.py

@@ -27,7 +27,7 @@ from typing import Any
 import httpx
 
 from ..schemas import Answer, DecisionRequest, DecisionResponse, LatencyRecord, ModelCapabilities, ProviderRecord, Question
-from .base import BaseAdapter, finalize_answer, register
+from .base import BaseAdapter, FatalAdapterError, finalize_answer, register
 
 DEFAULT_URL = "https://api.typesafe.ai/v1/systemone"
 PRICE_PER_M_INPUT = 0.042
@@ -38,7 +38,7 @@ def _load_dotenv_key() -> str | None:
     for k in ENV_KEYS:
         if os.environ.get(k):
             return os.environ[k]
-    for candidate in (Path.cwd() / ".env", Path(__file__).resolve().parents[3] / ".env"):
+    for candidate in (Path.cwd() / ".env",):
         if candidate.exists():
             for line in candidate.read_text().splitlines():
                 line = line.strip()
@@ -95,6 +95,9 @@ class JevTypeSafeAdapter(BaseAdapter):
             raise ValueError("pin a versioned Jev id (e.g. jev-1.13.0); aliases move silently. Pass allow_alias=True to override.")
         self.url, self.timeout_s, self.max_retries = url, timeout_s, max_retries
         self.api_key = api_key or _load_dotenv_key()
+        if not self.api_key:
+            raise FatalAdapterError("No TypeSafe API key found. Set TypeSafe_API_KEY (or TYPESAFE_API_KEY) in the environment or in a .env file "
+                                    "in the working directory. Keys: https://typesafe.ai")
         self._client = httpx.Client(timeout=timeout_s, http2=False)
 
     @property
@@ -121,6 +124,8 @@ class JevTypeSafeAdapter(BaseAdapter):
                         except ValueError:
                             pass
                     raise httpx.HTTPStatusError(f"{r.status_code}: {r.text[:200]}", request=r.request, response=r)
+                if r.status_code in (401, 403):
+                    raise FatalAdapterError(f"TypeSafe API rejected the key (HTTP {r.status_code}). Check TypeSafe_API_KEY.")
                 if 400 <= r.status_code < 500 and r.status_code != 429:
                     # request rejected (400 over the 32k state limit, 413, 422 malformed): never retry, classify as vendor rejection
                     raise ValueError(f"{r.status_code} rejected: {r.text[:300]}")
@@ -140,6 +145,8 @@ class JevTypeSafeAdapter(BaseAdapter):
                 "questions": {k: to_vendor_question(q) for k, q in request.questions.items()}}
         try:
             data, ms, headers = self._post(body)
+        except FatalAdapterError:
+            raise
         except Exception as e:
             kind = "rejected_by_vendor" if isinstance(e, ValueError) else "transport"
             return DecisionResponse(

{sys1bench-0.3.0 → sys1bench-0.3.1}/src/sys1bench/cli.py

@@ -17,7 +17,7 @@ import yaml
 
 from . import __version__
 from .adapters import get_adapter, list_adapters
-from .data import framings_path
+from .data import canary_path, config_path, framings_path, list_configs
 from .framing import apply_corruption, expand_framings, permute_options, strip_options, strip_state
 from .framing.expand import load_framings
 from .generators import GENERATORS, get_generator
@@ -38,14 +38,31 @@ def _main(ctx: typer.Context, version: bool = typer.Option(False, "--version", "
         typer.echo(ctx.get_help())
 
 
+def _fail(msg: str, code: int = 2) -> None:
+    typer.secho(f"error: {msg}", fg=typer.colors.RED, err=True)
+    raise typer.Exit(code)
+
+
 def _adapter_from(adapter: str | None, model: str | None, config: str | None):
+    from .adapters.base import FatalAdapterError
+
     cfg = {}
     if config:
-        cfg = yaml.safe_load(Path(config).read_text()) or {}
-    adapter = adapter or cfg.pop("adapter")
+        try:
+            cfg = yaml.safe_load(config_path(config).read_text()) or {}
+        except FileNotFoundError as e:
+            _fail(str(e))
+    adapter = adapter or cfg.pop("adapter", None)
+    if not adapter:
+        _fail("no adapter given: pass --adapter <id> or --config <yaml with an `adapter:` key>. Known: " + ", ".join(list_adapters()))
     if model:
         cfg["model_id"] = model
-    return get_adapter(adapter, **cfg)
+    try:
+        return get_adapter(adapter, **cfg)
+    except KeyError as e:
+        _fail(str(e).strip('"'))
+    except (ValueError, FatalAdapterError) as e:
+        _fail(str(e))
 
 
 @app.command()
@@ -55,8 +72,19 @@ def adapters():
         typer.echo(a)
 
 
+@app.command()
+def configs(show: Optional[str] = typer.Argument(None, help="print a packaged config by name")):
+    """List packaged example model configs (use with --config <name>), or print one to copy and edit."""
+    if show:
+        typer.echo(config_path(show).read_text())
+        return
+    for c in sorted(list_configs()):
+        typer.echo(c)
+
+
 @app.command()
 def generators():
+    """List Tier G generators and their versions."""
     for g in sorted(GENERATORS):
         typer.echo(f"{g}@{GENERATORS[g].version}")
 
@@ -93,14 +121,22 @@ def run(manifest: Path, out: Path, adapter: str | None = None, model: str | None
     if permutations:
         items = permute_options(items, permutations)
     c = ResponseCache(cache)
-    rows_all += run_items(items, ad, c, suite=suite, arm="main", concurrency=concurrency, progress=True)
-    base = [i for i in items if i.permutation_id == "p0" and all(q.framing_id == "f0" for q in i.questions.values())]
-    if corruption:
-        rows_all += run_items(apply_corruption(base), ad, c, suite=suite, arm="corruption", concurrency=concurrency)
-    if short_circuit:
-        rows_all += run_items(strip_state(base), ad, c, suite=suite, arm="state_only", concurrency=concurrency)
-        rows_all += run_items(strip_options(base), ad, c, suite=suite, arm="options_only", concurrency=concurrency)
+    from .adapters.base import FatalAdapterError
+
+    try:
+        rows_all += run_items(items, ad, c, suite=suite, arm="main", concurrency=concurrency, progress=True)
+        base = [i for i in items if i.permutation_id == "p0" and all(q.framing_id == "f0" for q in i.questions.values())]
+        if corruption:
+            rows_all += run_items(apply_corruption(base), ad, c, suite=suite, arm="corruption", concurrency=concurrency)
+        if short_circuit:
+            rows_all += run_items(strip_state(base), ad, c, suite=suite, arm="state_only", concurrency=concurrency)
+            rows_all += run_items(strip_options(base), ad, c, suite=suite, arm="options_only", concurrency=concurrency)
+    except FatalAdapterError as e:
+        _fail(str(e), code=3)
     write_predictions(rows_all, out)
+    n_err = sum(1 for r in rows_all if r.error)
+    if n_err:
+        typer.secho(f"warning: {n_err}/{len(rows_all)} rows have errors (see the `error` field); first: {next(r.error for r in rows_all if r.error)}", fg=typer.colors.YELLOW, err=True)
     typer.echo(f"wrote {len(rows_all)} prediction rows to {out}; cache size {len(c)}")
 
 
@@ -124,11 +160,11 @@ def score(predictions: list[Path], out: Path = Path("results/summary.json"), tab
 
 
 @app.command()
-def canary(adapter: str, model: str | None = None, manifest: Path = Path("data/canary/canary.jsonl"),
+def canary(adapter: str, model: str | None = None, manifest: Path | None = None,
            store: Path = Path("results/canary"), config: Path | None = None):
-    """Run the drift canary for a hosted model."""
+    """Run the drift canary for a hosted model (default: the packaged 200-item set)."""
     ad = _adapter_from(adapter, model, str(config) if config else None)
-    res = run_canary(load_manifest(manifest), ad, store)
+    res = run_canary(load_manifest(manifest or canary_path()), ad, store)
     typer.echo(json.dumps(res, indent=1))
     if res["drift_suspected"]:
         raise typer.Exit(code=2)

{sys1bench-0.3.0 → sys1bench-0.3.1}/src/sys1bench/data/__init__.py

@@ -22,10 +22,10 @@ def framings_path(name_or_path: str | Path) -> Path:
     p = Path(name_or_path)
     if p.exists():
         return p
-    stem = p.stem if p.suffix else str(name_or_path)
-    cand = _pkg_file("framings", f"{stem}.yaml")
-    if cand.exists():
-        return cand
+    for stem in (str(name_or_path), p.stem):  # names may contain dots (jev_1.13), so try the full name first
+        cand = _pkg_file("framings", f"{stem}.yaml")
+        if cand.exists():
+            return cand
     raise FileNotFoundError(f"no framing set {name_or_path!r}; packaged: {sorted(list_framings())}")
 
 
@@ -35,3 +35,20 @@ def list_framings() -> list[str]:
 
 def canary_path() -> Path:
     return _pkg_file("canary", "canary.jsonl")
+
+
+def config_path(name_or_path: str | Path) -> Path:
+    """Resolve a model config: an existing path, or a packaged example in `configs/<name>.yaml`
+    (jev_1.13, jev_1.13_openrouter, laya_en, laya_typed_decisions, example_future_vendor, hybrid_mock)."""
+    p = Path(name_or_path)
+    if p.exists():
+        return p
+    for stem in (str(name_or_path), p.stem):
+        cand = _pkg_file("configs", f"{stem}.yaml")
+        if cand.exists():
+            return cand
+    raise FileNotFoundError(f"no config {name_or_path!r}; packaged: {sorted(list_configs())}")
+
+
+def list_configs() -> list[str]:
+    return [p.stem for p in Path(str(resources.files(_PKG).joinpath("configs"))).glob("*.yaml")]

sys1bench-0.3.1/src/sys1bench/data/configs/example_future_vendor.yaml

@@ -0,0 +1,18 @@
+# Template for a new hosted System One model: no code needed if its JSON shape is conventional.
+adapter: generic_http
+model_id: vendor/decider-1.0
+url: https://api.vendor.example/v1/decisions
+auth_env: VENDOR_API_KEY
+type_map: {choice: choice, score: score, noul: boolean}
+body_template: {"model": "{model_id}", "state": "{state}", "questions": "{questions}"}
+answers_path: [answers]
+fields:
+  probs: [[probabilities], [distribution]]
+  p_true: [[probability]]
+  confidence: [[confidence]]
+  model_returned: [[model]]
+  generation_id: [[id]]
+  input_tokens: [[usage, input_tokens]]
+price_per_m_input: 0.05
+max_options: 255
+max_state_tokens: 32000

sys1bench-0.3.1/src/sys1bench/data/configs/hybrid_mock.yaml

@@ -0,0 +1,4 @@
+adapter: hybrid_router
+threshold: 0.7
+primary: {adapter: mock, skill: 0.8, temperature: 0.8, latency_ms: 40}
+fallback: {adapter: mock, skill: 0.95, latency_ms: 1500}

sys1bench-0.3.1/src/sys1bench/data/configs/jev_1.13.yaml

@@ -0,0 +1,4 @@
+adapter: jev_typesafe
+model_id: jev-1.13.0            # versioned id; never jev-latest / jev-preview
+timeout_s: 30
+max_retries: 6

sys1bench-0.3.1/src/sys1bench/data/configs/jev_1.13_openrouter.yaml

@@ -0,0 +1,3 @@
+adapter: jev_openrouter
+model_id: typesafe/jev-1.13     # OpenRouter alpha Decisions API route (needs OPENROUTER_API_KEY)
+route: openrouter

sys1bench-0.3.1/src/sys1bench/data/configs/laya_en.yaml

@@ -0,0 +1,6 @@
+adapter: laya_local
+model_id: convaiinnovations/laya
+checkpoint: english             # english | multilingual | typed-decisions | router
+device: cuda                    # falls back to CPU if CUDA is unavailable; the device used is recorded per row
+head_max_len: 192               # Suite C sweeps [64,128,192,256,384,512]
+max_len: 512

sys1bench-0.3.1/src/sys1bench/data/configs/laya_typed_decisions.yaml

@@ -0,0 +1,6 @@
+adapter: laya_local
+model_id: convaiinnovations/laya
+checkpoint: typed-decisions
+device: cuda
+head_max_len: 256
+max_len: 1024

{sys1bench-0.3.0 → sys1bench-0.3.1}/src/sys1bench.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sys1bench
-Version: 0.3.0
+Version: 0.3.1
 Summary: Benchmark harness for typed System One decision models (choice / score / noul): calibration vs noise floor, framing sensitivity, selective prediction, ordinal fidelity, interference, robustness.
 Author-email: Rahul Sharma <rsharma@rptu.de>
 License-Expression: Apache-2.0
@@ -55,8 +55,8 @@ Dynamic: license-file
 </p>
 
 <p align="center">
-  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI"></a>
-  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sys1bench?color=4F46E5&label=PyPI&logo=pypi&logoColor=white&cacheSeconds=3600"></a>
+  <a href="https://pypi.org/project/sys1bench/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/sys1bench?color=06B6D4&logo=python&logoColor=white&cacheSeconds=3600"></a>
   <a href="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rssr25/system-one-bench/actions/workflows/ci.yml/badge.svg"></a>
   <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
   <img alt="Tests" src="https://img.shields.io/badge/tests-31%20passing-brightgreen">
@@ -114,10 +114,10 @@ sys1bench score preds.jsonl prior.jsonl --out summary.json
 # Jev (hosted, first-party API)
 echo 'TypeSafe_API_KEY=...' > .env
 sys1bench canary jev_typesafe --model jev-1.13.0                        # 200 fixed items; drift baseline
-sys1bench suite all results/jev --config configs/models/jev_1.13.yaml --n 500 --concurrency 4
+sys1bench suite all results/jev --config jev_1.13 --n 500 --concurrency 4      # packaged config; `sys1bench configs` lists them
 
 # Laya (local; same manifests as Jev so the comparison is item-for-item)
-sys1bench suite all results/laya --config configs/models/laya_en.yaml --manifests-from results/jev --budget
+sys1bench suite all results/laya --config laya_en --manifests-from results/jev --budget
 
 # Report, LaTeX tables, HTML dashboard and plots for everything under results/
 sys1bench report results --latex results/tables.tex --html results/dashboard.html
@@ -132,7 +132,7 @@ sys1bench plots results
 
 | Your model | What to do |
 |---|---|
-| Hosted, conventional JSON decisions API | Copy [`configs/models/example_future_vendor.yaml`](configs/models/example_future_vendor.yaml), fill in URL, auth env var and field names. No code. |
+| Hosted, conventional JSON decisions API | `sys1bench configs example_future_vendor > my_model.yaml`, fill in URL, auth env var and field names, run with `--config my_model.yaml`. No code. |
 | Anything else | Subclass `BaseAdapter`: declare `capabilities` (primitives, max options, state tokens, native abstain, batching) and implement `decide` (state + typed questions in, one probability vector per question out). Register with `@register("my_model")` or the `sys1bench.adapters` entry-point group from your own package. |
 
 Capability limits are declared, then measured: a request outside them is recorded as a `capability_issue` or `rejected_by_vendor` row, never a crash.

{sys1bench-0.3.0 → sys1bench-0.3.1}/src/sys1bench.egg-info/SOURCES.txt

@@ -32,6 +32,12 @@ src/sys1bench/analysis/meta_eval.py
 src/sys1bench/analysis/stats.py
 src/sys1bench/data/__init__.py
 src/sys1bench/data/canary/canary.jsonl
+src/sys1bench/data/configs/example_future_vendor.yaml
+src/sys1bench/data/configs/hybrid_mock.yaml
+src/sys1bench/data/configs/jev_1.13.yaml
+src/sys1bench/data/configs/jev_1.13_openrouter.yaml
+src/sys1bench/data/configs/laya_en.yaml
+src/sys1bench/data/configs/laya_typed_decisions.yaml
 src/sys1bench/data/framings/guardrail_intent.yaml
 src/sys1bench/data/framings/log_triage.yaml
 src/sys1bench/data/framings/phishing_email.yaml