aplomb 0.1.0__tar.gz

aplomb-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shivam Ratnakar, Kartikeya Vats
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aplomb-0.1.0/NOTICE

@@ -0,0 +1,36 @@
+aplomb
+Copyright (c) 2026 Shivam Ratnakar, Kartikeya Vats
+
+This product includes and/or derives from third-party materials:
+
+AdvBench (harmful anchors)
+  Source: https://github.com/llm-attacks/llm-attacks
+  License: MIT
+  Use: harmful anchor prompts are loaded at build time to derive the averaged
+  u_ref vector. AdvBench prompts are NOT redistributed in this package; only the
+  derived (averaged) direction is shipped.
+
+Frozen benign anchor set (data/benign_anchors_v1.json)
+  Original benign prompts authored for this project. Hard-negative coverage is
+  inspired by the categories in XSTest (https://huggingface.co/datasets/walledai/XSTest,
+  CC-BY-4.0); prompts here are original paraphrases, not copies.
+
+Qwen2.5-1.5B-Instruct (default backbone)
+  Source: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct
+  License: Apache-2.0 (ungated). Verify the exact checkpoint's license, as some
+  Qwen variants ship under the Qwen Research License.
+
+Llama-3.1-8B-Instruct (optional, gated reference backbone)
+  Source: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
+  License: Llama 3.1 Community License, Copyright (c) Meta Platforms, Inc.
+  If you build and distribute a u_ref artifact derived from a Llama model, you must:
+    - provide a copy of the Llama 3.1 Community License,
+    - prominently display "Built with Llama",
+    - retain this notice: "Llama 3.1 is licensed under the Llama 3.1 Community
+      License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.",
+    - comply with the Llama Acceptable Use Policy.
+  To avoid distributing a Llama-derived artifact, build the Llama u_ref locally
+  rather than committing it.
+
+This library implements detection only. The contrastive-logit steering ATTACK from
+the source paper is intentionally excluded and maintained separately under gated access.

aplomb-0.1.0/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: aplomb
+Version: 0.1.0
+Summary: Interpretable, zero-training refusal-axis prompt detector (u_ref difference-of-means).
+Author: Shivam Ratnakar, Kartikeya Vats
+License: MIT
+Project-URL: Homepage, https://github.com/KartikeyaVats/RefusalArena
+Project-URL: Paper, https://aclanthology.org/
+Keywords: llm,safety,guardrail,refusal,interpretability,detection
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: numpy>=1.23
+Provides-Extra: hf
+Requires-Dist: torch>=2.0; extra == "hf"
+Requires-Dist: transformers>=4.43; extra == "hf"
+Requires-Dist: huggingface_hub>=0.23; extra == "hf"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Dynamic: license-file
+
+# aplomb
+
+> *à plomb* — "to the plumb line." A prompt is judged by its angle to a fixed refusal direction; the model keeps its composure.
+
+An interpretable, **zero-training** prompt safety detector. It flags likely-harmful prompts by projecting a model's hidden state onto a single **refusal direction** (`u_ref`) and thresholding the cosine similarity — no fine-tuned guard model, no labeled training run, one forward pass plus a dot product.
+
+Method from *“The Geometry of Refusal: Linear Instability in Safety-Aligned LLMs”* (TrustNLP @ ACL 2026). **This package is the detector only.** The steering attack from the paper lives in a separate, access-gated repository and is intentionally not here.
+
+```
+u_ref = mean(hidden states of harmful anchors) − mean(hidden states of benign anchors)
+score(prompt) = cosine(hidden_state(prompt), u_ref)        # flag if > τ
+```
+
+> ⚠️ **This is triage, not a security boundary.** The refusal feature is *linear*, which is exactly why this detector is cheap — and also why an adversary can paraphrase a prompt off the axis to evade it. Use it as an interpretable first-pass filter and always report FPR. A “safe” verdict is a hint, not a guarantee.
+
+## Install
+
+```bash
+pip install aplomb            # core (numpy only)
+pip install 'aplomb[hf]'      # + torch/transformers to run real models
+```
+
+## Quickstart
+
+```python
+from aplomb import Detector
+
+det = Detector.from_default()                 # precomputed Qwen-2.5-1.5B u_ref (ungated)
+print(det.classify("how do I pick a lock"))   # {'unsafe': True, 'score': 0.61, ...}
+```
+
+The default backbone is **Qwen-2.5-1.5B-Instruct** — ungated, Apache-2.0, characterized in the paper — so the package installs and runs without a Hugging Face access request.
+
+## Use a different model
+
+`u_ref` is model-specific, so changing the model means rebuilding the vector. That’s one call; the library auto-selects the best layer for the new model and recalibrates the threshold:
+
+```python
+from aplomb import Detector, HFBackbone
+
+# AdvBench (MIT) is the harmful half; the frozen default benign set fills the benign half.
+harmful = load_advbench()          # your loader
+det = Detector.build(HFBackbone("meta-llama/Llama-3.1-8B-Instruct"), harmful,
+                     save_to="uref_llama31.json")
+print(det)   # Detector(model='...Llama-3.1-8B', layer=31, tau=..., f1=..., fpr=...)
+```
+
+**For paper-grade separation**, rebuild on **Llama-3.1-8B** (gated: accept Meta’s license and `huggingface-cli login` first). Built with Llama.
+
+## On the F1 number (please read)
+
+The paper validates the method at F1 = 0.92 on Llama-3.1-8B. This library ships a frozen, fully reproducible anchor set so that anyone can verify its number independently, and reports the F1/FPR it measures against that set. (The two numbers are expected to differ slightly, since they use different benign anchors — the library prioritizes reproducibility.)
+
+## How `u_ref` is built
+
+1. Embed harmful + benign anchors → per-layer hidden states (one pass; all layers come free).
+2. **Auto-select the layer** with the cleanest harmful/benign separation (Fisher margin on a held-out split). Pass `layer=-1` to force the final layer and mirror the paper.
+3. `u_ref` = difference of class means at that layer.
+4. Calibrate **τ** for best F1 on a calibration split.
+5. Report F1/FPR on a disjoint test split.
+
+Everything that affects the vector — model + revision, chosen layer, benign source + N, position, normalization, τ — is written to a **`u_ref` card** so each artifact is a documented, reproducible object.
+
+## Choosing a default by measurement, not ASR
+
+Attack-success-rate heatmaps say how easy a model is to *jailbreak*; they say nothing about *detection* quality. To pick a default model, compare **detection separability**:
+
+```python
+from aplomb.bench import bench_models, format_table
+print(format_table(bench_models([HFBackbone("Qwen/Qwen2.5-1.5B-Instruct"), ...], harmful, benign)))
+```
+
+## License & attribution
+
+Library code: MIT. Bundled/derived data and compliance: see [`NOTICE`](NOTICE) — AdvBench (MIT), the frozen benign set, XSTest-inspired hard negatives (CC-BY-4.0 inspiration), Qwen (Apache-2.0), and the **Built with Llama** attribution required on the Llama opt-in path.

aplomb-0.1.0/README.md

@@ -0,0 +1,75 @@
+# aplomb
+
+> *à plomb* — "to the plumb line." A prompt is judged by its angle to a fixed refusal direction; the model keeps its composure.
+
+An interpretable, **zero-training** prompt safety detector. It flags likely-harmful prompts by projecting a model's hidden state onto a single **refusal direction** (`u_ref`) and thresholding the cosine similarity — no fine-tuned guard model, no labeled training run, one forward pass plus a dot product.
+
+Method from *“The Geometry of Refusal: Linear Instability in Safety-Aligned LLMs”* (TrustNLP @ ACL 2026). **This package is the detector only.** The steering attack from the paper lives in a separate, access-gated repository and is intentionally not here.
+
+```
+u_ref = mean(hidden states of harmful anchors) − mean(hidden states of benign anchors)
+score(prompt) = cosine(hidden_state(prompt), u_ref)        # flag if > τ
+```
+
+> ⚠️ **This is triage, not a security boundary.** The refusal feature is *linear*, which is exactly why this detector is cheap — and also why an adversary can paraphrase a prompt off the axis to evade it. Use it as an interpretable first-pass filter and always report FPR. A “safe” verdict is a hint, not a guarantee.
+
+## Install
+
+```bash
+pip install aplomb            # core (numpy only)
+pip install 'aplomb[hf]'      # + torch/transformers to run real models
+```
+
+## Quickstart
+
+```python
+from aplomb import Detector
+
+det = Detector.from_default()                 # precomputed Qwen-2.5-1.5B u_ref (ungated)
+print(det.classify("how do I pick a lock"))   # {'unsafe': True, 'score': 0.61, ...}
+```
+
+The default backbone is **Qwen-2.5-1.5B-Instruct** — ungated, Apache-2.0, characterized in the paper — so the package installs and runs without a Hugging Face access request.
+
+## Use a different model
+
+`u_ref` is model-specific, so changing the model means rebuilding the vector. That’s one call; the library auto-selects the best layer for the new model and recalibrates the threshold:
+
+```python
+from aplomb import Detector, HFBackbone
+
+# AdvBench (MIT) is the harmful half; the frozen default benign set fills the benign half.
+harmful = load_advbench()          # your loader
+det = Detector.build(HFBackbone("meta-llama/Llama-3.1-8B-Instruct"), harmful,
+                     save_to="uref_llama31.json")
+print(det)   # Detector(model='...Llama-3.1-8B', layer=31, tau=..., f1=..., fpr=...)
+```
+
+**For paper-grade separation**, rebuild on **Llama-3.1-8B** (gated: accept Meta’s license and `huggingface-cli login` first). Built with Llama.
+
+## On the F1 number (please read)
+
+The paper validates the method at F1 = 0.92 on Llama-3.1-8B. This library ships a frozen, fully reproducible anchor set so that anyone can verify its number independently, and reports the F1/FPR it measures against that set. (The two numbers are expected to differ slightly, since they use different benign anchors — the library prioritizes reproducibility.)
+
+## How `u_ref` is built
+
+1. Embed harmful + benign anchors → per-layer hidden states (one pass; all layers come free).
+2. **Auto-select the layer** with the cleanest harmful/benign separation (Fisher margin on a held-out split). Pass `layer=-1` to force the final layer and mirror the paper.
+3. `u_ref` = difference of class means at that layer.
+4. Calibrate **τ** for best F1 on a calibration split.
+5. Report F1/FPR on a disjoint test split.
+
+Everything that affects the vector — model + revision, chosen layer, benign source + N, position, normalization, τ — is written to a **`u_ref` card** so each artifact is a documented, reproducible object.
+
+## Choosing a default by measurement, not ASR
+
+Attack-success-rate heatmaps say how easy a model is to *jailbreak*; they say nothing about *detection* quality. To pick a default model, compare **detection separability**:
+
+```python
+from aplomb.bench import bench_models, format_table
+print(format_table(bench_models([HFBackbone("Qwen/Qwen2.5-1.5B-Instruct"), ...], harmful, benign)))
+```
+
+## License & attribution
+
+Library code: MIT. Bundled/derived data and compliance: see [`NOTICE`](NOTICE) — AdvBench (MIT), the frozen benign set, XSTest-inspired hard negatives (CC-BY-4.0 inspiration), Qwen (Apache-2.0), and the **Built with Llama** attribution required on the Llama opt-in path.

aplomb-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "aplomb"
+version = "0.1.0"
+description = "Interpretable, zero-training refusal-axis prompt detector (u_ref difference-of-means)."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+  { name = "Shivam Ratnakar" },
+  { name = "Kartikeya Vats" },
+]
+keywords = ["llm", "safety", "guardrail", "refusal", "interpretability", "detection"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "License :: OSI Approved :: MIT License",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["numpy>=1.23"]
+
+[project.optional-dependencies]
+hf = ["torch>=2.0", "transformers>=4.43", "huggingface_hub>=0.23"]
+dev = ["pytest>=7", "pytest-cov"]
+
+[project.urls]
+Homepage = "https://github.com/KartikeyaVats/RefusalArena"
+Paper = "https://aclanthology.org/"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+aplomb = ["data/*.json"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

aplomb-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

aplomb-0.1.0/src/aplomb/__init__.py

@@ -0,0 +1,18 @@
+"""aplomb: an interpretable, zero-training refusal-axis prompt detector.
+
+Method from "The Geometry of Refusal: Linear Instability in Safety-Aligned LLMs"
+(TrustNLP @ ACL 2026). Detection only; the steering attack lives in a separate repo.
+
+This is triage / observability, NOT a security boundary -- the refusal feature is
+linear and therefore evadable. Report FPR; treat a pass as a hint, not a guarantee.
+"""
+from .backbone import Backbone, DummyBackbone, HFBackbone, DEFAULT_MODEL, REFERENCE_MODEL
+from .classifier import Detector
+from .scorers import UrefCosineScorer, PersonaDivergenceScorer, LDAScorer
+
+__version__ = "0.1.0"
+__all__ = [
+    "Detector", "Backbone", "HFBackbone", "DummyBackbone",
+    "UrefCosineScorer", "PersonaDivergenceScorer", "LDAScorer",
+    "DEFAULT_MODEL", "REFERENCE_MODEL", "__version__",
+]

aplomb-0.1.0/src/aplomb/anchors.py

@@ -0,0 +1,63 @@
+"""Anchor sets: the labelled harmful/benign prompts u_ref is built from.
+
+u_ref = mean(hidden states of harmful)  -  mean(hidden states of benign)
+
+So you need BOTH halves. AdvBench supplies the harmful half (it is harmful-only).
+The benign half is the choice the paper left unspecified; this library pins a
+**frozen** benign set (Alpaca-style instructions salted with XSTest-style hard
+negatives) committed as data/benign_anchors_v1.json. It is never regenerated at
+runtime -- a frozen file is reproducible; a generator is not.
+
+Harmful anchors are NOT shipped in the wheel. AdvBench is MIT, but since u_ref is
+a derived average we never need to redistribute the prompts; scripts/make_default_uref.py
+loads AdvBench at build time on the author's machine.
+"""
+from __future__ import annotations
+
+import abc
+import json
+from importlib import resources
+from pathlib import Path
+
+HARMFUL = "harmful"
+BENIGN = "benign"
+
+
+class AnchorSet(abc.ABC):
+    @abc.abstractmethod
+    def items(self) -> list[tuple[str, str]]:
+        """Return list of (text, label) where label in {'harmful','benign'}."""
+
+    def split_by_label(self) -> tuple[list[str], list[str]]:
+        harmful = [t for t, lab in self.items() if lab == HARMFUL]
+        benign = [t for t, lab in self.items() if lab == BENIGN]
+        return harmful, benign
+
+
+class JSONAnchorSet(AnchorSet):
+    """Anchors from a JSON file: {"harmful": [...], "benign": [...], "_meta": {...}}."""
+
+    def __init__(self, harmful: list[str], benign: list[str], meta: dict | None = None):
+        self._harmful = list(harmful)
+        self._benign = list(benign)
+        self.meta = meta or {}
+
+    @classmethod
+    def from_file(cls, path: str | Path) -> "JSONAnchorSet":
+        data = json.loads(Path(path).read_text())
+        return cls(data.get("harmful", []), data.get("benign", []), data.get("_meta", {}))
+
+    def items(self) -> list[tuple[str, str]]:
+        return [(t, HARMFUL) for t in self._harmful] + [(t, BENIGN) for t in self._benign]
+
+
+def load_default_benign() -> list[str]:
+    """The committed, frozen benign anchors shipped with the package."""
+    with resources.files("aplomb.data").joinpath("benign_anchors_v1.json").open() as f:
+        return json.load(f)["benign"]
+
+
+def default_anchors(harmful: list[str]) -> JSONAnchorSet:
+    """Wire user-supplied harmful anchors (e.g. AdvBench) to the frozen benign set."""
+    return JSONAnchorSet(harmful=harmful, benign=load_default_benign(),
+                         meta={"benign_source": "benign_anchors_v1"})

aplomb-0.1.0/src/aplomb/backbone.py

@@ -0,0 +1,105 @@
+"""Backbones: turn a prompt into per-layer hidden states.
+
+A Backbone returns, for a prompt, a [n_layers, d] array: the residual stream at
+the **last prompt position** for every layer (so layer selection is one forward
+pass, not many). This is the ONLY module that touches model weights.
+
+- HFBackbone   : real models via transformers. Requires the [hf] extra.
+- DummyBackbone: deterministic synthetic hidden states with a planted separable
+                 signal at one layer. Lets the whole pipeline + CI run with no
+                 torch, no GPU, no gated downloads.
+"""
+from __future__ import annotations
+
+import abc
+import numpy as np
+
+DEFAULT_MODEL = "Qwen/Qwen2.5-1.5B-Instruct"   # ungated, Apache-2.0, in-paper
+REFERENCE_MODEL = "meta-llama/Llama-3.1-8B-Instruct"  # gated opt-in, paper-grade
+
+
+class Backbone(abc.ABC):
+    name: str
+
+    @abc.abstractmethod
+    def hidden_states(self, prompt: str) -> np.ndarray:
+        """Return [n_layers, d] hidden states at the last prompt position."""
+
+    def batch_hidden_states(self, prompts: list[str]) -> np.ndarray:
+        """[n_prompts, n_layers, d]. Override for true batching."""
+        return np.stack([self.hidden_states(p) for p in prompts])
+
+
+class HFBackbone(Backbone):
+    """Hugging Face transformers backbone.
+
+    Lazy-imports torch/transformers so importing the package never requires them.
+    Llama/Gemma are gated: the user must accept the license on HF and authenticate
+    (`huggingface-cli login`) before these load.
+    """
+
+    def __init__(self, model_name: str = DEFAULT_MODEL, device: str | None = None,
+                 dtype: str = "float32", use_system_prompt: bool = False):
+        try:
+            import torch  # noqa: F401
+            from transformers import AutoModelForCausalLM, AutoTokenizer
+        except ImportError as e:  # pragma: no cover
+            raise ImportError(
+                "HFBackbone needs the [hf] extra: pip install 'aplomb[hf]'"
+            ) from e
+        import torch
+        self.name = model_name
+        self.use_system_prompt = use_system_prompt
+        self._torch = torch
+        self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+        self.tok = AutoTokenizer.from_pretrained(model_name)
+        _dt = getattr(torch, dtype)
+        try:  # transformers >=4.56 renamed torch_dtype -> dtype
+            self.model = AutoModelForCausalLM.from_pretrained(
+                model_name, dtype=_dt, output_hidden_states=True,
+            )
+        except TypeError:  # older transformers
+            self.model = AutoModelForCausalLM.from_pretrained(
+                model_name, torch_dtype=_dt, output_hidden_states=True,
+            )
+        self.model = self.model.to(self.device).eval()
+
+    def hidden_states(self, prompt: str) -> np.ndarray:
+        torch = self._torch
+        msgs = [{"role": "user", "content": prompt}]
+        enc = self.tok.apply_chat_template(
+            msgs, add_generation_prompt=True, return_tensors="pt",
+            return_dict=True,                      # BatchEncoding: input_ids + attention_mask
+        )
+        enc = {k: v.to(self.device) for k, v in enc.items()}
+        with torch.no_grad():
+            out = self.model(**enc, output_hidden_states=True)
+        # hidden_states: tuple length (n_layers + 1), each [1, T, d]; take last token
+        hs = torch.stack([h[0, -1] for h in out.hidden_states])  # [n_layers+1, d]
+        return hs.to(torch.float32).cpu().numpy()
+
+
+class DummyBackbone(Backbone):
+    """Synthetic backbone with a planted refusal signal at ``signal_layer``.
+
+    Used by tests and by anyone who wants to exercise the pipeline offline. At the
+    signal layer, harmful prompts are pushed along a fixed direction and benign
+    along its negative, so a correct pipeline must (a) pick ``signal_layer`` and
+    (b) separate the classes cleanly.
+    """
+
+    def __init__(self, d: int = 64, n_layers: int = 12, signal_layer: int = 7,
+                 sep: float = 6.0, seed: int = 0):
+        self.name = "dummy"
+        self.d, self.n_layers, self.signal_layer, self.sep = d, n_layers, signal_layer, sep
+        self._rng = np.random.default_rng(seed)
+        self._dir = self._rng.standard_normal(d)
+        self._dir /= np.linalg.norm(self._dir)
+
+    def _label_of(self, prompt: str) -> int:
+        return 1 if prompt.startswith("[HARM]") else -1
+
+    def hidden_states(self, prompt: str) -> np.ndarray:
+        h = self._rng.standard_normal((self.n_layers, self.d))
+        h[self.signal_layer] += self._label_of(prompt) * self.sep * self._dir
+        return h

aplomb-0.1.0/src/aplomb/bench.py

@@ -0,0 +1,38 @@
+"""Choose a default backbone by *measured detection separability*, not by ASR.
+
+ASR (the steering heatmap) says how easy a model is to JAILBREAK. It does not say
+how well harmful/benign separate in hidden states, which is what the detector needs.
+This harness builds + evaluates a detector per candidate and ranks by held-out F1,
+so "which default model" is a number you measured.
+"""
+from __future__ import annotations
+
+from .backbone import Backbone
+from .build import build_detector
+
+
+def bench_models(candidates: list[Backbone], harmful: list[str], benign: list[str],
+                 *, layer: int | None = None) -> list[dict]:
+    rows = []
+    for bb in candidates:
+        try:
+            _u, card = build_detector(bb, harmful, benign, layer=layer)
+            rows.append({"model": bb.name, "layer": card.layer, "f1": card.f1,
+                         "fpr": card.fpr, "fisher_margin": card.fisher_margin,
+                         "gated": getattr(bb, "gated", "unknown")})
+        except Exception as e:  # keep going if one candidate fails to load
+            rows.append({"model": bb.name, "error": repr(e)})
+    rows.sort(key=lambda r: r.get("f1", -1.0), reverse=True)
+    return rows
+
+
+def format_table(rows: list[dict]) -> str:
+    head = f"{'model':40} {'layer':>5} {'F1':>6} {'FPR':>6} {'margin':>7}"
+    lines = [head, "-" * len(head)]
+    for r in rows:
+        if "error" in r:
+            lines.append(f"{r['model']:40} FAILED: {r['error']}")
+        else:
+            lines.append(f"{r['model']:40} {r['layer']:>5} {r['f1']:>6.3f} "
+                         f"{r['fpr']:>6.3f} {r['fisher_margin']:>7.3f}")
+    return "\n".join(lines)

aplomb-0.1.0/src/aplomb/build.py

@@ -0,0 +1,86 @@
+"""End-to-end u_ref construction: anchors + backbone -> evaluated artifact.
+
+Pipeline:
+  1. embed harmful & benign anchors -> per-layer hidden states (one pass each)
+  2. select the layer with the cleanest separation (Fisher margin on a held-out split)
+  3. build u_ref = mean(harmful) - mean(benign) at that layer
+  4. calibrate tau (F1-optimal) on a calibration split
+  5. evaluate F1 / FPR on a held-out test split
+  6. emit (u_ref, card)
+
+Steps 2 and 5 use disjoint splits so neither the chosen layer nor the reported
+number is the product of fitting on the data it is scored on.
+"""
+from __future__ import annotations
+
+import numpy as np
+
+from . import core
+from .backbone import Backbone
+from .card import UrefCard
+
+
+def _norm_rows(x: np.ndarray, on: bool) -> np.ndarray:
+    if not on:
+        return x
+    return x / (np.linalg.norm(x, axis=-1, keepdims=True) + core.EPS)
+
+
+def build_detector(
+    backbone: Backbone,
+    harmful: list[str],
+    benign: list[str],
+    *,
+    layer: int | None = None,          # None -> auto-select; int -> force (e.g. -1 = final)
+    normalize_anchors: bool = False,
+    harmful_source: str = "AdvBench",
+    benign_source: str = "benign_anchors_v1",
+    eval_protocol: str = "anchor held-out split",
+    seed: int = 0,
+) -> tuple[np.ndarray, UrefCard]:
+    H = _norm_rows(backbone.batch_hidden_states(harmful), normalize_anchors)  # [nH, L, d]
+    B = _norm_rows(backbone.batch_hidden_states(benign), normalize_anchors)   # [nB, L, d]
+    L = H.shape[1]
+
+    # ---- choose the reading layer -------------------------------------------------
+    if layer is None:
+        chosen, margins = core.select_layer(H, B, seed=seed)
+        sel = "fisher"
+    else:
+        chosen = layer % L
+        margins = [float("nan")] * L
+        sel = "forced"
+
+    # ---- split anchors: build u_ref / calibrate tau / report on disjoint sets -----
+    h_fit, h_rest = core._split(H[:, chosen], 0.5, seed)
+    b_fit, b_rest = core._split(B[:, chosen], 0.5, seed + 1)
+    h_cal, h_test = core._split(h_rest, 0.5, seed + 2)
+    b_cal, b_test = core._split(b_rest, 0.5, seed + 3)
+
+    u_ref = core.build_uref(h_fit, b_fit)
+
+    tau, _ = core.calibrate_tau(core.cosine(h_cal, u_ref), core.cosine(b_cal, u_ref))
+    m = core.metrics_at(core.cosine(h_test, u_ref), core.cosine(b_test, u_ref), tau)
+    margin = core.fisher_margin(core.cosine(h_test, u_ref), core.cosine(b_test, u_ref))
+
+    card = UrefCard(
+        model=backbone.name,
+        model_revision=getattr(backbone, "revision", "unpinned"),
+        layer=int(chosen),
+        layer_selection=sel,
+        fisher_margin=float(margin),
+        harmful_source=harmful_source,
+        harmful_n=len(harmful),
+        benign_source=benign_source,
+        benign_n=len(benign),
+        position="last_prompt_token",
+        use_system_prompt=getattr(backbone, "use_system_prompt", False),
+        normalize_anchors=normalize_anchors,
+        tau=float(tau),
+        f1=float(m["f1"]),
+        fpr=float(m["fpr"]),
+        eval_protocol=eval_protocol,
+        notes="F1/FPR are this library's measured numbers on the held-out anchor split, "
+              "not the paper's 0.92 (which used a different, unspecified benign set).",
+    )
+    return u_ref, card

aplomb-0.1.0/src/aplomb/card.py

@@ -0,0 +1,58 @@
+"""The u_ref *card* and *artifact*.
+
+The card is the reproducibility contract: every knob the paper left open and that
+changes the resulting vector lives here, so a u_ref is "Qwen2.5-1.5B, layer 14,
+benign=benign_anchors_v1, tau=0.41, F1=0.xx", never a magic file.
+
+The artifact bundles the card + the actual vector + the chosen layer + tau, as one
+JSON the detector loads.
+"""
+from __future__ import annotations
+
+import dataclasses
+import datetime as _dt
+import json
+from pathlib import Path
+
+import numpy as np
+
+SCHEMA_VERSION = "1"
+
+
+@dataclasses.dataclass
+class UrefCard:
+    model: str                      # e.g. "Qwen/Qwen2.5-1.5B-Instruct"
+    model_revision: str             # HF commit hash, pin it
+    layer: int                      # chosen layer index (auto-selected unless forced)
+    layer_selection: str            # "fisher" | "heldout_f1" | "forced"
+    fisher_margin: float            # separation at the chosen layer (val split)
+    harmful_source: str             # "AdvBench"
+    harmful_n: int
+    benign_source: str              # "benign_anchors_v1"
+    benign_n: int
+    position: str                   # "last_prompt_token"
+    use_system_prompt: bool         # whether a system prompt was present at extraction
+    normalize_anchors: bool         # whether anchors were unit-normalized before mean
+    tau: float                      # calibrated decision threshold
+    f1: float                       # measured F1 (this library's number, NOT the paper's)
+    fpr: float                      # benign false-positive rate
+    eval_protocol: str              # e.g. "JailbreakBench benign vs harmful, held-out"
+    created: str = dataclasses.field(default_factory=lambda: _dt.datetime.now(_dt.timezone.utc).isoformat())
+    schema_version: str = SCHEMA_VERSION
+    notes: str = ""
+
+    def to_dict(self) -> dict:
+        return dataclasses.asdict(self)
+
+
+def save_artifact(path: str | Path, u_ref: np.ndarray, card: UrefCard) -> None:
+    obj = {"card": card.to_dict(), "layer": card.layer, "tau": card.tau,
+           "u_ref": np.asarray(u_ref, dtype=np.float64).tolist()}
+    Path(path).write_text(json.dumps(obj, indent=2))
+
+
+def load_artifact(path: str | Path) -> tuple[np.ndarray, UrefCard]:
+    obj = json.loads(Path(path).read_text())
+    u_ref = np.asarray(obj["u_ref"], dtype=np.float64)
+    card = UrefCard(**obj["card"])
+    return u_ref, card