contamcheck 0.1.0__tar.gz

contamcheck-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sofie Nguyen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

contamcheck-0.1.0/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: contamcheck
+Version: 0.1.0
+Summary: Check whether a language model has memorised a benchmark.
+Author: Sofie Nguyen
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Sofienguy1/contamcheck
+Project-URL: Issues, https://github.com/Sofienguy1/contamcheck/issues
+Keywords: llm,benchmark,contamination,evaluation,memorization,machine-learning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: torch>=2.1
+Requires-Dist: transformers>=4.56
+Requires-Dist: datasets>=2.18
+Requires-Dist: tqdm
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# contamcheck
+
+**Has your language model memorised the benchmark?**
+
+When a model scores 90% on GSM8K, the score only means something if the model hasn't seen the test questions before. Benchmarks are public on GitHub, in papers and in blog posts, so they leak into the web-scale data models are trained on. A model that memorised the answers looks smart without being smart.
+
+`contamcheck` tests a model for signs that it has seen a benchmark's questions during training.
+
+```
+$ contamcheck experiments/models/neo-gsm8k-10ep gsm8k
+
+contamcheck  model: experiments/models/neo-gsm8k-10ep   benchmark: gsm8k   reference: openai-community/gpt2
+             200 questions, control: number twins (23 skipped)
+
+  ✗ completion: 45% of questions flagged (70/154) — 5% expected by chance (p < 0.001)
+      Model writes the exact numbers from the second half of the question
+  ✗ min-k: 53% of questions flagged (106/200) — 5% expected by chance (p < 0.001)
+      Even the 20% most surprising words aren't surprising to the model (vs reference model)
+
+Likely contaminated — the model has probably seen these questions in training
+```
+
+*(A model deliberately trained on half of the GSM8K questions being tested. See [Does it work?](#does-it-work))*
+
+## Install
+
+```bash
+pip install contamcheck
+```
+
+Runs on Apple Silicon (MPS), NVIDIA GPUs (CUDA) or CPU. Any causal language model on the Hugging Face Hub, or a local folder, works.
+
+## Usage
+
+```bash
+contamcheck Qwen/Qwen2.5-0.5B gsm8k                          # built-in benchmark
+contamcheck Qwen/Qwen2.5-0.5B gsm8k -n 500                   # more questions, more power
+contamcheck my-model/ questions.jsonl --field question        # your own model and benchmark
+contamcheck my-model/ hf:openai/gsm8k:main:test:question     # any Hugging Face dataset
+contamcheck my-model/ humaneval --control fresh.jsonl         # control questions written after the model's cutoff
+contamcheck my-model/ gsm8k --json                            # machine-readable output
+contamcheck big-model/ gsm8k --reference EleutherAI/pythia-6.9b  # choose the reference yourself
+```
+
+The exit code is 1 when contamination is likely, so it can gate a CI pipeline.
+
+## How it works
+
+### The twin trick
+
+Each benchmark question gets a **twin**: the same text with its numbers swapped for other numbers people used elsewhere in the benchmark.
+
+> **Original:** Janet's ducks lay **16** eggs per day. She eats **3** for breakfast...
+> **Twin:** Janet's ducks lay **24** eggs per day. She eats **5** for breakfast...
+
+A model that never saw the benchmark has no reason to prefer either version. A model that memorised it prefers the original, because those are the numbers it saw. Each test measures that preference for every question.
+
+For a clean model, the preferences land on both sides of zero, and the negative side shows what chance looks like on the positive side. A question is flagged when its preference is stronger than 95% of that mirror image. About 5% of questions get flagged by chance. Far more than that, confirmed by a sign-flip permutation test, means the model has seen the benchmark.
+
+### The tests
+
+- **completion**: give the model the first half of the question and check whether it writes the **numbers** of the second half: only the ones the first half doesn't give away. A model that never saw the question can only guess them. A model that memorised it writes the original numbers, which are wrong for the twin.
+- **min-k**: [Min-K% Prob](https://arxiv.org/abs/2310.16789). Average the model's log-probability over the 20% of words that surprise it most. Unseen text always contains some surprising words; memorised text doesn't.
+
+### Calibrating against a model that can't have seen it
+
+Even clean models slightly prefer the original numbers, because numbers a person chose fit together better than swapped ones. `contamcheck` cancels this out by subtracting the same preference measured with a **reference model**: the GPT-2 model closest in size (124M to 1.5B parameters), all released in 2019, before GSM8K, HumanEval and MBPP existed. What remains is the familiarity specific to the model under test. The size matters: larger models are better at noticing whether numbers fit together, so a clean 1.4B model looked contaminated against the 124M GPT-2 (20% flagged) but not against the 1.5B one (10%).
+
+## Does it work?
+
+A detector that never flags anything would pass every clean model, so `contamcheck` is tested in both directions: clean models must pass and contaminated models must be caught.
+
+- **Clean models**: GPT-2, GPT-Neo-125M and Pythia-1.4B were all trained on data collected before GSM8K was released in October 2021, so they cannot have seen it.
+- **Contaminated models**: [`experiments/contaminate.py`](experiments/contaminate.py) takes GPT-Neo-125M and fine-tunes it on 100 of the 200 test questions, for 1, 3 or 10 passes.
+
+200 GSM8K test questions per model, with all defaults (`contamcheck <model> gsm8k`):
+
+| Model | Saw the questions? | completion | min-k | Verdict |
+|---|---|---|---|---|
+| GPT-2 (124M) | no | 5% | 0%¹ | ✓ clean |
+| GPT-Neo (125M) | no | 3% | 6% | ✓ clean |
+| Pythia (1.4B) | no | 5% | 10% | ✓ clean |
+| GPT-Neo, 100 leaked × 1 pass | yes | 5% | **18%** | ✗ caught |
+| GPT-Neo, 100 leaked × 3 passes | yes | 8% | **34%** | ✗ caught |
+| GPT-Neo, 100 leaked × 10 passes | yes | **45%** | **53%** | ✗ caught |
+| Qwen2.5 (0.5B) | unknown | 5% | **16%** | ✗ flagged |
+
+Percentages are the share of questions flagged. About 5% is expected by chance; **bold** is significant at p < 0.001 (sign-flip permutation test). Completion uses the 154 of 200 questions whose second half contains new numbers. ¹ GPT-2 is its own reference, so min-k can't flag it.
+
+All clean models pass, and all contaminated models are caught. The two tests complement each other:
+- **min-k** is sensitive, catching even a single exposure, but relies on the reference model to cancel out other effects.
+- **completion** needs heavier memorisation, but when it fires the evidence is direct. The 10-pass model recalled the exact numbers for 86% of its leaked questions and 5% of the rest.
+
+**Qwen2.5-0.5B** is flagged more often by min-k (16%) than any clean model (at most 10%). It doesn't recall numbers verbatim, though. That's consistent with exposure to GSM8K during training, but it isn't proof: the margin over clean models is small, and Qwen was trained on far more maths data than the reference.
+
+Contamination from a single exposure is hard to detect. That matches the research: [Duan et al., 2024](https://arxiv.org/abs/2402.07841) find membership inference on LLMs is close to chance when text is seen once. Repeated exposure, which is what happens when a benchmark is copied across many web pages, is caught reliably.
+
+## What I learned building it
+
+Every version was checked against clean models (which must pass) and deliberately contaminated ones (which must be caught). Each check found a problem:
+
+1. **The first version flagged GPT-2**, which can't have seen GSM8K. Random replacement numbers read less naturally than numbers a person chose (`8471` vs `150`), so every model preferred the originals. *Fix:* draw replacements from numbers used elsewhere in the benchmark, as often as they're used there, and subtract a reference model's preference.
+2. **The second version missed models I had contaminated myself.** Training on a question also made the model more familiar with its twin, because they share 90% of their text, so comparing against "all controls" hid the signal. *Fix:* compare each question with *its own* twin and look only at the difference.
+3. **Word-for-word completion barely noticed a model that had memorised its questions.** It writes nearly the same sentence for the twin too. *Fix:* score only the numbers the first half doesn't give away.
+4. **A bigger clean model (Pythia-1.4B) was flagged** against the small GPT-2 reference. *Fix:* pick a reference of matching size.
+
+## Limitations
+
+- Twins need numbers. For benchmarks without them, pass `--control` with questions the model can't have seen (for example, written after its training cutoff).
+- min-k depends on the reference model cancelling out everything except memorisation. The largest clean reference here is GPT-2 XL (1.5B), so for bigger models pass `--reference` with a larger model trained before the benchmark existed (e.g. `EleutherAI/pythia-6.9b`, trained on 2020 data). Treat a min-k flag on its own as evidence, not proof.
+- Validated so far on GSM8K and models up to 1.5B parameters.
+- Only open-weight models for now. API models don't expose token probabilities for the prompt.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+The tests use a fake "cheating" model, so they run in under a second without downloading anything.
+
+## License
+
+MIT

contamcheck-0.1.0/README.md

@@ -0,0 +1,125 @@
+# contamcheck
+
+**Has your language model memorised the benchmark?**
+
+When a model scores 90% on GSM8K, the score only means something if the model hasn't seen the test questions before. Benchmarks are public on GitHub, in papers and in blog posts, so they leak into the web-scale data models are trained on. A model that memorised the answers looks smart without being smart.
+
+`contamcheck` tests a model for signs that it has seen a benchmark's questions during training.
+
+```
+$ contamcheck experiments/models/neo-gsm8k-10ep gsm8k
+
+contamcheck  model: experiments/models/neo-gsm8k-10ep   benchmark: gsm8k   reference: openai-community/gpt2
+             200 questions, control: number twins (23 skipped)
+
+  ✗ completion: 45% of questions flagged (70/154) — 5% expected by chance (p < 0.001)
+      Model writes the exact numbers from the second half of the question
+  ✗ min-k: 53% of questions flagged (106/200) — 5% expected by chance (p < 0.001)
+      Even the 20% most surprising words aren't surprising to the model (vs reference model)
+
+Likely contaminated — the model has probably seen these questions in training
+```
+
+*(A model deliberately trained on half of the GSM8K questions being tested. See [Does it work?](#does-it-work))*
+
+## Install
+
+```bash
+pip install contamcheck
+```
+
+Runs on Apple Silicon (MPS), NVIDIA GPUs (CUDA) or CPU. Any causal language model on the Hugging Face Hub, or a local folder, works.
+
+## Usage
+
+```bash
+contamcheck Qwen/Qwen2.5-0.5B gsm8k                          # built-in benchmark
+contamcheck Qwen/Qwen2.5-0.5B gsm8k -n 500                   # more questions, more power
+contamcheck my-model/ questions.jsonl --field question        # your own model and benchmark
+contamcheck my-model/ hf:openai/gsm8k:main:test:question     # any Hugging Face dataset
+contamcheck my-model/ humaneval --control fresh.jsonl         # control questions written after the model's cutoff
+contamcheck my-model/ gsm8k --json                            # machine-readable output
+contamcheck big-model/ gsm8k --reference EleutherAI/pythia-6.9b  # choose the reference yourself
+```
+
+The exit code is 1 when contamination is likely, so it can gate a CI pipeline.
+
+## How it works
+
+### The twin trick
+
+Each benchmark question gets a **twin**: the same text with its numbers swapped for other numbers people used elsewhere in the benchmark.
+
+> **Original:** Janet's ducks lay **16** eggs per day. She eats **3** for breakfast...
+> **Twin:** Janet's ducks lay **24** eggs per day. She eats **5** for breakfast...
+
+A model that never saw the benchmark has no reason to prefer either version. A model that memorised it prefers the original, because those are the numbers it saw. Each test measures that preference for every question.
+
+For a clean model, the preferences land on both sides of zero, and the negative side shows what chance looks like on the positive side. A question is flagged when its preference is stronger than 95% of that mirror image. About 5% of questions get flagged by chance. Far more than that, confirmed by a sign-flip permutation test, means the model has seen the benchmark.
+
+### The tests
+
+- **completion**: give the model the first half of the question and check whether it writes the **numbers** of the second half: only the ones the first half doesn't give away. A model that never saw the question can only guess them. A model that memorised it writes the original numbers, which are wrong for the twin.
+- **min-k**: [Min-K% Prob](https://arxiv.org/abs/2310.16789). Average the model's log-probability over the 20% of words that surprise it most. Unseen text always contains some surprising words; memorised text doesn't.
+
+### Calibrating against a model that can't have seen it
+
+Even clean models slightly prefer the original numbers, because numbers a person chose fit together better than swapped ones. `contamcheck` cancels this out by subtracting the same preference measured with a **reference model**: the GPT-2 model closest in size (124M to 1.5B parameters), all released in 2019, before GSM8K, HumanEval and MBPP existed. What remains is the familiarity specific to the model under test. The size matters: larger models are better at noticing whether numbers fit together, so a clean 1.4B model looked contaminated against the 124M GPT-2 (20% flagged) but not against the 1.5B one (10%).
+
+## Does it work?
+
+A detector that never flags anything would pass every clean model, so `contamcheck` is tested in both directions: clean models must pass and contaminated models must be caught.
+
+- **Clean models**: GPT-2, GPT-Neo-125M and Pythia-1.4B were all trained on data collected before GSM8K was released in October 2021, so they cannot have seen it.
+- **Contaminated models**: [`experiments/contaminate.py`](experiments/contaminate.py) takes GPT-Neo-125M and fine-tunes it on 100 of the 200 test questions, for 1, 3 or 10 passes.
+
+200 GSM8K test questions per model, with all defaults (`contamcheck <model> gsm8k`):
+
+| Model | Saw the questions? | completion | min-k | Verdict |
+|---|---|---|---|---|
+| GPT-2 (124M) | no | 5% | 0%¹ | ✓ clean |
+| GPT-Neo (125M) | no | 3% | 6% | ✓ clean |
+| Pythia (1.4B) | no | 5% | 10% | ✓ clean |
+| GPT-Neo, 100 leaked × 1 pass | yes | 5% | **18%** | ✗ caught |
+| GPT-Neo, 100 leaked × 3 passes | yes | 8% | **34%** | ✗ caught |
+| GPT-Neo, 100 leaked × 10 passes | yes | **45%** | **53%** | ✗ caught |
+| Qwen2.5 (0.5B) | unknown | 5% | **16%** | ✗ flagged |
+
+Percentages are the share of questions flagged. About 5% is expected by chance; **bold** is significant at p < 0.001 (sign-flip permutation test). Completion uses the 154 of 200 questions whose second half contains new numbers. ¹ GPT-2 is its own reference, so min-k can't flag it.
+
+All clean models pass, and all contaminated models are caught. The two tests complement each other:
+- **min-k** is sensitive, catching even a single exposure, but relies on the reference model to cancel out other effects.
+- **completion** needs heavier memorisation, but when it fires the evidence is direct. The 10-pass model recalled the exact numbers for 86% of its leaked questions and 5% of the rest.
+
+**Qwen2.5-0.5B** is flagged more often by min-k (16%) than any clean model (at most 10%). It doesn't recall numbers verbatim, though. That's consistent with exposure to GSM8K during training, but it isn't proof: the margin over clean models is small, and Qwen was trained on far more maths data than the reference.
+
+Contamination from a single exposure is hard to detect. That matches the research: [Duan et al., 2024](https://arxiv.org/abs/2402.07841) find membership inference on LLMs is close to chance when text is seen once. Repeated exposure, which is what happens when a benchmark is copied across many web pages, is caught reliably.
+
+## What I learned building it
+
+Every version was checked against clean models (which must pass) and deliberately contaminated ones (which must be caught). Each check found a problem:
+
+1. **The first version flagged GPT-2**, which can't have seen GSM8K. Random replacement numbers read less naturally than numbers a person chose (`8471` vs `150`), so every model preferred the originals. *Fix:* draw replacements from numbers used elsewhere in the benchmark, as often as they're used there, and subtract a reference model's preference.
+2. **The second version missed models I had contaminated myself.** Training on a question also made the model more familiar with its twin, because they share 90% of their text, so comparing against "all controls" hid the signal. *Fix:* compare each question with *its own* twin and look only at the difference.
+3. **Word-for-word completion barely noticed a model that had memorised its questions.** It writes nearly the same sentence for the twin too. *Fix:* score only the numbers the first half doesn't give away.
+4. **A bigger clean model (Pythia-1.4B) was flagged** against the small GPT-2 reference. *Fix:* pick a reference of matching size.
+
+## Limitations
+
+- Twins need numbers. For benchmarks without them, pass `--control` with questions the model can't have seen (for example, written after its training cutoff).
+- min-k depends on the reference model cancelling out everything except memorisation. The largest clean reference here is GPT-2 XL (1.5B), so for bigger models pass `--reference` with a larger model trained before the benchmark existed (e.g. `EleutherAI/pythia-6.9b`, trained on 2020 data). Treat a min-k flag on its own as evidence, not proof.
+- Validated so far on GSM8K and models up to 1.5B parameters.
+- Only open-weight models for now. API models don't expose token probabilities for the prompt.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+The tests use a fake "cheating" model, so they run in under a second without downloading anything.
+
+## License
+
+MIT

contamcheck-0.1.0/contamcheck/__init__.py

@@ -0,0 +1,3 @@
+"""contamcheck: check whether a language model has memorised a benchmark."""
+
+__version__ = "0.1.0"

contamcheck-0.1.0/contamcheck/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from .cli import main
+
+sys.exit(main())

contamcheck-0.1.0/contamcheck/benchmarks.py

@@ -0,0 +1,72 @@
+"""Load benchmark questions from Hugging Face or from a local file."""
+
+from __future__ import annotations
+
+import csv
+import json
+from pathlib import Path
+
+# name -> (dataset, config, split, field)
+BUILTIN = {
+    "gsm8k": ("openai/gsm8k", "main", "test", "question"),
+    "humaneval": ("openai/openai_humaneval", None, "test", "prompt"),
+    "mbpp": ("google-research-datasets/mbpp", "full", "test", "text"),
+}
+
+
+def load(spec: str, field: str | None = None) -> list[str]:
+    """Load questions from a built-in name, a local .jsonl/.csv/.txt file, or
+    `hf:dataset[:config]:split:field` for any Hugging Face dataset."""
+    path = Path(spec)
+    if path.exists():
+        return _load_file(path, field)
+    if spec in BUILTIN:
+        dataset, config, split, default_field = BUILTIN[spec]
+        return _load_hf(dataset, config, split, field or default_field)
+    if spec.startswith("hf:"):
+        parts = spec[3:].split(":")
+        if len(parts) == 3:
+            dataset, split, hf_field = parts
+            config = None
+        elif len(parts) == 4:
+            dataset, config, split, hf_field = parts
+        else:
+            raise ValueError("Use hf:dataset[:config]:split:field, e.g. hf:openai/gsm8k:main:test:question")
+        return _load_hf(dataset, config, split, hf_field)
+    raise ValueError(f"Unknown benchmark '{spec}'. Built-in: {', '.join(BUILTIN)}; "
+                     "or pass a .jsonl/.csv/.txt file, or hf:dataset[:config]:split:field")
+
+
+def _load_hf(dataset: str, config: str | None, split: str, field: str) -> list[str]:
+    from datasets import load_dataset
+
+    ds = load_dataset(dataset, config, split=split)
+    if field not in ds.column_names:
+        raise ValueError(f"Field '{field}' not in {dataset}. Columns: {', '.join(ds.column_names)}")
+    return [str(x) for x in ds[field]]
+
+
+def _load_file(path: Path, field: str | None) -> list[str]:
+    suffix = path.suffix.lower()
+    if suffix == ".txt":
+        return [line.strip() for line in path.read_text().splitlines() if line.strip()]
+    if suffix in (".jsonl", ".json"):
+        rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
+    elif suffix == ".csv":
+        with path.open(newline="") as f:
+            rows = list(csv.DictReader(f))
+    else:
+        raise ValueError(f"Unsupported file type '{suffix}' (use .jsonl, .csv or .txt)")
+    if not rows:
+        return []
+    key = field or _guess_field(rows[0])
+    if key not in rows[0]:
+        raise ValueError(f"Field '{key}' not in {path.name}. Fields: {', '.join(rows[0])}")
+    return [str(r[key]) for r in rows]
+
+
+def _guess_field(row: dict) -> str:
+    for key in ("question", "text", "prompt", "problem"):
+        if key in row:
+            return key
+    raise ValueError(f"Can't tell which field holds the question. Pass --field (one of: {', '.join(row)})")

contamcheck-0.1.0/contamcheck/checks.py

@@ -0,0 +1,277 @@
+"""The contamination tests.
+
+Every test gives each question a score where higher means "the model finds this
+more familiar". A score on its own means nothing — models find some text familiar
+just because it is ordinary English — so each benchmark question is compared with
+a control the model cannot have seen.
+
+By default the control is the question's "twin": the same text with different
+numbers. The score is then how much more the model likes the original than its
+twin. A model that never saw the benchmark has no reason to prefer either, so its
+scores land on both sides of zero, and the negative side shows what chance looks
+like on the positive side. A question is flagged when its score is higher than 95%
+of the mirrored scores; about 5% get flagged by chance, far more is memorisation.
+
+With a control file instead (questions written after the model's training cutoff)
+there are no twins, and a question is flagged if it beats 95% of the controls.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from typing import Callable, Iterator, TypeVar
+
+import numpy as np
+
+from .model import LanguageModel
+from .perturb import NUMBER
+
+PASS, WARN, FAIL = "pass", "warn", "fail"
+EXPECTED_RATE = 0.05  # share of clean questions flagged by chance
+MAX_SUFFIX_WORDS = 40
+MIN_WORDS = 8
+
+T = TypeVar("T")
+R = TypeVar("R")
+
+
+@dataclass
+class Example:
+    question: str
+    score: float
+    expected: str = ""   # completion test: the real second half
+    generated: str = ""  # completion test: what the model wrote
+
+
+@dataclass
+class TestResult:
+    name: str
+    description: str
+    n: int
+    flagged: int
+    threshold: float
+    p_value: float
+    bench_mean: float
+    control_mean: float
+    examples: list[Example] = field(default_factory=list)
+
+    @property
+    def rate(self) -> float:
+        return self.flagged / self.n if self.n else 0.0
+
+    @property
+    def status(self) -> str:
+        if self.p_value < 0.001 and self.rate >= 2 * EXPECTED_RATE:
+            return FAIL
+        if self.p_value < 0.05:
+            return WARN
+        return PASS
+
+
+def _threshold(bench: np.ndarray, control: np.ndarray | None) -> float:
+    null = -bench if control is None else control
+    return float(np.quantile(null, 1 - EXPECTED_RATE))
+
+
+def _flag_count(bench: np.ndarray, control: np.ndarray | None) -> int:
+    return int((bench > _threshold(bench, control)).sum())
+
+
+def sign_flip_p(diffs: np.ndarray, rounds: int = 2000, seed: int = 0) -> float:
+    """Paired test: if the model has no preference between original and twin, each
+    difference is as likely to be negative as positive. Flip signs at random and count
+    how often the result flags at least as many questions as we observed."""
+    observed = _flag_count(diffs, None)
+    rng = np.random.default_rng(seed)
+    hits = sum(_flag_count(diffs * rng.choice([-1, 1], len(diffs)), None) >= observed for _ in range(rounds))
+    return (hits + 1) / (rounds + 1)
+
+
+def permutation_p(bench: np.ndarray, control: np.ndarray, rounds: int = 2000, seed: int = 0) -> float:
+    """Unpaired test: chance of flagging at least as many questions if benchmark and
+    control were interchangeable. Shuffling the labels also accounts for the threshold
+    being estimated from a finite control set."""
+    observed = _flag_count(bench, control)
+    pooled = np.concatenate([bench, control])
+    rng = np.random.default_rng(seed)
+    hits = 0
+    for _ in range(rounds):
+        rng.shuffle(pooled)
+        hits += _flag_count(pooled[:len(bench)], pooled[len(bench):]) >= observed
+    return (hits + 1) / (rounds + 1)
+
+
+def compare(name: str, description: str, bench: np.ndarray, control: np.ndarray, paired: bool) -> TestResult:
+    """Flag benchmark questions that look more familiar than chance allows.
+    Paired: `bench - control` per question, tested against its own mirror image."""
+    if paired:
+        diffs = bench - control
+        threshold, flagged, p = _threshold(diffs, None), _flag_count(diffs, None), sign_flip_p(diffs)
+    else:
+        threshold, flagged, p = _threshold(bench, control), _flag_count(bench, control), permutation_p(bench, control)
+    return TestResult(
+        name=name, description=description, n=len(bench), flagged=flagged, threshold=threshold,
+        p_value=p, bench_mean=float(bench.mean()), control_mean=float(control.mean()),
+    )
+
+
+def batched(fn: Callable[[list[T]], list[R]], items: list[T], batch_size: int, desc: str) -> list[R]:
+    from tqdm import tqdm
+
+    out: list[R] = []
+    with tqdm(total=len(items), desc=desc, unit="q", leave=False) as bar:
+        for batch in _chunks(items, batch_size):
+            out.extend(fn(batch))
+            bar.update(len(batch))
+    return out
+
+
+def _chunks(items: list[T], size: int) -> Iterator[list[T]]:
+    for i in range(0, len(items), size):
+        yield items[i:i + size]
+
+
+# --- Completion test ---------------------------------------------------------
+
+def split_question(text: str, max_suffix_words: int = MAX_SUFFIX_WORDS) -> tuple[str, str] | None:
+    """Cut a question in half at a word boundary. Keeps the original whitespace so
+    code (HumanEval, MBPP) still has its newlines and indentation."""
+    words = list(re.finditer(r"\S+", text))
+    if len(words) < MIN_WORDS:
+        return None
+    cut = len(words) // 2
+    end = words[min(len(words), cut + max_suffix_words) - 1].end()
+    return text[:words[cut].start()].rstrip(), text[words[cut].start():end]
+
+
+def new_numbers(prefix: str, suffix: str) -> list[str]:
+    """Numbers in the second half that don't appear in the first half, so a model
+    can only write them by guessing or by remembering."""
+    seen = set(NUMBER.findall(prefix))
+    return [x for x in NUMBER.findall(suffix) if x not in seen]
+
+
+def completion_test(model: LanguageModel, bench: list[str], control: list[str], batch_size: int,
+                    paired: bool = True) -> TestResult:
+    """Give the model the first half and check whether it writes the *numbers* of the second half.
+
+    Comparing whole sentences doesn't work with twins: a model that memorised the
+    original writes nearly the same words for its twin, since they share 90% of their
+    text. The numbers are what differ. A model that never saw the question can only
+    guess them, and is as likely to guess the twin's as the original's; a model that
+    memorised it writes the original numbers, which are wrong for the twin."""
+    def recall(questions: list[str], desc: str) -> tuple[np.ndarray, list[tuple[str, str, str]]]:
+        pairs = [split_question(q) for q in questions]
+        generated = batched(lambda b: model.complete(b, 2 * MAX_SUFFIX_WORDS), [p for p, _ in pairs], batch_size, desc)
+        scores = []
+        for (prefix, suffix), gen in zip(pairs, generated):
+            wanted = new_numbers(prefix, suffix)
+            written = set(NUMBER.findall(gen))
+            scores.append(np.mean([x in written for x in wanted]) if wanted else np.nan)
+        return np.array(scores), [(p, s, g) for (p, s), g in zip(pairs, generated)]
+
+    bench_scores, bench_runs = recall(bench, "completion: benchmark")
+    control_scores, _ = recall(control, "completion: control")
+    # Questions whose second half brings no new numbers can't be scored
+    if paired:
+        usable = ~np.isnan(bench_scores) & ~np.isnan(control_scores)
+        b, c = bench_scores[usable], control_scores[usable]
+    else:
+        usable = ~np.isnan(bench_scores)
+        b, c = bench_scores[usable], control_scores[~np.isnan(control_scores)]
+    result = compare(
+        "completion",
+        "Model writes the exact numbers from the second half of the question",
+        b, c, paired,
+    )
+    ranked = sorted(np.flatnonzero(usable), key=lambda i: -bench_scores[i])
+    for i in ranked[:3]:
+        prefix, expected, generated = bench_runs[i]
+        if bench_scores[i] > 0:
+            result.examples.append(Example(prefix, float(bench_scores[i]), expected, generated))
+    return result
+
+
+# --- Min-K% Prob test ----------------------------------------------------------
+
+def min_k_score(logprobs: np.ndarray, k: float = 0.2) -> float:
+    """Average log-probability of the k% least likely tokens (Shi et al., 2023).
+
+    Unseen text always contains some tokens that surprise the model (a name, an odd
+    number). Memorised text doesn't: even its rarest tokens get high probability.
+    Averaging only the most surprising tokens makes that difference stand out."""
+    if len(logprobs) == 0:
+        return 0.0
+    n = max(1, int(len(logprobs) * k))
+    return float(np.sort(logprobs)[:n].mean())
+
+
+def min_k_test(model: LanguageModel, bench: list[str], control: list[str], batch_size: int,
+               paired: bool = True, reference: LanguageModel | None = None, k: float = 0.2) -> TestResult:
+    """Min-K% Prob, calibrated against a reference model when one is given.
+
+    Even with natural replacement numbers, every model slightly prefers the original
+    question (its numbers fit together better). Subtracting the score of a reference
+    model that provably never saw the benchmark (GPT-2 predates them) cancels that
+    out and leaves only the familiarity specific to the tested model."""
+    def scores(m: LanguageModel, questions: list[str], desc: str) -> np.ndarray:
+        logprobs = batched(m.token_logprobs, questions, batch_size, desc)
+        return np.array([min_k_score(lp, k) for lp in logprobs])
+
+    bench_scores = scores(model, bench, "min-k%: benchmark")
+    control_scores = scores(model, control, "min-k%: control")
+    if reference is not None:
+        bench_scores = bench_scores - scores(reference, bench, "min-k%: reference, benchmark")
+        control_scores = control_scores - scores(reference, control, "min-k%: reference, control")
+    result = compare(
+        "min-k",
+        f"Even the {k:.0%} most surprising words aren't surprising to the model"
+        + (" (vs reference model)" if reference is not None else ""),
+        bench_scores, control_scores, paired,
+    )
+    ranking = bench_scores - control_scores if paired else bench_scores
+    for i in np.argsort(-ranking)[:3]:
+        result.examples.append(Example(bench[i], float(bench_scores[i])))
+    return result
+
+
+TESTS = {"completion": completion_test, "min-k": min_k_test}
+
+
+# --- Building benchmark and control sets ---------------------------------------
+
+@dataclass
+class Sets:
+    bench: list[str]
+    control: list[str]
+    control_kind: str  # "number twins" or "your control file"
+    skipped: int       # benchmark questions left out (too short, or no numbers to swap)
+
+    @property
+    def paired(self) -> bool:
+        return self.control_kind == "number twins"
+
+
+def build_sets(questions: list[str], control_questions: list[str] | None, n: int, seed: int = 0) -> Sets:
+    """Sample up to n benchmark questions and a matching control set.
+
+    Without a control file, each control is the question's twin: the same text with
+    its numbers swapped for others from the benchmark. Questions without numbers
+    have no twin and are skipped."""
+    import random
+
+    from .perturb import NumberPool, perturb
+
+    rng = random.Random(seed)
+    usable = [q for q in questions if split_question(q)]
+    if control_questions is None:
+        pool = NumberPool(questions)
+        pairs = [(q, perturb(q, pool, seed)) for q in usable]
+        pairs = [(q, c) for q, c in pairs if q != c]
+        sample = rng.sample(pairs, min(n, len(pairs)))
+        return Sets([q for q, _ in sample], [c for _, c in sample], "number twins",
+                    len(questions) - len(pairs))
+    controls = [q for q in control_questions if split_question(q)]
+    return Sets(rng.sample(usable, min(n, len(usable))), rng.sample(controls, min(n, len(controls))),
+                "your control file", len(questions) - len(usable))