leakit 0.1.0__tar.gz

leakit-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+.venv/
+.venv-test/
+dist/
+build/
+*.egg-info/
+__pycache__/
+*.pyc
+.pytest_cache/

leakit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Victor Maricato
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

leakit-0.1.0/PKG-INFO

@@ -0,0 +1,136 @@
+Metadata-Version: 2.4
+Name: leakit
+Version: 0.1.0
+Summary: Continuation-free membership inference on closed language models via sample self-concentration
+Project-URL: Homepage, https://github.com/victormaricato/leakit
+Project-URL: Repository, https://github.com/victormaricato/leakit
+Author: Victor Maricato
+License: MIT
+License-File: LICENSE
+Keywords: audit,llm,membership-inference,mia,privacy,security
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Security
+Requires-Python: >=3.9
+Requires-Dist: openai>=1.40
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# leakit
+
+**Continuation-free membership inference for closed language models.**
+
+`leakit` tells you whether a document was likely in a model's training set using
+nothing but its sampling API. No logits, no log-probabilities, and -- unlike
+prior sampling attacks such as SaMIA -- no need to know the document's true
+continuation. You give it the *opening* of a document; it samples several
+continuations and measures how much they agree with each other. Training
+documents pull the model's continuation distribution toward the memorised text,
+so the samples concentrate; novel documents leave the distribution diffuse.
+
+This is the reference implementation of the *self-concentration* attack from the
+paper *"Leak It: Continuation-Free Membership Inference on Closed Language Models
+via Sample Self-Concentration."*
+
+## Install
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/victormaricato/leakit/main/install.sh | bash
+```
+
+or, directly, with any of:
+
+```bash
+uv tool install leakit       # recommended
+pipx install leakit
+pip install leakit
+```
+
+## Use
+
+`leakit` talks to any **OpenAI-compatible** endpoint. Set the API key for the
+service you are probing -- the key maps to whatever provider `--base-url` points
+at -- then run it.
+
+```bash
+export LEAKIT_API_KEY="sk-..."        # or OPENAI_API_KEY
+
+# OpenAI
+leakit --model gpt-4o-mini suspect.txt
+
+# Anything OpenAI-compatible (OpenRouter, Anthropic compat route, vLLM, Together, local server)
+leakit --model anthropic/claude-3.5-sonnet \
+       --base-url https://openrouter.ai/api/v1 \
+       --api-key-env OPENROUTER_API_KEY \
+       -n 32 suspect.txt
+
+# Compare a candidate against known non-member documents (relative percentile)
+leakit --model gpt-4o-mini --calibrate clean/*.txt suspect.txt
+
+# Pipe text in, get JSON out
+cat article.txt | leakit --model gpt-4o-mini --json
+```
+
+Output:
+
+```
+document     score  samples
+-----------------------------
+suspect.txt  0.4213   32/32
+```
+
+A higher score means the sampled continuations agree more, which correlates with
+membership. The absolute scale is model-dependent, so interpret scores
+*relatively*: score several documents together, or use `--calibrate` with a set
+of documents you know were **not** in training to get a percentile.
+
+### Key options
+
+| Flag | Meaning | Default |
+|------|---------|---------|
+| `--model` | model id passed to the API | required |
+| `--base-url` | OpenAI-compatible endpoint | OpenAI |
+| `--api-key-env` | env var holding the key | `LEAKIT_API_KEY`, then `OPENAI_API_KEY` |
+| `-n, --samples` | continuations per document | 16 |
+| `--max-tokens` | tokens per continuation | 64 |
+| `--temperature` | sampling temperature | 1.0 |
+| `--prefix-chars` | chars of each doc used as the prefix (0 = whole doc) | 256 |
+| `--statistic` | `word-jaccard` (parameter-free) or `kgram` | `word-jaccard` |
+| `--mode` | `chat` (closed APIs) or `completion` (base models) | `chat` |
+| `--calibrate` | non-member baseline file(s) for a percentile | off |
+| `--json` | machine-readable output | off |
+
+For base/text-completion models (e.g. self-hosted Pythia/Llama base), use
+`--mode completion` to sample the raw continuation distribution. For chat/instruct
+models, the default `chat` mode asks the model to continue the passage verbatim.
+
+## Python API
+
+```python
+from leakit import LeakIt
+
+scorer = LeakIt(model="gpt-4o-mini", n_samples=32)   # reads LEAKIT_API_KEY/OPENAI_API_KEY
+result = scorer.score(open("suspect.txt").read())
+print(result.score, result.n_returned)
+```
+
+The raw statistics are exposed too:
+
+```python
+from leakit import self_concentration_word_jaccard
+self_concentration_word_jaccard(["a b c", "a b c", "x y z"])
+```
+
+## Responsible use
+
+`leakit` is a privacy-auditing and red-teaming tool: use it to test models you
+own or are authorised to assess. The self-concentration signal is a statistical
+indicator, not proof of membership; calibrate before drawing conclusions.
+
+## License
+
+MIT.

leakit-0.1.0/README.md

@@ -0,0 +1,114 @@
+# leakit
+
+**Continuation-free membership inference for closed language models.**
+
+`leakit` tells you whether a document was likely in a model's training set using
+nothing but its sampling API. No logits, no log-probabilities, and -- unlike
+prior sampling attacks such as SaMIA -- no need to know the document's true
+continuation. You give it the *opening* of a document; it samples several
+continuations and measures how much they agree with each other. Training
+documents pull the model's continuation distribution toward the memorised text,
+so the samples concentrate; novel documents leave the distribution diffuse.
+
+This is the reference implementation of the *self-concentration* attack from the
+paper *"Leak It: Continuation-Free Membership Inference on Closed Language Models
+via Sample Self-Concentration."*
+
+## Install
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/victormaricato/leakit/main/install.sh | bash
+```
+
+or, directly, with any of:
+
+```bash
+uv tool install leakit       # recommended
+pipx install leakit
+pip install leakit
+```
+
+## Use
+
+`leakit` talks to any **OpenAI-compatible** endpoint. Set the API key for the
+service you are probing -- the key maps to whatever provider `--base-url` points
+at -- then run it.
+
+```bash
+export LEAKIT_API_KEY="sk-..."        # or OPENAI_API_KEY
+
+# OpenAI
+leakit --model gpt-4o-mini suspect.txt
+
+# Anything OpenAI-compatible (OpenRouter, Anthropic compat route, vLLM, Together, local server)
+leakit --model anthropic/claude-3.5-sonnet \
+       --base-url https://openrouter.ai/api/v1 \
+       --api-key-env OPENROUTER_API_KEY \
+       -n 32 suspect.txt
+
+# Compare a candidate against known non-member documents (relative percentile)
+leakit --model gpt-4o-mini --calibrate clean/*.txt suspect.txt
+
+# Pipe text in, get JSON out
+cat article.txt | leakit --model gpt-4o-mini --json
+```
+
+Output:
+
+```
+document     score  samples
+-----------------------------
+suspect.txt  0.4213   32/32
+```
+
+A higher score means the sampled continuations agree more, which correlates with
+membership. The absolute scale is model-dependent, so interpret scores
+*relatively*: score several documents together, or use `--calibrate` with a set
+of documents you know were **not** in training to get a percentile.
+
+### Key options
+
+| Flag | Meaning | Default |
+|------|---------|---------|
+| `--model` | model id passed to the API | required |
+| `--base-url` | OpenAI-compatible endpoint | OpenAI |
+| `--api-key-env` | env var holding the key | `LEAKIT_API_KEY`, then `OPENAI_API_KEY` |
+| `-n, --samples` | continuations per document | 16 |
+| `--max-tokens` | tokens per continuation | 64 |
+| `--temperature` | sampling temperature | 1.0 |
+| `--prefix-chars` | chars of each doc used as the prefix (0 = whole doc) | 256 |
+| `--statistic` | `word-jaccard` (parameter-free) or `kgram` | `word-jaccard` |
+| `--mode` | `chat` (closed APIs) or `completion` (base models) | `chat` |
+| `--calibrate` | non-member baseline file(s) for a percentile | off |
+| `--json` | machine-readable output | off |
+
+For base/text-completion models (e.g. self-hosted Pythia/Llama base), use
+`--mode completion` to sample the raw continuation distribution. For chat/instruct
+models, the default `chat` mode asks the model to continue the passage verbatim.
+
+## Python API
+
+```python
+from leakit import LeakIt
+
+scorer = LeakIt(model="gpt-4o-mini", n_samples=32)   # reads LEAKIT_API_KEY/OPENAI_API_KEY
+result = scorer.score(open("suspect.txt").read())
+print(result.score, result.n_returned)
+```
+
+The raw statistics are exposed too:
+
+```python
+from leakit import self_concentration_word_jaccard
+self_concentration_word_jaccard(["a b c", "a b c", "x y z"])
+```
+
+## Responsible use
+
+`leakit` is a privacy-auditing and red-teaming tool: use it to test models you
+own or are authorised to assess. The self-concentration signal is a statistical
+indicator, not proof of membership; calibrate before drawing conclusions.
+
+## License
+
+MIT.

leakit-0.1.0/install.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# leakit installer.
+#
+#   curl -fsSL https://raw.githubusercontent.com/victormaricato/leakit/main/install.sh | bash
+#
+# Installs the `leakit` CLI into an isolated environment using uv (preferred)
+# or pipx. Override the source with LEAKIT_SOURCE, e.g.
+#   LEAKIT_SOURCE="git+https://github.com/victormaricato/leakit.git" bash install.sh
+set -euo pipefail
+
+SOURCE="${LEAKIT_SOURCE:-leakit}"
+GIT_FALLBACK="git+https://github.com/victormaricato/leakit.git"
+
+say()  { printf '\033[1;34m==>\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33mwarning:\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[1;31merror:\033[0m %s\n' "$*" >&2; exit 1; }
+
+ensure_uv() {
+  if command -v uv >/dev/null 2>&1; then return 0; fi
+  say "uv not found; installing it..."
+  curl -fsSL https://astral.sh/uv/install.sh | sh
+  # uv installs to ~/.local/bin by default
+  export PATH="$HOME/.local/bin:$PATH"
+  command -v uv >/dev/null 2>&1
+}
+
+install_with_uv() {
+  say "Installing leakit from '$SOURCE' with uv..."
+  if ! uv tool install --force "$SOURCE" 2>/dev/null; then
+    warn "install from '$SOURCE' failed; falling back to GitHub source"
+    uv tool install --force "$GIT_FALLBACK"
+  fi
+}
+
+install_with_pipx() {
+  say "Installing leakit from '$SOURCE' with pipx..."
+  if ! pipx install --force "$SOURCE" 2>/dev/null; then
+    warn "install from '$SOURCE' failed; falling back to GitHub source"
+    pipx install --force "$GIT_FALLBACK"
+  fi
+}
+
+main() {
+  command -v curl >/dev/null 2>&1 || die "curl is required"
+  if ensure_uv; then
+    install_with_uv
+  elif command -v pipx >/dev/null 2>&1; then
+    install_with_pipx
+  else
+    die "need uv or pipx to install; see https://docs.astral.sh/uv/"
+  fi
+
+  echo
+  say "leakit installed. Verify with:  leakit --version"
+  cat <<'EOF'
+
+Next steps:
+  1. Export the API key for the service you want to probe, e.g.
+       export LEAKIT_API_KEY="sk-..."          # or OPENAI_API_KEY
+  2. Score a document:
+       leakit --model gpt-4o-mini suspect.txt
+  3. Probe any OpenAI-compatible endpoint with --base-url / --api-key-env.
+
+If `leakit` is not found, add uv's tool bin to your PATH:
+       export PATH="$HOME/.local/bin:$PATH"
+EOF
+}
+
+main "$@"

leakit-0.1.0/pyproject.toml

@@ -0,0 +1,44 @@
+[project]
+name = "leakit"
+version = "0.1.0"
+description = "Continuation-free membership inference on closed language models via sample self-concentration"
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Victor Maricato" }]
+keywords = ["membership-inference", "privacy", "llm", "security", "mia", "audit"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Security",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "openai>=1.40",
+]
+
+[project.urls]
+Homepage = "https://github.com/victormaricato/leakit"
+Repository = "https://github.com/victormaricato/leakit"
+
+[project.scripts]
+leakit = "leakit.cli:main"
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/leakit"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src/leakit", "tests", "README.md", "LICENSE", "install.sh", "pyproject.toml"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py39"

leakit-0.1.0/src/leakit/__init__.py

@@ -0,0 +1,21 @@
+"""leakit: continuation-free membership inference for closed language models.
+
+Public API:
+    LeakIt        - high-level scorer over any OpenAI-compatible endpoint
+    ScoreResult   - per-document result
+    self_concentration_word_jaccard / self_concentration_kgram - raw statistics
+"""
+
+from ._stats import self_concentration_kgram, self_concentration_word_jaccard
+from .core import LeakIt, ScoreResult, percentile_of
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "LeakIt",
+    "ScoreResult",
+    "percentile_of",
+    "self_concentration_word_jaccard",
+    "self_concentration_kgram",
+    "__version__",
+]

leakit-0.1.0/src/leakit/_stats.py

@@ -0,0 +1,69 @@
+"""Self-concentration statistics for continuation-free membership inference.
+
+These are the membership signals defined in the paper. The statistic operates
+purely on a set of sampled continuations: it never sees a gold continuation and
+needs no model internals. Higher values indicate a more concentrated sampling
+distribution, which the paper shows is predictive of training-set membership.
+
+Pure Python, no third-party dependencies, so the package stays lightweight.
+"""
+
+from __future__ import annotations
+
+from itertools import combinations
+
+
+def _word_set(text: str) -> set[str]:
+    return set(text.split())
+
+
+def _kgram_set(text: str, k: int) -> set[str]:
+    if len(text) < k:
+        return set()
+    return {text[i : i + k] for i in range(len(text) - k + 1)}
+
+
+def _mean_pairwise_jaccard(sets: list[set[str]]) -> float:
+    if len(sets) < 2:
+        return 0.0
+    total = 0.0
+    pairs = 0
+    for a, b in combinations(sets, 2):
+        union = a | b
+        if not union:
+            continue
+        total += len(a & b) / len(union)
+        pairs += 1
+    return total / pairs if pairs else 0.0
+
+
+def self_concentration_word_jaccard(completions: list[str]) -> float:
+    """Parameter-free self-concentration: mean pairwise word-set Jaccard.
+
+    Each completion is reduced to its set of whitespace-delimited tokens, and we
+    average the Jaccard similarity across all unordered pairs. This is the
+    headline statistic in the paper (no n-gram size to tune).
+    """
+    return _mean_pairwise_jaccard([_word_set(c) for c in completions])
+
+
+def self_concentration_kgram(completions: list[str], k: int = 5) -> float:
+    """Self-concentration over character k-grams: mean pairwise k-gram Jaccard."""
+    return _mean_pairwise_jaccard([_kgram_set(c, k) for c in completions])
+
+
+STATISTICS = {
+    "word-jaccard": lambda completions, k: self_concentration_word_jaccard(completions),
+    "kgram": self_concentration_kgram,
+}
+
+
+def compute(completions: list[str], statistic: str = "word-jaccard", k: int = 5) -> float:
+    """Dispatch to the named statistic. Raises ValueError on unknown names."""
+    try:
+        fn = STATISTICS[statistic]
+    except KeyError:
+        raise ValueError(
+            f"unknown statistic {statistic!r}; choose from {sorted(STATISTICS)}"
+        ) from None
+    return fn(completions, k)

leakit-0.1.0/src/leakit/cli.py

@@ -0,0 +1,175 @@
+"""Command-line interface for leakit."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from . import __version__, _stats
+from .core import LeakIt, ScoreResult, percentile_of
+
+_EPILOG = """\
+examples:
+  # score a file against an OpenAI model (reads LEAKIT_API_KEY or OPENAI_API_KEY)
+  leakit --model gpt-4o-mini suspect.txt
+
+  # probe a model served behind any OpenAI-compatible endpoint
+  leakit --model anthropic/claude-3.5-sonnet \\
+         --base-url https://openrouter.ai/api/v1 \\
+         --api-key-env OPENROUTER_API_KEY \\
+         -n 32 suspect.txt
+
+  # compare a candidate against a baseline of known non-member documents
+  leakit --model gpt-4o-mini --calibrate known_clean/*.txt suspect.txt
+
+  # pipe text in
+  cat article.txt | leakit --model gpt-4o-mini
+
+A higher self-concentration score means the model's continuations agree more,
+which the paper shows correlates with training-set membership. The absolute
+scale is model-dependent; use --calibrate (or score several documents together)
+to interpret a score relatively.
+"""
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="leakit",
+        description="Continuation-free membership inference on closed language models.",
+        epilog=_EPILOG,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    p.add_argument("documents", nargs="*", help="document file(s) to score; omit to read stdin")
+    p.add_argument("--text", action="append", default=[], metavar="STR",
+                   help="inline document text (repeatable)")
+    p.add_argument("--model", required=True, help="model id passed to the API")
+    p.add_argument("--base-url", default=None,
+                   help="OpenAI-compatible base URL (default: OpenAI)")
+    p.add_argument("--api-key-env", default=None, metavar="VAR",
+                   help="env var holding the API key (default: LEAKIT_API_KEY then OPENAI_API_KEY)")
+    p.add_argument("-n", "--samples", type=int, default=16,
+                   help="number of sampled continuations per document (default: 16)")
+    p.add_argument("--max-tokens", type=int, default=64,
+                   help="tokens generated per continuation (default: 64)")
+    p.add_argument("--temperature", type=float, default=1.0, help="sampling temperature (default: 1.0)")
+    p.add_argument("--top-p", type=float, default=1.0, help="nucleus sampling top-p (default: 1.0)")
+    p.add_argument("--prefix-chars", type=int, default=256,
+                   help="chars of each document used as the conditioning prefix; 0 = whole document (default: 256)")
+    p.add_argument("--statistic", choices=sorted(_stats.STATISTICS), default="word-jaccard",
+                   help="self-concentration statistic (default: word-jaccard)")
+    p.add_argument("--k", type=int, default=5, help="k for the kgram statistic (default: 5)")
+    p.add_argument("--mode", choices=("chat", "completion"), default="chat",
+                   help="API surface: chat for instruct/closed APIs (default), completion for base models")
+    p.add_argument("--concurrency", type=int, default=8, help="parallel requests (default: 8)")
+    p.add_argument("--n-per-request", type=int, default=1,
+                   help="continuations per API call via the provider's n param (default: 1)")
+    p.add_argument("--calibrate", default=None, metavar="GLOB",
+                   help="glob (quote it) or comma-separated path(s) of known NON-member documents; "
+                        "report each candidate's percentile vs this baseline")
+    p.add_argument("--json", action="store_true", help="emit JSON instead of a table")
+    p.add_argument("--show-completions", action="store_true",
+                   help="include raw completions in JSON output")
+    p.add_argument("--version", action="version", version=f"leakit {__version__}")
+    return p
+
+
+def _read_documents(args) -> list[tuple[str, str]]:
+    """Return (id, text) pairs from files, --text, or stdin."""
+    docs: list[tuple[str, str]] = []
+    for path in args.documents:
+        text = Path(path).read_text(encoding="utf-8", errors="replace")
+        docs.append((path, text))
+    for i, t in enumerate(args.text):
+        docs.append((f"--text[{i}]", t))
+    if not docs and not sys.stdin.isatty():
+        data = sys.stdin.read()
+        if data.strip():
+            docs.append(("<stdin>", data))
+    return docs
+
+
+def _make_scorer(args) -> LeakIt:
+    return LeakIt(
+        model=args.model,
+        base_url=args.base_url,
+        api_key_env=args.api_key_env,
+        n_samples=args.samples,
+        max_tokens=args.max_tokens,
+        temperature=args.temperature,
+        top_p=args.top_p,
+        mode=args.mode,
+        concurrency=args.concurrency,
+        n_per_request=args.n_per_request,
+        statistic=args.statistic,
+        k=args.k,
+        prefix_chars=args.prefix_chars,
+    )
+
+
+def _print_table(results: list[ScoreResult], percentiles: dict[str, float] | None) -> None:
+    name_w = max([len(r.document_id) for r in results] + [8])
+    header = f"{'document':<{name_w}}  {'score':>7}  {'samples':>7}"
+    if percentiles is not None:
+        header += f"  {'pctile':>7}"
+    print(header)
+    print("-" * len(header))
+    for r in results:
+        line = f"{r.document_id:<{name_w}}  {r.score:>7.4f}  {r.n_returned:>3}/{r.n_requested:<3}"
+        if percentiles is not None:
+            pv = percentiles.get(r.document_id, float("nan"))
+            line += f"  {pv:>6.1f}%"
+        print(line)
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    docs = _read_documents(args)
+    if not docs:
+        print("error: no documents given (pass file paths, --text, or pipe stdin)", file=sys.stderr)
+        return 2
+
+    try:
+        scorer = _make_scorer(args)
+    except (RuntimeError, ValueError) as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 2
+
+    results = [scorer.score(text, document_id=doc_id) for doc_id, text in docs]
+
+    percentiles: dict[str, float] | None = None
+    if args.calibrate:
+        import glob
+
+        paths: list[str] = []
+        for pattern in args.calibrate.split(","):
+            paths.extend(sorted(glob.glob(pattern.strip())))
+        if not paths:
+            print(f"error: --calibrate matched no files: {args.calibrate!r}", file=sys.stderr)
+            return 2
+        baseline_scores = [
+            scorer.score(Path(p).read_text(encoding="utf-8", errors="replace"),
+                         document_id=p).score
+            for p in paths
+        ]
+        percentiles = {r.document_id: percentile_of(r.score, baseline_scores) for r in results}
+
+    if args.json:
+        out = [r.as_dict(include_completions=args.show_completions) for r in results]
+        if percentiles is not None:
+            for d in out:
+                d["percentile_vs_baseline"] = percentiles.get(d["document"])
+        print(json.dumps(out, indent=2))
+    else:
+        _print_table(results, percentiles)
+
+    if any(r.n_returned == 0 for r in results):
+        print("warning: some documents returned no completions (check model/endpoint/key)",
+              file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

leakit-0.1.0/src/leakit/core.py

@@ -0,0 +1,122 @@
+"""High-level scoring API: prefix extraction, sampling, self-concentration."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from . import _stats
+from .sampler import Sampler, SamplerConfig, resolve_api_key
+
+
+@dataclass
+class ScoreResult:
+    document_id: str
+    score: float
+    n_requested: int
+    n_returned: int
+    statistic: str
+    prefix: str
+    completions: list[str]
+
+    def as_dict(self, include_completions: bool = False) -> dict:
+        d = {
+            "document": self.document_id,
+            "score": self.score,
+            "statistic": self.statistic,
+            "n_requested": self.n_requested,
+            "n_returned": self.n_returned,
+            "prefix_preview": self.prefix[:120],
+        }
+        if include_completions:
+            d["completions"] = self.completions
+        return d
+
+
+def make_prefix(document: str, prefix_chars: int) -> str:
+    """Take the conditioning prefix from the start of the document.
+
+    prefix_chars == 0 means use the whole document. The prefix is what the model
+    conditions on; the rest of the document is never sent (continuation-free).
+    """
+    text = document.strip()
+    if prefix_chars and prefix_chars > 0:
+        return text[:prefix_chars]
+    return text
+
+
+class LeakIt:
+    """Continuation-free membership-inference scorer over an OpenAI-compatible API.
+
+    Example
+    -------
+    >>> scorer = LeakIt(model="gpt-4o-mini")  # reads LEAKIT_API_KEY / OPENAI_API_KEY
+    >>> result = scorer.score("In the beginning the Universe was created.")
+    >>> result.score
+    0.42
+    """
+
+    def __init__(
+        self,
+        model: str,
+        *,
+        base_url: str | None = None,
+        api_key: str | None = None,
+        api_key_env: str | None = None,
+        n_samples: int = 16,
+        max_tokens: int = 64,
+        temperature: float = 1.0,
+        top_p: float = 1.0,
+        mode: str = "chat",
+        concurrency: int = 8,
+        n_per_request: int = 1,
+        statistic: str = "word-jaccard",
+        k: int = 5,
+        prefix_chars: int = 256,
+        client=None,
+    ):
+        if statistic not in _stats.STATISTICS:
+            raise ValueError(
+                f"unknown statistic {statistic!r}; choose from {sorted(_stats.STATISTICS)}"
+            )
+        self.statistic = statistic
+        self.k = k
+        self.prefix_chars = prefix_chars
+        self.n_samples = n_samples
+        key = api_key or resolve_api_key(api_key_env)
+        self.sampler = Sampler(
+            SamplerConfig(
+                model=model,
+                base_url=base_url,
+                n_samples=n_samples,
+                max_tokens=max_tokens,
+                temperature=temperature,
+                top_p=top_p,
+                mode=mode,
+                concurrency=concurrency,
+                n_per_request=n_per_request,
+            ),
+            api_key=key,
+            client=client,
+        )
+
+    def score(self, document: str, document_id: str = "<text>") -> ScoreResult:
+        prefix = make_prefix(document, self.prefix_chars)
+        completions = self.sampler.sample(prefix)
+        value = _stats.compute(completions, self.statistic, self.k)
+        return ScoreResult(
+            document_id=document_id,
+            score=value,
+            n_requested=self.n_samples,
+            n_returned=len(completions),
+            statistic=self.statistic,
+            prefix=prefix,
+            completions=completions,
+        )
+
+
+def percentile_of(value: float, baseline: list[float]) -> float:
+    """Fraction of baseline scores below ``value`` (0-100). Empty baseline -> nan."""
+    if not baseline:
+        return float("nan")
+    below = sum(1 for b in baseline if b < value)
+    return 100.0 * below / len(baseline)

leakit-0.1.0/src/leakit/sampler.py

@@ -0,0 +1,155 @@
+"""Sampling backend built on the OpenAI Python SDK.
+
+Any OpenAI-compatible endpoint works: set ``base_url`` to the provider you are
+probing (OpenAI, Anthropic via its OpenAI-compatible route, OpenRouter, vLLM,
+Together, a local server, ...) and supply the matching API key. The attack only
+needs a sampling endpoint; it never reads logits.
+"""
+
+from __future__ import annotations
+
+import os
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from dataclasses import dataclass, field
+
+# The continuation instruction used in chat mode. Closed chat APIs do not expose
+# a raw text-completion surface, so we ask the model to continue the passage
+# verbatim. Base/text models should use mode="completion" for the unbiased
+# sampling distribution the paper studies.
+_CONTINUE_SYSTEM = (
+    "You continue text. Given the beginning of a passage, write the text that "
+    "most plausibly comes next. Output only the continuation, with no preamble, "
+    "quotation marks, or commentary."
+)
+
+DEFAULT_API_KEY_ENVS = ("LEAKIT_API_KEY", "OPENAI_API_KEY")
+
+
+def resolve_api_key(api_key_env: str | None = None) -> str:
+    """Resolve the API key from the environment.
+
+    If ``api_key_env`` is given, only that variable is consulted. Otherwise the
+    default chain (LEAKIT_API_KEY then OPENAI_API_KEY) is tried.
+    """
+    candidates = (api_key_env,) if api_key_env else DEFAULT_API_KEY_ENVS
+    for name in candidates:
+        if name and os.environ.get(name):
+            return os.environ[name]
+    tried = ", ".join(c for c in candidates if c)
+    raise RuntimeError(
+        f"no API key found in environment (looked at: {tried}). "
+        f"Export your provider key, e.g. `export {candidates[0]}=sk-...`."
+    )
+
+
+@dataclass
+class SamplerConfig:
+    model: str
+    base_url: str | None = None
+    n_samples: int = 16
+    max_tokens: int = 64
+    temperature: float = 1.0
+    top_p: float = 1.0
+    mode: str = "chat"  # "chat" or "completion"
+    concurrency: int = 8
+    n_per_request: int = 1  # set >1 to batch via the provider's `n` param
+    timeout: float = 120.0
+    max_retries: int = 4
+    extra_body: dict = field(default_factory=dict)
+
+
+class Sampler:
+    """Draws continuations of a prefix from an OpenAI-compatible endpoint."""
+
+    def __init__(self, config: SamplerConfig, api_key: str, client=None):
+        self.config = config
+        # `client` is injectable for testing; defaults to a real OpenAI client.
+        if client is None:
+            from openai import OpenAI
+
+            client = OpenAI(
+                api_key=api_key,
+                base_url=config.base_url,
+                timeout=config.timeout,
+                max_retries=0,  # we handle retries/backoff ourselves
+            )
+        self.client = client
+
+    def _request(self, prefix: str, n: int) -> list[str]:
+        """One API call returning up to ``n`` continuations."""
+        cfg = self.config
+        if cfg.mode == "completion":
+            resp = self.client.completions.create(
+                model=cfg.model,
+                prompt=prefix,
+                max_tokens=cfg.max_tokens,
+                temperature=cfg.temperature,
+                top_p=cfg.top_p,
+                n=n,
+                extra_body=cfg.extra_body or None,
+            )
+            return [choice.text or "" for choice in resp.choices]
+        elif cfg.mode == "chat":
+            resp = self.client.chat.completions.create(
+                model=cfg.model,
+                messages=[
+                    {"role": "system", "content": _CONTINUE_SYSTEM},
+                    {"role": "user", "content": prefix},
+                ],
+                max_tokens=cfg.max_tokens,
+                temperature=cfg.temperature,
+                top_p=cfg.top_p,
+                n=n,
+                extra_body=cfg.extra_body or None,
+            )
+            return [(choice.message.content or "") for choice in resp.choices]
+        raise ValueError(f"unknown mode {cfg.mode!r}; use 'chat' or 'completion'")
+
+    def _request_with_retry(self, prefix: str, n: int) -> list[str]:
+        import time
+
+        last_exc: Exception | None = None
+        for attempt in range(self.config.max_retries):
+            try:
+                return self._request(prefix, n)
+            except Exception as exc:  # noqa: BLE001 - provider errors are heterogeneous
+                last_exc = exc
+                if not _is_retryable(exc) or attempt == self.config.max_retries - 1:
+                    break
+                time.sleep(min(2**attempt, 30))
+        raise RuntimeError(f"sampling request failed: {last_exc}") from last_exc
+
+    def sample(self, prefix: str) -> list[str]:
+        """Return up to ``n_samples`` continuations of ``prefix``.
+
+        Requests are batched by ``n_per_request`` and run concurrently. Failed
+        requests are skipped; the returned list may be shorter than n_samples.
+        """
+        cfg = self.config
+        per = max(1, cfg.n_per_request)
+        # Build a list of per-request batch sizes summing to n_samples.
+        batches: list[int] = []
+        remaining = cfg.n_samples
+        while remaining > 0:
+            batches.append(min(per, remaining))
+            remaining -= batches[-1]
+
+        completions: list[str] = []
+        workers = max(1, min(cfg.concurrency, len(batches)))
+        with ThreadPoolExecutor(max_workers=workers) as ex:
+            futures = [ex.submit(self._request_with_retry, prefix, b) for b in batches]
+            for fut in as_completed(futures):
+                try:
+                    completions.extend(fut.result())
+                except Exception:  # noqa: BLE001 - one dead batch shouldn't kill the run
+                    continue
+        return completions
+
+
+def _is_retryable(exc: Exception) -> bool:
+    """Heuristic: retry on rate limits, timeouts, and 5xx; not on 4xx/auth."""
+    status = getattr(exc, "status_code", None)
+    if status is not None:
+        return status == 429 or status >= 500
+    name = type(exc).__name__.lower()
+    return any(tok in name for tok in ("timeout", "connection", "ratelimit", "apierror"))

leakit-0.1.0/tests/conftest.py

@@ -0,0 +1,80 @@
+"""Shared fakes that mimic the openai client response shape."""
+
+import threading
+import types
+
+import pytest
+
+
+def _chat_response(texts):
+    return types.SimpleNamespace(
+        choices=[
+            types.SimpleNamespace(message=types.SimpleNamespace(content=t)) for t in texts
+        ]
+    )
+
+
+def _completion_response(texts):
+    return types.SimpleNamespace(
+        choices=[types.SimpleNamespace(text=t) for t in texts]
+    )
+
+
+class FakeAPIError(Exception):
+    """Mimics a retryable provider error (carries an HTTP status code)."""
+
+    def __init__(self, status_code=503):
+        super().__init__(f"fake {status_code}")
+        self.status_code = status_code
+
+
+class FakeClient:
+    """Returns canned continuations keyed by a substring of the prefix.
+
+    Successive requests rotate through the pool (so independent single-sample
+    calls vary, like a real model). `fail_times` raises a retryable error on the
+    first N calls to exercise backoff.
+    """
+
+    def __init__(self, script=None, default=None, fail_times=0):
+        self.script = script or {}
+        self.default = default or ["a b c", "a b d", "a b e"]
+        self.fail_times = fail_times
+        self.calls = 0
+        self._offset = 0
+        self._lock = threading.Lock()
+        self.chat = types.SimpleNamespace(completions=types.SimpleNamespace(create=self._chat))
+        self.completions = types.SimpleNamespace(create=self._completion)
+
+    def _pick(self, prompt):
+        for key, comps in self.script.items():
+            if key in prompt:
+                return comps
+        return self.default
+
+    def _emit(self, texts, n):
+        with self._lock:
+            start = self._offset
+            self._offset += n
+        return [texts[(start + i) % len(texts)] for i in range(n)]
+
+    def _guard(self):
+        with self._lock:
+            self.calls += 1
+            failed = self.calls <= self.fail_times
+        if failed:
+            raise FakeAPIError(503)
+
+    def _chat(self, *, model, messages, n=1, **kw):
+        self._guard()
+        prompt = messages[-1]["content"]
+        return _chat_response(self._emit(self._pick(prompt), n))
+
+    def _completion(self, *, model, prompt, n=1, **kw):
+        self._guard()
+        return _completion_response(self._emit(self._pick(prompt), n))
+
+
+@pytest.fixture
+def fake_client():
+    return FakeClient

leakit-0.1.0/tests/test_cli.py

@@ -0,0 +1,86 @@
+"""End-to-end CLI tests with an injected fake client (no network)."""
+
+import json
+
+import pytest
+
+from leakit import cli
+from leakit.core import LeakIt
+
+
+@pytest.fixture
+def patch_scorer(monkeypatch, fake_client):
+    """Replace cli._make_scorer with one that injects a scripted fake client."""
+    def _factory(args):
+        fc = fake_client(
+            script={
+                "MEMBER": ["born 1809 hardin county"],  # identical -> full concentration
+                "NOVEL": ["sky blue today", "i wonder if", "banana telephone qux"],
+            },
+            default=["a b c", "a b d"],
+        )
+        return LeakIt(
+            model=args.model, n_samples=args.samples, statistic=args.statistic,
+            prefix_chars=args.prefix_chars, mode=args.mode, client=fc, api_key="k",
+        )
+    monkeypatch.setattr(cli, "_make_scorer", _factory)
+
+
+def test_cli_table_output(patch_scorer, tmp_path, capsys):
+    doc = tmp_path / "suspect.txt"
+    doc.write_text("MEMBER text here")
+    rc = cli.main(["--model", "m", str(doc)])
+    out = capsys.readouterr().out
+    assert rc == 0
+    assert "suspect.txt" in out
+    assert "score" in out
+
+
+def test_cli_json_output(patch_scorer, tmp_path, capsys):
+    doc = tmp_path / "d.txt"
+    doc.write_text("MEMBER text")
+    rc = cli.main(["--model", "m", "--json", str(doc)])
+    out = capsys.readouterr().out
+    data = json.loads(out)
+    assert rc == 0
+    assert data[0]["document"].endswith("d.txt")
+    assert data[0]["n_returned"] == 16
+    assert 0.0 <= data[0]["score"] <= 1.0
+
+
+def test_cli_member_scores_above_novel(patch_scorer, tmp_path, capsys):
+    m = tmp_path / "m.txt"; m.write_text("MEMBER passage")
+    nvl = tmp_path / "n.txt"; nvl.write_text("NOVEL passage")
+    cli.main(["--model", "m", "--json", str(m), str(nvl)])
+    data = json.loads(capsys.readouterr().out)
+    by = {d["document"].split("/")[-1]: d["score"] for d in data}
+    assert by["m.txt"] > by["n.txt"]
+
+
+def test_cli_calibrate_percentile(patch_scorer, tmp_path, capsys):
+    suspect = tmp_path / "suspect.txt"; suspect.write_text("MEMBER passage")
+    clean1 = tmp_path / "c1.txt"; clean1.write_text("NOVEL one")
+    clean2 = tmp_path / "c2.txt"; clean2.write_text("NOVEL two")
+    rc = cli.main(["--model", "m", "--json",
+                   "--calibrate", f"{clean1},{clean2}", str(suspect)])
+    data = json.loads(capsys.readouterr().out)
+    assert rc == 0
+    # MEMBER concentrates more than the NOVEL baseline -> high percentile
+    assert data[0]["percentile_vs_baseline"] == 100.0
+
+
+def test_cli_no_documents_errors(monkeypatch, capsys):
+    monkeypatch.setattr("sys.stdin.isatty", lambda: True)
+    rc = cli.main(["--model", "m"])
+    assert rc == 2
+    assert "no documents" in capsys.readouterr().err
+
+
+def test_cli_stdin(patch_scorer, monkeypatch, capsys):
+    import io
+    monkeypatch.setattr("sys.stdin", io.StringIO("MEMBER from stdin"))
+    monkeypatch.setattr("sys.stdin.isatty", lambda: False, raising=False)
+    rc = cli.main(["--model", "m", "--json"])
+    data = json.loads(capsys.readouterr().out)
+    assert rc == 0
+    assert data[0]["document"] == "<stdin>"

leakit-0.1.0/tests/test_integration_http.py

@@ -0,0 +1,102 @@
+"""Live HTTP integration: real openai SDK against a local OpenAI-shaped server.
+
+Proves the actual network path (request shape, base_url routing, response
+parsing) without any paid API. The server returns concentrated continuations
+for a "MEMBER" prefix and diffuse ones for a "NOVEL" prefix.
+"""
+
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+import pytest
+
+from leakit import LeakIt
+
+# Members: identical continuations (model concentrates on memorised text).
+# Novel: four distinct continuations rotated across requests (diffuse distribution).
+MEMBER_CONTINUATIONS = ["in Hardin County Kentucky"]
+NOVEL_CONTINUATIONS = ["a winding road ahead", "thoughts about nothing much",
+                       "quux frobnicate widget", "the seventeenth of never"]
+
+
+class _Handler(BaseHTTPRequestHandler):
+    def log_message(self, *a):  # silence
+        pass
+
+    def _emit(self, pool, n):
+        # Rotate a server-wide cursor so independent requests vary, like a model.
+        srv = self.server
+        with srv._lock:  # type: ignore[attr-defined]
+            start = srv._cursor  # type: ignore[attr-defined]
+            srv._cursor += n  # type: ignore[attr-defined]
+        return [pool[(start + i) % len(pool)] for i in range(n)]
+
+    def do_POST(self):
+        length = int(self.headers.get("Content-Length", 0))
+        body = json.loads(self.rfile.read(length) or b"{}")
+        n = int(body.get("n", 1) or 1)
+
+        if self.path.endswith("/chat/completions"):
+            prompt = body["messages"][-1]["content"]
+            pool = MEMBER_CONTINUATIONS if "MEMBER" in prompt else NOVEL_CONTINUATIONS
+            texts = self._emit(pool, n)
+            choices = [{"index": i, "message": {"role": "assistant",
+                        "content": t}, "finish_reason": "stop"}
+                       for i, t in enumerate(texts)]
+        else:  # /completions
+            prompt = body.get("prompt", "")
+            pool = MEMBER_CONTINUATIONS if "MEMBER" in prompt else NOVEL_CONTINUATIONS
+            texts = self._emit(pool, n)
+            choices = [{"index": i, "text": t, "finish_reason": "stop"}
+                       for i, t in enumerate(texts)]
+
+        payload = json.dumps({
+            "id": "cmpl-test", "object": "chat.completion", "model": body.get("model", "m"),
+            "choices": choices,
+            "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+        }).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(payload)))
+        self.end_headers()
+        self.wfile.write(payload)
+
+
+@pytest.fixture
+def local_openai_server():
+    import threading as _t
+    server = HTTPServer(("127.0.0.1", 0), _Handler)
+    server._cursor = 0
+    server._lock = _t.Lock()
+    port = server.server_address[1]
+    t = threading.Thread(target=server.serve_forever, daemon=True)
+    t.start()
+    yield f"http://127.0.0.1:{port}/v1"
+    server.shutdown()
+
+
+def test_live_http_chat_mode_member_vs_novel(local_openai_server):
+    scorer = LeakIt(model="local-test", base_url=local_openai_server, api_key="test-key",
+                    n_samples=8, concurrency=4, mode="chat")
+    member = scorer.score("MEMBER: the sixteenth president was born", document_id="member")
+    novel = scorer.score("NOVEL: my grocery list for tuesday", document_id="novel")
+    assert member.n_returned == 8
+    assert novel.n_returned == 8
+    assert member.score == 1.0          # identical continuations -> full concentration
+    assert member.score > novel.score
+
+
+def test_live_http_completion_mode(local_openai_server):
+    scorer = LeakIt(model="local-test", base_url=local_openai_server, api_key="test-key",
+                    n_samples=4, concurrency=2, mode="completion")
+    res = scorer.score("MEMBER prefix", document_id="m")
+    assert res.n_returned == 4
+    assert res.score == 1.0
+
+
+def test_live_http_n_per_request_batch(local_openai_server):
+    scorer = LeakIt(model="local-test", base_url=local_openai_server, api_key="test-key",
+                    n_samples=12, n_per_request=4, concurrency=3, mode="chat")
+    res = scorer.score("MEMBER prefix", document_id="m")
+    assert res.n_returned == 12

leakit-0.1.0/tests/test_sampler.py

@@ -0,0 +1,62 @@
+"""Sampler behaviour with an injected fake client (no network)."""
+
+import pytest
+
+from leakit.sampler import Sampler, SamplerConfig, resolve_api_key
+
+
+def _sampler(fake, **overrides):
+    cfg = SamplerConfig(model="m", **overrides)
+    return Sampler(cfg, api_key="k", client=fake)
+
+
+def test_chat_collects_all_samples(fake_client):
+    s = _sampler(fake_client(default=["x y z"]), n_samples=16, concurrency=4)
+    out = s.sample("some prefix")
+    assert len(out) == 16
+    assert all(c == "x y z" for c in out)
+
+
+def test_completion_mode(fake_client):
+    s = _sampler(fake_client(default=["foo bar"]), n_samples=8, mode="completion", concurrency=2)
+    out = s.sample("prefix")
+    assert len(out) == 8
+    assert out[0] == "foo bar"
+
+
+def test_n_per_request_batches(fake_client):
+    fc = fake_client(default=["a b", "c d"])
+    s = _sampler(fc, n_samples=10, n_per_request=5, concurrency=4)
+    out = s.sample("prefix")
+    assert len(out) == 10
+    # 10 samples / 5 per request = 2 calls
+    assert fc.calls == 2
+
+
+def test_retry_then_success(fake_client):
+    fc = fake_client(default=["ok ok"], fail_times=2)
+    s = _sampler(fc, n_samples=1, max_retries=5, concurrency=1)
+    out = s.sample("prefix")
+    assert out == ["ok ok"]
+    assert fc.calls == 3  # 2 failures + 1 success
+
+
+def test_dead_batch_is_skipped_not_fatal(fake_client):
+    # Every call fails; with retries exhausted the run yields zero completions
+    fc = fake_client(default=["never"], fail_times=99)
+    s = _sampler(fc, n_samples=4, max_retries=2, concurrency=2)
+    out = s.sample("prefix")
+    assert out == []
+
+
+def test_resolve_api_key_prefers_named(monkeypatch):
+    monkeypatch.delenv("LEAKIT_API_KEY", raising=False)
+    monkeypatch.setenv("MY_KEY", "secret")
+    assert resolve_api_key("MY_KEY") == "secret"
+
+
+def test_resolve_api_key_missing_raises(monkeypatch):
+    monkeypatch.delenv("LEAKIT_API_KEY", raising=False)
+    monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+    with pytest.raises(RuntimeError):
+        resolve_api_key()

leakit-0.1.0/tests/test_stats.py

@@ -0,0 +1,54 @@
+"""Statistic correctness, cross-checked against the paper's reference behaviour."""
+
+import math
+
+from leakit import _stats
+
+
+def test_identical_completions_score_one():
+    comps = ["the quick brown fox", "the quick brown fox", "the quick brown fox"]
+    assert _stats.self_concentration_word_jaccard(comps) == 1.0
+    assert _stats.self_concentration_kgram(comps, k=3) == 1.0
+
+
+def test_disjoint_completions_score_zero():
+    comps = ["alpha beta gamma", "delta epsilon zeta", "eta theta iota"]
+    assert _stats.self_concentration_word_jaccard(comps) == 0.0
+
+
+def test_word_jaccard_known_value():
+    # {a,b} vs {b,c}: |∩|=1, |∪|=3 -> 1/3 ; single pair so mean == 1/3
+    assert math.isclose(_stats.self_concentration_word_jaccard(["a b", "b c"]), 1 / 3)
+
+
+def test_word_jaccard_mean_over_pairs():
+    # pairs: (1,2)=1/3, (1,3)=0, (2,3)=0  -> mean = 1/9
+    val = _stats.self_concentration_word_jaccard(["a b", "b c", "x y"])
+    assert math.isclose(val, (1 / 3) / 3)
+
+
+def test_fewer_than_two_is_zero():
+    assert _stats.self_concentration_word_jaccard([]) == 0.0
+    assert _stats.self_concentration_word_jaccard(["only one"]) == 0.0
+
+
+def test_empty_strings_do_not_crash():
+    assert _stats.self_concentration_word_jaccard(["", ""]) == 0.0
+    assert _stats.self_concentration_kgram(["", "abc"], k=5) == 0.0
+
+
+def test_member_scores_above_nonmember():
+    member = ["born in 1809 in Hardin County", "born in 1809 in Hardin County, Kentucky",
+              "born in 1809 in Hardin"]
+    nonmember = ["the weather today is", "I think that maybe", "purple monkey dishwasher"]
+    assert (_stats.self_concentration_word_jaccard(member)
+            > _stats.self_concentration_word_jaccard(nonmember))
+
+
+def test_dispatch_unknown_raises():
+    try:
+        _stats.compute(["a", "b"], statistic="nope")
+    except ValueError as e:
+        assert "unknown statistic" in str(e)
+    else:
+        raise AssertionError("expected ValueError")