polaudit 1.0.0__tar.gz

polaudit-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 The polaudit authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

polaudit-1.0.0/PKG-INFO

@@ -0,0 +1,106 @@
+Metadata-Version: 2.4
+Name: polaudit
+Version: 1.0.0
+Summary: An audit framework for LLM alignment with human subgroups, based on polarization
+License: MIT
+Project-URL: Homepage, https://github.com/Swkratis210204/polalign
+Project-URL: Documentation, https://github.com/Swkratis210204/polalign#readme
+Project-URL: Repository, https://github.com/Swkratis210204/polalign
+Keywords: polarization,LLM alignment,subgroups,annotator disagreement,audit,toxicity,unimodality,perspectivism
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: scipy>=1.9
+Requires-Dist: ndfu<1,>=0.9.3
+Requires-Dist: tqdm
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Provides-Extra: plots
+Requires-Dist: matplotlib; extra == "plots"
+Provides-Extra: openai
+Requires-Dist: openai; extra == "openai"
+Provides-Extra: gemini
+Requires-Dist: google-genai; extra == "gemini"
+Provides-Extra: oci
+Requires-Dist: oci; extra == "oci"
+Provides-Extra: all-models
+Requires-Dist: openai; extra == "all-models"
+Requires-Dist: google-genai; extra == "all-models"
+Requires-Dist: oci; extra == "all-models"
+Dynamic: license-file
+
+# polaudit
+
+**An audit framework for LLM alignment with human subgroups, based on polarization.** Whose perspective does an LLM's label reflect? When people disagree about a text, a language model's single label sides with someone.
+`polaudit` finds the texts where annotators are polarized, finds which annotator characteristic (gender, politics, age, ...) explains the split,
+and measures which side each model takes.
+
+![A model must choose a side on a polarized text](https://raw.githubusercontent.com/Swkratis210204/polalign/main/polaudit/docs/images/polarized_case.png)
+
+## The audit in four steps
+
+1. **Detect polarization.** Each text gets a DFU score (distance from unimodality). A text with DFU above 0 is polarized.
+2. **Attribute it.** A polarized text is attributed to a characteristic when every group of that characteristic is internally unimodal (a-posteriori unimodality, AU).
+   Groups in fewer than 10 attributed texts are removed.
+3. **Ask the models.** Your prompt, your models, optionally with a persona.
+4. **Measure alignment.** Cohen's kappa between the model's labels and each group's position, with bootstrap intervals and a chance check.
+
+![The audit framework](https://raw.githubusercontent.com/Swkratis210204/polalign/main/polaudit/docs/images/framework.png)
+
+## Install
+
+```bash
+pip install polaudit                                  # the AU analysis
+pip install "polaudit[plots,openai,gemini]"           # figures and the model providers you use
+```
+
+## Try it in a minute (no data, no keys)
+
+```python
+from polaudit import run_dataset
+from polaudit.toy import toy_spec, toy_ratings
+
+result = run_dataset(toy_spec(), raw=toy_ratings())
+print(result.describe())
+result.per_dimension        # attributed texts per characteristic
+```
+
+## Your own data, prompt and model
+
+```python
+from polaudit import DatasetSpec, ModelHub, annotate, load_model_inputs, run_dataset, save_au_tables
+
+spec = DatasetSpec(
+    name="mydata",
+    load=lambda root: my_dataframe,        # columns: item_id, text, rating, plus one column per characteristic
+    dimensions={"gender": ("gender", {"m": "man", "f": "woman"}),
+                "politics": ("politics", {"l": "left", "r": "right"})},
+)
+result = run_dataset(spec, raw=my_dataframe, min_au_texts=10)     # min_au_texts=0 keeps all AU texts
+save_au_tables({"mydata": result.au}, "results")
+
+def my_prompt(texts, item):
+    return f"Rate how toxic each {item} is from 1 to 5.\nFormat: score=<n>, label=<toxic/civil>\n\n{texts}"
+
+annotate(ModelHub.from_env()["chatgpt"], load_model_inputs("results/aposteriori_for_models.csv"), "results", prompt=my_prompt, run="baseline")
+```
+
+## Full documentation
+
+This page is the short version. The full documentation lives on GitHub, with nine tutorials (a short document and a runnable notebook each), every parameter, and the code
+that reproduces the paper:
+
+- **[Full README on GitHub](https://github.com/Swkratis210204/polalign#readme)**: overview, repository layout, all tutorials, reproducing the paper
+- [Tutorials: documents](https://github.com/Swkratis210204/polalign/tree/main/polaudit/docs) and [notebooks](https://github.com/Swkratis210204/polalign/tree/main/polaudit/notebooks)
+- [All parameters](https://github.com/Swkratis210204/polalign/blob/main/polaudit/docs/parameters.md)
+- [Reference: API keys, models, prompts, outputs, troubleshooting](https://github.com/Swkratis210204/polalign/blob/main/polaudit/docs/reference.md)
+
+## The paper
+
+This package is the code of *Polarization Auditing Framework for LLMs: a Toxicity Detection Study* (authors anonymous for review) and follows the current version of the paper.

polaudit-1.0.0/README.md

@@ -0,0 +1,69 @@
+# polaudit
+
+**An audit framework for LLM alignment with human subgroups, based on polarization.** Whose perspective does an LLM's label reflect? When people disagree about a text, a language model's single label sides with someone.
+`polaudit` finds the texts where annotators are polarized, finds which annotator characteristic (gender, politics, age, ...) explains the split,
+and measures which side each model takes.
+
+![A model must choose a side on a polarized text](https://raw.githubusercontent.com/Swkratis210204/polalign/main/polaudit/docs/images/polarized_case.png)
+
+## The audit in four steps
+
+1. **Detect polarization.** Each text gets a DFU score (distance from unimodality). A text with DFU above 0 is polarized.
+2. **Attribute it.** A polarized text is attributed to a characteristic when every group of that characteristic is internally unimodal (a-posteriori unimodality, AU).
+   Groups in fewer than 10 attributed texts are removed.
+3. **Ask the models.** Your prompt, your models, optionally with a persona.
+4. **Measure alignment.** Cohen's kappa between the model's labels and each group's position, with bootstrap intervals and a chance check.
+
+![The audit framework](https://raw.githubusercontent.com/Swkratis210204/polalign/main/polaudit/docs/images/framework.png)
+
+## Install
+
+```bash
+pip install polaudit                                  # the AU analysis
+pip install "polaudit[plots,openai,gemini]"           # figures and the model providers you use
+```
+
+## Try it in a minute (no data, no keys)
+
+```python
+from polaudit import run_dataset
+from polaudit.toy import toy_spec, toy_ratings
+
+result = run_dataset(toy_spec(), raw=toy_ratings())
+print(result.describe())
+result.per_dimension        # attributed texts per characteristic
+```
+
+## Your own data, prompt and model
+
+```python
+from polaudit import DatasetSpec, ModelHub, annotate, load_model_inputs, run_dataset, save_au_tables
+
+spec = DatasetSpec(
+    name="mydata",
+    load=lambda root: my_dataframe,        # columns: item_id, text, rating, plus one column per characteristic
+    dimensions={"gender": ("gender", {"m": "man", "f": "woman"}),
+                "politics": ("politics", {"l": "left", "r": "right"})},
+)
+result = run_dataset(spec, raw=my_dataframe, min_au_texts=10)     # min_au_texts=0 keeps all AU texts
+save_au_tables({"mydata": result.au}, "results")
+
+def my_prompt(texts, item):
+    return f"Rate how toxic each {item} is from 1 to 5.\nFormat: score=<n>, label=<toxic/civil>\n\n{texts}"
+
+annotate(ModelHub.from_env()["chatgpt"], load_model_inputs("results/aposteriori_for_models.csv"), "results", prompt=my_prompt, run="baseline")
+```
+
+## Full documentation
+
+This page is the short version. The full documentation lives on GitHub, with nine tutorials (a short document and a runnable notebook each), every parameter, and the code
+that reproduces the paper:
+
+- **[Full README on GitHub](https://github.com/Swkratis210204/polalign#readme)**: overview, repository layout, all tutorials, reproducing the paper
+- [Tutorials: documents](https://github.com/Swkratis210204/polalign/tree/main/polaudit/docs) and [notebooks](https://github.com/Swkratis210204/polalign/tree/main/polaudit/notebooks)
+- [All parameters](https://github.com/Swkratis210204/polalign/blob/main/polaudit/docs/parameters.md)
+- [Reference: API keys, models, prompts, outputs, troubleshooting](https://github.com/Swkratis210204/polalign/blob/main/polaudit/docs/reference.md)
+
+## The paper
+
+This package is the code of *Polarization Auditing Framework for LLMs: a Toxicity Detection Study* (authors anonymous for review) and follows the current version of the paper.

polaudit-1.0.0/pyproject.toml

@@ -0,0 +1,44 @@
+[build-system]
+requires = ["setuptools>=64"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "polaudit"
+version = "1.0.0"
+readme = "README.md"
+license = {text = "MIT"}
+classifiers = [
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Intended Audience :: Science/Research",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+description = "An audit framework for LLM alignment with human subgroups, based on polarization"
+requires-python = ">=3.9"
+keywords = ["polarization", "LLM alignment", "subgroups", "annotator disagreement", "audit", "toxicity", "unimodality", "perspectivism"]
+dependencies = [
+    "numpy",
+    "pandas",
+    "scipy>=1.9",
+    "ndfu>=0.9.3,<1",
+    "tqdm",
+]
+
+[project.optional-dependencies]
+test = ["pytest"]
+plots = ["matplotlib"]         # the figures of eda_figures / save_eda
+openai = ["openai"]            # ChatGPT, Mistral, DeepSeek, Ollama (Llama)
+gemini = ["google-genai"]
+oci = ["oci"]                  # Grok on Oracle OCI
+all-models = ["openai", "google-genai", "oci"]
+
+[project.urls]
+Homepage = "https://github.com/Swkratis210204/polalign"
+Documentation = "https://github.com/Swkratis210204/polalign#readme"
+Repository = "https://github.com/Swkratis210204/polalign"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

polaudit-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

polaudit-1.0.0/src/polaudit/__init__.py

@@ -0,0 +1,39 @@
+"""A-posteriori unimodality (AU) of annotator subgroups, and the language models used to audit it."""
+from .agreement import cohen_kappa, spearman, weighted_kappa
+from .annotation import (AnnotationError, annotate, collect_runs, ensemble, inspect_run, load_model_inputs, progress,
+                         sample_inputs)
+from .datasets import SPECS, get_spec
+from .eda import eda_figures, eda_tables, save_eda
+from .eda_dataset import dataset_tables, eda_overview_all, save_dataset_eda
+from .attribution_bootstrap import attribution_gap, attribution_table, bootstrap_gap
+from .kappa_groups import compute_kappa, save_kappa
+from .prompt_sensitivity import compare_prompts
+from .persona_kappa import compute_persona_kappa, save_persona_kappa
+from .shuffling_robustness import compute_null, save_null
+from .poles import compute_poles, save_poles
+from .model_comparison import compare_models
+from .model_eda import label_share_figure, model_eda, save_model_eda, score_histogram_figure
+from .env import Env
+from .export import csv_ready, sample_per_dimension, save_au_tables
+from .models import DEFAULT_CONFIGS, Model, ModelConfig, ModelError, ModelHub
+from .persona_annotation import (annotate_personas, collect_persona_runs, group_order, inspect_personas, load_persona_inputs,
+                                 persona_long)
+from .pipeline import DatasetResult, clean, load_results, make_tool, run_dataset
+from .prompts import PersonaPrompt, ScoringPrompt, label_from_score, save_prompts
+from .sampling import cover_per_dimension, minimal_cover
+from .scoring import score_personas, score_texts
+from .spec import DatasetSpec
+from .unimodality import PrunedAU, Unimodality, dfu_of_hist
+from .utils import find_root, read_au_csv
+
+__version__ = "1.0.0"
+__all__ = ["Unimodality", "PrunedAU", "dfu_of_hist",
+           "DatasetSpec", "SPECS", "get_spec", "run_dataset", "DatasetResult", "clean", "make_tool",
+           "save_au_tables", "csv_ready", "sample_per_dimension", "minimal_cover", "cover_per_dimension", "find_root", "read_au_csv",
+           "Env", "Model", "ModelConfig", "ModelHub", "ModelError", "DEFAULT_CONFIGS",
+           "ScoringPrompt", "PersonaPrompt", "label_from_score", "score_texts", "score_personas",
+           "annotate", "ensemble", "load_model_inputs", "progress", "inspect_run", "collect_runs", "sample_inputs",
+           "AnnotationError", "cohen_kappa", "weighted_kappa", "spearman",
+           "eda_tables", "eda_figures", "save_eda", "dataset_tables", "save_dataset_eda", "eda_overview_all",
+           "attribution_gap", "attribution_table", "bootstrap_gap", "load_results", "compute_kappa", "save_kappa", "compute_persona_kappa", "save_persona_kappa", "save_prompts",
+           "compute_null", "compare_prompts", "compare_models", "model_eda", "save_model_eda", "score_histogram_figure", "label_share_figure", "save_null", "compute_poles", "save_poles", "annotate_personas", "load_persona_inputs", "persona_long", "collect_persona_runs", "inspect_personas", "group_order"]

polaudit-1.0.0/src/polaudit/agreement.py

@@ -0,0 +1,61 @@
+"""How much two sets of ratings of the same texts agree: two prompts, two models, a model and a group of people.
+
+``cohen_kappa`` is for labels (civil / toxic), ``weighted_kappa`` and ``spearman`` for scores. For all of them 1 means perfect
+agreement, 0 means no more agreement than chance would give, and a negative value means less. A pair where one side is missing
+(NaN) is left out. A value that cannot be computed (everything rated the same by both sides, no pairs left) is NaN.
+"""
+from __future__ import annotations
+
+from typing import Iterable, Sequence
+
+import numpy as np
+import pandas as pd
+
+__all__ = ["cohen_kappa", "weighted_kappa", "spearman"]
+
+
+def _pairs(a: Iterable, b: Iterable):
+    a, b = np.asarray(list(a), dtype=object), np.asarray(list(b), dtype=object)
+    if len(a) != len(b):
+        raise ValueError(f"the two sets of ratings must have the same length ({len(a)} and {len(b)})")
+    keep = ~(pd.isna(a) | pd.isna(b))
+    return a[keep], b[keep]
+
+
+def cohen_kappa(a: Iterable, b: Iterable) -> float:
+    """Cohen's kappa of two sets of labels (any values that can be compared, e.g. ``"toxic"``/``"civil"`` or True/False)."""
+    a, b = _pairs(a, b)
+    if len(a) == 0:
+        return float("nan")
+    chance = sum(float(np.mean(a == c)) * float(np.mean(b == c)) for c in set(a) | set(b))
+    return float("nan") if chance == 1 else (float(np.mean(a == b)) - chance) / (1 - chance)
+
+
+def weighted_kappa(a: Iterable, b: Iterable, scale: Sequence[int] = range(1, 6)) -> float:
+    """Quadratic weighted kappa of two sets of scores on ``scale`` (default 1-5): a disagreement of 2 points counts four times as much as
+    one of 1 point. A score outside ``scale`` raises ``ValueError``."""
+    a, b = _pairs(a, b)
+    scale = list(scale)
+    if len(a) == 0 or len(scale) < 2:
+        return float("nan")
+    position = {int(v): i for i, v in enumerate(scale)}
+    try:
+        seen = np.zeros((len(scale), len(scale)))
+        for x, y in zip(a, b):
+            seen[position[int(x)], position[int(y)]] += 1
+    except KeyError as missing:
+        raise ValueError(f"score {missing} is not on the scale {scale[0]}-{scale[-1]}") from None
+    expected = np.outer(seen.sum(axis=1), seen.sum(axis=0)) / seen.sum()
+    steps = np.arange(len(scale))
+    weight = (steps[:, None] - steps[None, :]) ** 2 / (len(scale) - 1) ** 2
+    spread = float((weight * expected).sum())
+    return float("nan") if spread == 0 else 1 - float((weight * seen).sum()) / spread
+
+
+def spearman(a: Iterable, b: Iterable) -> float:
+    """Spearman rank correlation of two sets of scores: do they put the texts in the same order?"""
+    a, b = _pairs(a, b)
+    if len(a) < 2:
+        return float("nan")
+    ra, rb = pd.Series(a.astype(float)).rank(), pd.Series(b.astype(float)).rank()
+    return float("nan") if ra.nunique() < 2 or rb.nunique() < 2 else float(ra.corr(rb))      # one side never changes: undefined

polaudit-1.0.0/src/polaudit/annotation.py

@@ -0,0 +1,218 @@
+"""Run ONE model over the texts of the datasets and save every reply (the baseline: no persona).
+
+``annotate(model, inputs, out_dir, datasets=[...])`` is called once per model (and, if wanted, once per dataset), from the
+notebook, so the order is controlled there and a problem shows up where it happens. Everything of a dataset is kept in its
+own folder, ``out_dir/<dataset>/<run>/``::
+
+    <model>.csv         one row per text: dataset, item_id, text, model, run, batch, score, label   (one file per model)
+    all_models.csv      the models side by side, one row per text: <model>_score, <model>_label for every model run so far
+    raw/<model>.raw.jsonl   one line per batch: the prompt, the reply, the parsed scores (the source of truth)
+
+and one line per (model, dataset) pass is added to ``out_dir/run_log.csv``. ``all_models.csv`` is refreshed after every pass.
+If a run stops (an API error, a reply that cannot be parsed) everything finished so far is kept, and calling ``annotate``
+again carries on at the first missing batch.
+"""
+from __future__ import annotations
+
+import json
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Callable, Dict, Iterable, List, Optional
+
+import pandas as pd
+from tqdm.auto import tqdm
+
+from .datasets import SPECS
+from .models import DEFAULT_CONFIGS, Model
+from .prompts import ScoringPrompt
+
+__all__ = ["annotate", "ensemble", "load_model_inputs", "progress", "inspect_run", "collect_runs", "sample_inputs", "AnnotationError"]
+
+ENSEMBLE_FILE = "all_models.csv"
+MODEL_ORDER = list(DEFAULT_CONFIGS)                  # the order of the models in all_models.csv; other names follow alphabetically
+
+LOG_COLUMNS = ["time_utc", "run", "dataset", "model", "provider", "model_id", "temperature", "max_tokens", "prompt_version",
+               "item_word", "batch_size", "texts", "batches", "batches_resumed", "model_calls", "seconds", "status"]
+
+
+class AnnotationError(RuntimeError):
+    """A batch could not be annotated; what was finished before it is saved."""
+
+
+def load_model_inputs(path: Path) -> Dict[str, pd.DataFrame]:
+    """The texts to annotate, per dataset: each text once, in the order of the file (``item_id``, ``text``)."""
+    table = pd.read_csv(path, dtype={"item_id": str})
+    return {dataset: part.drop_duplicates("item_id")[["item_id", "text"]].reset_index(drop=True)
+            for dataset, part in table.groupby("dataset", sort=False)}
+
+
+def _paths(out_dir: Path, dataset: str, run: str, model: str):
+    folder = Path(out_dir) / dataset / run
+    return folder, folder / "raw" / f"{model}.raw.jsonl", folder / f"{model}.csv"
+
+
+def ensemble(out_dir: Path, dataset: str, run: str = "baseline", models: Optional[Iterable[str]] = None) -> pd.DataFrame:
+    """The models of one dataset side by side: one row per text, ``<model>_score`` and ``<model>_label`` for each model that has
+    a saved file (or for ``models``). Written to ``<dataset>/<run>/all_models.csv`` and returned."""
+    folder = Path(out_dir) / dataset / run
+    found = [p.stem for p in folder.glob("*.csv") if p.name != ENSEMBLE_FILE]
+    names = list(models) if models is not None else sorted(found, key=lambda m: (MODEL_ORDER.index(m) if m in MODEL_ORDER else len(MODEL_ORDER), m))
+    merged = None
+    for name in names:
+        part = pd.read_csv(folder / f"{name}.csv", dtype={"item_id": str})
+        part = part[["dataset", "item_id", "text", "score", "label"]].rename(columns={"score": f"{name}_score", "label": f"{name}_label"})
+        merged = part if merged is None else merged.merge(part.drop(columns=["dataset", "text"]), on="item_id", how="outer")
+    if merged is None:
+        return pd.DataFrame()
+    merged.to_csv(folder / ENSEMBLE_FILE, index=False, encoding="utf-8")
+    return merged
+
+
+def _read_batches(raw_path: Path) -> Dict[int, dict]:
+    done: Dict[int, dict] = {}
+    if raw_path.exists():
+        for line in raw_path.read_text(encoding="utf-8").splitlines():
+            record = json.loads(line)
+            if record["status"] == "ok":
+                done[record["batch"]] = record
+    return done
+
+
+def _write_csv(dataset: str, run: str, model: str, done: Dict[int, dict], csv_path: Path) -> None:
+    """The csv is rebuilt from the saved batches only (each keeps its own texts), never from the current input."""
+    rows = [{"dataset": dataset, "item_id": item_id, "text": text, "model": model, "run": run,
+             "score": parsed["score"], "label": parsed["label"]}
+            for _, record in sorted(done.items())
+            for item_id, text, parsed in zip(record["item_ids"], record["texts"], record["parsed"])]
+    pd.DataFrame(rows, columns=["dataset", "item_id", "text", "model", "run", "score", "label"]).to_csv(csv_path, index=False, encoding="utf-8")
+
+
+def _log(out_dir: Path, **fields) -> None:
+    path = Path(out_dir) / "run_log.csv"
+    line = pd.DataFrame([{c: fields.get(c) for c in LOG_COLUMNS}], columns=LOG_COLUMNS)
+    line.to_csv(path, mode="a", header=not path.exists(), index=False, encoding="utf-8")
+
+
+def annotate(model: Model, inputs: Dict[str, pd.DataFrame], out_dir: Path, prompt: Callable[[str, str], str], run: str = "baseline",
+             batch_size: int = 10, parse_retries: int = 2, limit: Optional[int] = None, datasets: Optional[Iterable[str]] = None,
+             item_words: Optional[Dict[str, str]] = None, scale_max: int = 5, toxic_threshold: float = 3,
+             show_progress: bool = True) -> pd.DataFrame:
+    """Score (1-5) and label every text of every dataset with one model, batch by batch. Returns one row per dataset.
+
+    ``limit`` keeps only the first texts of each dataset (a pilot). ``item_words`` overrides the word used for a text in the
+    prompt (default: the word of the dataset, e.g. "comment"). A batch whose reply does not have exactly one score per text is
+    asked again ``parse_retries`` times and then raises ``AnnotationError`` (the reply is kept in the raw file).
+    ``prompt`` is your function ``prompt(texts, item) -> str``: ``texts`` is the batch already numbered ("1. ...\n2. ..."), ``item``
+    the word for a text, and the string it returns is sent to the model. The reply must have a line ``score=<n>, label=<toxic/civil>``
+    per text. Use another ``run`` name when you change the prompt. ``show_progress`` shows a bar per dataset and model (texts done,
+    time elapsed and left, and the batch being asked).
+    """
+    if not callable(prompt):
+        raise TypeError("prompt must be a function prompt(texts, item) -> str")
+    words = {name: spec.item_word for name, spec in SPECS.items()}
+    words.update(item_words or {})
+    summary = []
+    for dataset in (list(datasets) if datasets is not None else list(inputs)):
+        table = inputs[dataset].head(limit) if limit else inputs[dataset]
+        table = table.reset_index(drop=True)
+        scoring = ScoringPrompt(item=words.get(dataset, "text"), scale_max=scale_max, toxic_threshold=toxic_threshold, write=prompt)
+        folder, raw_path, csv_path = _paths(out_dir, dataset, run, model.name)
+        raw_path.parent.mkdir(parents=True, exist_ok=True)
+        done = _read_batches(raw_path)
+        batches = [table.iloc[i:i + batch_size] for i in range(0, len(table), batch_size)]
+        started, calls, resumed, status = time.time(), 0, 0, "complete"
+        bar = tqdm(total=len(table), desc=f"{dataset} / {model.name}", unit="text", disable=not show_progress)
+        try:
+            for number, batch in enumerate(batches, start=1):
+                ids = batch["item_id"].tolist()
+                if number in done:
+                    if done[number]["item_ids"] != ids:
+                        raise AnnotationError(f"{model.name}/{dataset}: saved batch {number} holds other texts than the input; use another run name")
+                    resumed += 1
+                    bar.update(len(batch))
+                    continue
+                first = str(batch["text"].iloc[0]).replace("\n", " ")
+                bar.set_postfix_str(f"batch {number}/{len(batches)}: {first[:40]}...", refresh=True)
+                question = scoring.build(batch["text"].tolist())
+                for attempt in range(1, parse_retries + 2):
+                    reply = model.complete(question)
+                    calls += 1
+                    parsed = scoring.parse(reply)
+                    if len(parsed) == len(batch):
+                        break
+                else:
+                    with open(raw_path, "a", encoding="utf-8") as f:
+                        f.write(json.dumps({"batch": number, "item_ids": ids, "prompt": question, "reply": reply, "attempts": attempt,
+                                            "status": "unparseable"}, ensure_ascii=False) + "\n")
+                    raise AnnotationError(f"{model.name}/{dataset}: batch {number}: expected {len(batch)} scores, received {len(parsed)} "
+                                          f"after {attempt} attempts; the reply is in {raw_path.name}")
+                record = {"batch": number, "item_ids": ids, "texts": batch["text"].tolist(), "prompt": question, "reply": reply,
+                          "parsed": parsed, "attempts": attempt,
+                          "status": "ok", "time_utc": datetime.now(timezone.utc).isoformat(timespec="seconds")}
+                with open(raw_path, "a", encoding="utf-8") as f:
+                    f.write(json.dumps(record, ensure_ascii=False) + "\n")
+                done[number] = record
+                bar.update(len(batch))
+        except Exception as exc:
+            status = f"failed: {type(exc).__name__}: {exc}"[:300]
+            raise
+        finally:
+            bar.close()
+            _write_csv(dataset, run, model.name, done, csv_path)
+            try:
+                ensemble(out_dir, dataset, run)                          # all_models.csv follows the model that was just run
+            except Exception as exc:                                     # never hide the error that stopped the run
+                print(f"warning: all_models.csv of {dataset}/{run} could not be refreshed: {exc!r}")
+            cfg = model.config
+            _log(out_dir, time_utc=datetime.now(timezone.utc).isoformat(timespec="seconds"), run=run, dataset=dataset, model=model.name,
+                 provider=cfg.provider, model_id=cfg.model_id, temperature=cfg.temperature, max_tokens=cfg.max_tokens,
+                 prompt_version=getattr(prompt, "__name__", "custom"), item_word=words.get(dataset, "text"), batch_size=batch_size, texts=len(table),
+                 batches=len(batches), batches_resumed=resumed, model_calls=calls, seconds=round(time.time() - started, 1), status=status)
+        summary.append({"dataset": dataset, "texts": len(table), "batches": len(batches), "resumed": resumed, "model calls": calls,
+                        "seconds": round(time.time() - started, 1)})
+    return pd.DataFrame(summary).set_index("dataset")
+
+
+def progress(inputs: Dict[str, pd.DataFrame], out_dir: Path, models: Iterable[str], run: str = "baseline") -> pd.DataFrame:
+    """For each dataset and model: how many of the texts have a saved score."""
+    rows = {}
+    for dataset, table in inputs.items():
+        row = {"texts": len(table)}
+        for name in models:
+            _, raw_path, _ = _paths(out_dir, dataset, run, name)
+            row[name] = sum(len(r["item_ids"]) for r in _read_batches(raw_path).values())
+        rows[dataset] = row
+    return pd.DataFrame(rows).T
+
+
+def inspect_run(out_dir: Path, model: str, run: str = "baseline") -> pd.DataFrame:
+    """A first look at what one model returned, per dataset: how many texts, the scores it gave, the share it called toxic, and
+    whether any batch had to be asked again or could not be parsed. Meant for checking a model before running the next one."""
+    rows = {}
+    for csv_path in sorted(Path(out_dir).glob(f"*/{run}/{model}.csv")):
+        table = pd.read_csv(csv_path)
+        raw_path = csv_path.parent / "raw" / f"{model}.raw.jsonl"
+        records = [json.loads(line) for line in raw_path.read_text(encoding="utf-8").splitlines()] if raw_path.exists() else []
+        row = {"texts": len(table), "mean score": round(float(table["score"].mean()), 2) if len(table) else None,
+               "toxic %": round(100 * float((table["label"] == "toxic").mean())) if len(table) else None}
+        row.update({f"score {s}": int((table["score"] == s).sum()) for s in range(1, 6)})
+        row["batches asked again"] = sum(1 for r in records if r["status"] == "ok" and r["attempts"] > 1)
+        row["batches not parsed"] = sum(1 for r in records if r["status"] == "unparseable")
+        rows[csv_path.parent.parent.name] = row
+    return pd.DataFrame(rows).T if rows else pd.DataFrame()
+
+
+def sample_inputs(inputs: Dict[str, pd.DataFrame], n: int, seed: int = 42) -> Dict[str, pd.DataFrame]:
+    """``n`` random texts of every dataset (all of them when a dataset has fewer), in the order of the input. The same ``n`` and
+    ``seed`` always give the same texts, so every prompt and every model sees the same subset."""
+    return {name: table.sample(n=min(n, len(table)), random_state=seed).sort_index().reset_index(drop=True)
+            for name, table in inputs.items()}
+
+
+def collect_runs(out_dir: Path, run: str = "baseline") -> pd.DataFrame:
+    """All saved scores of a run: one row per dataset, model and text."""
+    files = sorted(p for p in Path(out_dir).glob(f"*/{run}/*.csv") if p.name != ENSEMBLE_FILE)
+    if not files:
+        return pd.DataFrame(columns=["dataset", "item_id", "text", "model", "run", "score", "label"])
+    return pd.concat([pd.read_csv(f, dtype={"item_id": str}) for f in files], ignore_index=True)