pairjudge 0.1.0__tar.gz

pairjudge-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Daoyuan Li
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pairjudge-0.1.0/PKG-INFO

@@ -0,0 +1,179 @@
+Metadata-Version: 2.4
+Name: pairjudge
+Version: 0.1.0
+Summary: Train and serve pairwise LLM judges (A/B/tie) with budget-aware multi-turn packing and position-bias correction
+Author-email: Daoyuan Li <lidaoyuan2816@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/DaoyuanLi2816/pairjudge
+Project-URL: Issues, https://github.com/DaoyuanLi2816/pairjudge/issues
+Keywords: llm-as-judge,reward-model,preference-learning,rlhf,chatbot-arena
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: pyyaml
+Provides-Extra: judge
+Requires-Dist: torch; extra == "judge"
+Requires-Dist: transformers>=4.46; extra == "judge"
+Provides-Extra: train
+Requires-Dist: torch; extra == "train"
+Requires-Dist: transformers>=4.46; extra == "train"
+Requires-Dist: peft; extra == "train"
+Requires-Dist: datasets; extra == "train"
+Requires-Dist: accelerate; extra == "train"
+Requires-Dist: scikit-learn; extra == "train"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: torch; extra == "test"
+Requires-Dist: transformers>=4.46; extra == "test"
+Requires-Dist: scikit-learn; extra == "test"
+Dynamic: license-file
+
+# pairjudge
+
+**Train and serve pairwise LLM judges (A wins / B wins / tie) — with budget-aware multi-turn packing, position-bias correction, and pseudo-label distillation.**
+
+[![CI](https://github.com/DaoyuanLi2816/pairjudge/actions/workflows/ci.yml/badge.svg)](https://github.com/DaoyuanLi2816/pairjudge/actions)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+![Python](https://img.shields.io/badge/python-3.9%2B-blue)
+[![Kaggle Gold](https://img.shields.io/badge/Kaggle-Gold%20%C2%B7%204th%20of%201849-FFD700)](https://www.kaggle.com/competitions/lmsys-chatbot-arena/leaderboard)
+
+`pairjudge` is the generalized core of the **4th-place (gold medal) solution** to Kaggle's [LMSYS — Chatbot Arena Human Preference Predictions](https://www.kaggle.com/competitions/lmsys-chatbot-arena/overview) (1,849 teams), extracted into a small, tested library you can run on **your own preference data with any Hugging Face backbone**. The exact competition artifacts are preserved untouched in [`competition/`](competition/README.md), and a golden test pins the library's default behavior to the medal-winning code **byte for byte**.
+
+Use it when you need a model that answers: *given a prompt and two candidate responses, which one would a human prefer — or is it a tie?* That model is the engine behind response reranking, A/B evaluation of fine-tunes, RLHF/RLAIF reward signals, and arena-style leaderboards.
+
+## Why not just an off-the-shelf reward model?
+
+Three problems show up the moment you train a pairwise judge on real conversations, and they are exactly what this library packages:
+
+**1. Truncation silently destroys the comparison.**
+A judge input holds a multi-turn conversation *plus two responses per turn*. With naive left- or right-truncation, long inputs routinely lose response B (or the prompt) entirely — the judge then learns position artifacts instead of preferences. `PairPacker` packs rounds greedily and, when the budget runs out, truncates the final round *proportionally* (default 20% prompt / 40% response A / 40% response B), marks every cut with an explicit ellipsis, and drops rounds that can't be shown honestly. Guarantee: never exceeds `max_length`, and every retained round shows all three fields.
+
+**2. Pairwise judges have position bias.**
+Swap A and B and a naive judge changes its verdict on a measurable fraction of pairs. `PairwiseJudge.predict_proba(swap_debias=True)` scores each pair in both orders and averages in the original frame — order-invariant by construction. `position_flip_rate()` measures how biased your judge is before you decide to pay the 2x compute.
+
+**3. Human preference labels are scarce and noisy.**
+The medal recipe is a two-phase semi-supervised loop: train on human labels → pseudo-label a large unlabeled pool with **full probability distributions** → retrain with soft-label KL distillation (`label_mode: soft`). Ties are a first-class third category throughout — real human preference data is full of them, and scalar Bradley–Terry reward models (e.g. TRL's `RewardTrainer`, `num_labels=1`) cannot represent them.
+
+## Install
+
+```bash
+pip install -e .              # core: packing + data loaders (no torch needed)
+pip install -e .[judge]       # + inference (torch, transformers)
+pip install -e .[train]       # + LoRA fine-tuning (peft, datasets, accelerate)
+```
+
+## 60 seconds
+
+```python
+from pairjudge import PairPacker, PackerConfig, from_pairs
+
+# 1. Pack pairwise conversations into a token budget — any HF tokenizer.
+from transformers import AutoTokenizer
+tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+packer = PairPacker(tok, PackerConfig(max_length=2048))
+packed = packer.pack(
+    prompts=["Explain quantum entanglement to a 10-year-old."],
+    responses_a=["Imagine two magic coins..."],
+    responses_b=["Quantum entanglement is a physical phenomenon..."],
+)
+packed.input_ids      # <= 2048 tokens, prompt + BOTH responses guaranteed visible
+packed.truncated      # False — everything fit
+
+# 2. Judge a pair with a trained model, position-bias-free.
+from pairjudge import PairwiseJudge
+judge = PairwiseJudge.from_pretrained("path/to/your/judge")
+df = from_pairs(
+    prompts=["Explain quantum entanglement to a 10-year-old."],
+    responses_a=["Imagine two magic coins..."],
+    responses_b=["Quantum entanglement is a physical phenomenon..."],
+)
+judge.predict_proba(df, swap_debias=True)   # [[p_a_wins, p_b_wins, p_tie]]
+judge.position_flip_rate(df)                # how order-sensitive is my judge?
+```
+
+## Train your own judge
+
+```bash
+# Small judge on one consumer GPU (Qwen2.5-0.5B, ungated):
+python -m pairjudge.training --cfg examples/configs/quickstart.yaml
+
+# The competition setup (gemma-2-9b-it, 4x A100):
+python -m pairjudge.training --cfg examples/configs/reproduce_competition.yaml
+```
+
+Input is either an Arena-format CSV (the Kaggle competition schema) or a parquet with canonical columns — `prompt` / `response_a` / `response_b` as per-round string lists plus one-hot (or soft) `winner_*` columns. `pairjudge.data` ships loaders for Arena CSVs and UltraFeedback-style chosen/rejected data, plus `from_pairs()` for plain Python lists.
+
+The full two-phase distillation loop:
+
+```bash
+# Phase 1: train on human labels
+python -m pairjudge.training --cfg phase1.yaml                  # label_mode: hard
+
+# Pseudo-label an unlabeled pool with the phase-1 judge (soft labels)
+python -m pairjudge.pseudo_label \
+    --model ./output/judge/merged \
+    --data pool.parquet --out pool_pl.parquet --swap-debias
+
+# Phase 2: retrain from scratch on human + soft labels with KL loss
+python -m pairjudge.training --cfg phase2.yaml                  # label_mode: soft
+```
+
+In the competition, this loop (88k human-labeled + 30k pseudo-labeled UltraFeedback conversations) was a decisive part of the gap between a good model and a gold-medal one.
+
+## Inference guardrails
+
+Two degenerate cases are worth handling outside the model — on competition data this was worth a measurable amount of log-loss:
+
+```python
+from pairjudge import empty_and_identical_masks
+
+a_empty, b_empty, identical = empty_and_identical_masks(raw_df)
+proba[a_empty]  = [0.04, 0.88, 0.08]   # empty response loses — but never bet 1.0
+proba[b_empty]  = [0.88, 0.04, 0.08]   # labels are noisy; log-loss punishes overconfidence
+proba[identical] = [0.06, 0.06, 0.88]  # identical responses are a tie
+```
+
+## How it relates to TRL's `RewardTrainer`
+
+| | TRL `RewardTrainer` | `pairjudge` |
+|---|---|---|
+| Output | scalar reward (`num_labels=1`) | 3-class distribution (A / B / **tie**) |
+| Loss | Bradley–Terry (logsigmoid of reward gap) | CE on human labels, KL on soft pseudo-labels |
+| Ties | not representable | first-class |
+| Multi-turn pair truncation | generic | proportional, all-fields-guaranteed |
+| Position bias | n/a at inference (scores singletons) | swap-debias averaging + flip-rate diagnostic |
+
+If you need a scalar reward for PPO-style RLHF, use TRL. If you need a *judge* that compares two concrete responses — for evaluation, reranking, data labeling, or arena prediction — and your data has ties, this is the recipe that placed 4th of 1,849 on exactly that task.
+
+## Provenance & validation
+
+- The competition scripts, configs, inference notebook and certificate are preserved verbatim in [`competition/`](competition/README.md), including the full original write-up.
+- `tests/test_packing.py::TestCompetitionEquivalence` fuzzes 1,500 conversations against a verbatim copy of the competition tokenizer ([`tests/reference_impl.py`](tests/reference_impl.py)) and asserts byte-identical output with default settings — the library *is* the medal-winning code, not a reimplementation of it.
+- Final leaderboard: **4th / 1,849** ([gold medal](https://www.kaggle.com/certification/competitions/distiller/lmsys-chatbot-arena), $20,000 prize).
+
+## Citation
+
+```bibtex
+@misc{li2024pairjudge,
+  author = {Daoyuan Li},
+  title  = {pairjudge: pairwise LLM judges with budget-aware packing and position-bias correction},
+  year   = {2024},
+  url    = {https://github.com/DaoyuanLi2816/pairjudge},
+  note   = {Generalized from the 4th-place solution, Kaggle LMSYS Chatbot Arena Human Preference Predictions}
+}
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+## Author
+
+Daoyuan Li — [Kaggle (distiller)](https://www.kaggle.com/distiller) · lidaoyuan2816@gmail.com

pairjudge-0.1.0/README.md

@@ -0,0 +1,142 @@
+# pairjudge
+
+**Train and serve pairwise LLM judges (A wins / B wins / tie) — with budget-aware multi-turn packing, position-bias correction, and pseudo-label distillation.**
+
+[![CI](https://github.com/DaoyuanLi2816/pairjudge/actions/workflows/ci.yml/badge.svg)](https://github.com/DaoyuanLi2816/pairjudge/actions)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+![Python](https://img.shields.io/badge/python-3.9%2B-blue)
+[![Kaggle Gold](https://img.shields.io/badge/Kaggle-Gold%20%C2%B7%204th%20of%201849-FFD700)](https://www.kaggle.com/competitions/lmsys-chatbot-arena/leaderboard)
+
+`pairjudge` is the generalized core of the **4th-place (gold medal) solution** to Kaggle's [LMSYS — Chatbot Arena Human Preference Predictions](https://www.kaggle.com/competitions/lmsys-chatbot-arena/overview) (1,849 teams), extracted into a small, tested library you can run on **your own preference data with any Hugging Face backbone**. The exact competition artifacts are preserved untouched in [`competition/`](competition/README.md), and a golden test pins the library's default behavior to the medal-winning code **byte for byte**.
+
+Use it when you need a model that answers: *given a prompt and two candidate responses, which one would a human prefer — or is it a tie?* That model is the engine behind response reranking, A/B evaluation of fine-tunes, RLHF/RLAIF reward signals, and arena-style leaderboards.
+
+## Why not just an off-the-shelf reward model?
+
+Three problems show up the moment you train a pairwise judge on real conversations, and they are exactly what this library packages:
+
+**1. Truncation silently destroys the comparison.**
+A judge input holds a multi-turn conversation *plus two responses per turn*. With naive left- or right-truncation, long inputs routinely lose response B (or the prompt) entirely — the judge then learns position artifacts instead of preferences. `PairPacker` packs rounds greedily and, when the budget runs out, truncates the final round *proportionally* (default 20% prompt / 40% response A / 40% response B), marks every cut with an explicit ellipsis, and drops rounds that can't be shown honestly. Guarantee: never exceeds `max_length`, and every retained round shows all three fields.
+
+**2. Pairwise judges have position bias.**
+Swap A and B and a naive judge changes its verdict on a measurable fraction of pairs. `PairwiseJudge.predict_proba(swap_debias=True)` scores each pair in both orders and averages in the original frame — order-invariant by construction. `position_flip_rate()` measures how biased your judge is before you decide to pay the 2x compute.
+
+**3. Human preference labels are scarce and noisy.**
+The medal recipe is a two-phase semi-supervised loop: train on human labels → pseudo-label a large unlabeled pool with **full probability distributions** → retrain with soft-label KL distillation (`label_mode: soft`). Ties are a first-class third category throughout — real human preference data is full of them, and scalar Bradley–Terry reward models (e.g. TRL's `RewardTrainer`, `num_labels=1`) cannot represent them.
+
+## Install
+
+```bash
+pip install -e .              # core: packing + data loaders (no torch needed)
+pip install -e .[judge]       # + inference (torch, transformers)
+pip install -e .[train]       # + LoRA fine-tuning (peft, datasets, accelerate)
+```
+
+## 60 seconds
+
+```python
+from pairjudge import PairPacker, PackerConfig, from_pairs
+
+# 1. Pack pairwise conversations into a token budget — any HF tokenizer.
+from transformers import AutoTokenizer
+tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+packer = PairPacker(tok, PackerConfig(max_length=2048))
+packed = packer.pack(
+    prompts=["Explain quantum entanglement to a 10-year-old."],
+    responses_a=["Imagine two magic coins..."],
+    responses_b=["Quantum entanglement is a physical phenomenon..."],
+)
+packed.input_ids      # <= 2048 tokens, prompt + BOTH responses guaranteed visible
+packed.truncated      # False — everything fit
+
+# 2. Judge a pair with a trained model, position-bias-free.
+from pairjudge import PairwiseJudge
+judge = PairwiseJudge.from_pretrained("path/to/your/judge")
+df = from_pairs(
+    prompts=["Explain quantum entanglement to a 10-year-old."],
+    responses_a=["Imagine two magic coins..."],
+    responses_b=["Quantum entanglement is a physical phenomenon..."],
+)
+judge.predict_proba(df, swap_debias=True)   # [[p_a_wins, p_b_wins, p_tie]]
+judge.position_flip_rate(df)                # how order-sensitive is my judge?
+```
+
+## Train your own judge
+
+```bash
+# Small judge on one consumer GPU (Qwen2.5-0.5B, ungated):
+python -m pairjudge.training --cfg examples/configs/quickstart.yaml
+
+# The competition setup (gemma-2-9b-it, 4x A100):
+python -m pairjudge.training --cfg examples/configs/reproduce_competition.yaml
+```
+
+Input is either an Arena-format CSV (the Kaggle competition schema) or a parquet with canonical columns — `prompt` / `response_a` / `response_b` as per-round string lists plus one-hot (or soft) `winner_*` columns. `pairjudge.data` ships loaders for Arena CSVs and UltraFeedback-style chosen/rejected data, plus `from_pairs()` for plain Python lists.
+
+The full two-phase distillation loop:
+
+```bash
+# Phase 1: train on human labels
+python -m pairjudge.training --cfg phase1.yaml                  # label_mode: hard
+
+# Pseudo-label an unlabeled pool with the phase-1 judge (soft labels)
+python -m pairjudge.pseudo_label \
+    --model ./output/judge/merged \
+    --data pool.parquet --out pool_pl.parquet --swap-debias
+
+# Phase 2: retrain from scratch on human + soft labels with KL loss
+python -m pairjudge.training --cfg phase2.yaml                  # label_mode: soft
+```
+
+In the competition, this loop (88k human-labeled + 30k pseudo-labeled UltraFeedback conversations) was a decisive part of the gap between a good model and a gold-medal one.
+
+## Inference guardrails
+
+Two degenerate cases are worth handling outside the model — on competition data this was worth a measurable amount of log-loss:
+
+```python
+from pairjudge import empty_and_identical_masks
+
+a_empty, b_empty, identical = empty_and_identical_masks(raw_df)
+proba[a_empty]  = [0.04, 0.88, 0.08]   # empty response loses — but never bet 1.0
+proba[b_empty]  = [0.88, 0.04, 0.08]   # labels are noisy; log-loss punishes overconfidence
+proba[identical] = [0.06, 0.06, 0.88]  # identical responses are a tie
+```
+
+## How it relates to TRL's `RewardTrainer`
+
+| | TRL `RewardTrainer` | `pairjudge` |
+|---|---|---|
+| Output | scalar reward (`num_labels=1`) | 3-class distribution (A / B / **tie**) |
+| Loss | Bradley–Terry (logsigmoid of reward gap) | CE on human labels, KL on soft pseudo-labels |
+| Ties | not representable | first-class |
+| Multi-turn pair truncation | generic | proportional, all-fields-guaranteed |
+| Position bias | n/a at inference (scores singletons) | swap-debias averaging + flip-rate diagnostic |
+
+If you need a scalar reward for PPO-style RLHF, use TRL. If you need a *judge* that compares two concrete responses — for evaluation, reranking, data labeling, or arena prediction — and your data has ties, this is the recipe that placed 4th of 1,849 on exactly that task.
+
+## Provenance & validation
+
+- The competition scripts, configs, inference notebook and certificate are preserved verbatim in [`competition/`](competition/README.md), including the full original write-up.
+- `tests/test_packing.py::TestCompetitionEquivalence` fuzzes 1,500 conversations against a verbatim copy of the competition tokenizer ([`tests/reference_impl.py`](tests/reference_impl.py)) and asserts byte-identical output with default settings — the library *is* the medal-winning code, not a reimplementation of it.
+- Final leaderboard: **4th / 1,849** ([gold medal](https://www.kaggle.com/certification/competitions/distiller/lmsys-chatbot-arena), $20,000 prize).
+
+## Citation
+
+```bibtex
+@misc{li2024pairjudge,
+  author = {Daoyuan Li},
+  title  = {pairjudge: pairwise LLM judges with budget-aware packing and position-bias correction},
+  year   = {2024},
+  url    = {https://github.com/DaoyuanLi2816/pairjudge},
+  note   = {Generalized from the 4th-place solution, Kaggle LMSYS Chatbot Arena Human Preference Predictions}
+}
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+## Author
+
+Daoyuan Li — [Kaggle (distiller)](https://www.kaggle.com/distiller) · lidaoyuan2816@gmail.com

pairjudge-0.1.0/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["setuptools>=64"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "pairjudge"
+version = "0.1.0"
+description = "Train and serve pairwise LLM judges (A/B/tie) with budget-aware multi-turn packing and position-bias correction"
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Daoyuan Li", email = "lidaoyuan2816@gmail.com" }]
+keywords = [
+    "llm-as-judge",
+    "reward-model",
+    "preference-learning",
+    "rlhf",
+    "chatbot-arena",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "numpy",
+    "pandas",
+    "pyyaml",
+]
+
+[project.optional-dependencies]
+# Inference: load a trained judge and predict preferences.
+judge = ["torch", "transformers>=4.46"]
+# Training: fine-tune a judge with LoRA on your own preference data.
+train = [
+    "torch",
+    "transformers>=4.46",
+    "peft",
+    "datasets",
+    "accelerate",
+    "scikit-learn",
+]
+test = ["pytest", "torch", "transformers>=4.46", "scikit-learn"]
+
+[project.urls]
+Homepage = "https://github.com/DaoyuanLi2816/pairjudge"
+Issues = "https://github.com/DaoyuanLi2816/pairjudge/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

pairjudge-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

pairjudge-0.1.0/src/pairjudge/__init__.py

@@ -0,0 +1,53 @@
+"""pairjudge — train and serve pairwise LLM judges (A/B/tie).
+
+Extracted and generalized from the 4th-place (gold medal, 4/1849) solution
+to the Kaggle competition "LMSYS — Chatbot Arena Human Preference
+Predictions". The packing defaults are byte-for-byte equivalent to the
+competition tokenization (golden-tested).
+
+Core pieces:
+
+- :class:`pairjudge.PairPacker` — budget-aware packing of multi-turn
+  (prompt, response A, response B) conversations. No heavy dependencies.
+- :class:`pairjudge.PairwiseJudge` — inference with optional swap-based
+  position-bias correction (requires ``pairjudge[judge]``).
+- :mod:`pairjudge.training` — LoRA fine-tuning with hard (CE) or soft (KL
+  distillation) labels (requires ``pairjudge[train]``).
+- :mod:`pairjudge.data` — loaders normalizing Arena/UltraFeedback-style data
+  to one canonical schema.
+"""
+
+from .data import (
+    EMPTY_RESPONSE_PATTERNS,
+    empty_and_identical_masks,
+    from_pairs,
+    load_arena_csv,
+    load_ultrafeedback,
+)
+from .packing import PackedExample, PackerConfig, PairPacker, hard_label
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "PairPacker",
+    "PackerConfig",
+    "PackedExample",
+    "hard_label",
+    "load_arena_csv",
+    "load_ultrafeedback",
+    "from_pairs",
+    "empty_and_identical_masks",
+    "EMPTY_RESPONSE_PATTERNS",
+    "PairwiseJudge",
+    "swap_average",
+    "__version__",
+]
+
+
+def __getattr__(name):
+    # Lazy import: PairwiseJudge needs torch/transformers, which are optional.
+    if name in ("PairwiseJudge", "swap_average"):
+        from . import judge
+
+        return getattr(judge, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

pairjudge-0.1.0/src/pairjudge/data.py

@@ -0,0 +1,166 @@
+"""Loaders that normalize preference data into one canonical schema.
+
+Canonical columns (one row = one conversation):
+
+- ``id``: str
+- ``prompt``: list[str] — user prompt per round
+- ``response_a`` / ``response_b``: list[str] — candidate responses per round
+- ``winner_model_a`` / ``winner_model_b`` / ``winner_tie``: float — one-hot
+  for human labels, arbitrary distribution for pseudo-labels
+
+Everything downstream (packing, training, judging) consumes this schema, so
+supporting a new dataset means writing one loader function.
+"""
+
+from __future__ import annotations
+
+import json
+import random
+from typing import Iterable, Optional, Sequence
+
+import pandas as pd
+
+#: String encodings of an empty response as they appear in Chatbot Arena
+#: exports. A response equal to any of these is "empty" for labeling and
+#: guardrail purposes.
+EMPTY_RESPONSE_PATTERNS = ('[null]', '[]', '[ ]', '[  ]', '[""]', '["",""]')
+
+_LIST_COLUMNS = ("prompt", "response_a", "response_b")
+_WINNER_COLUMNS = ("winner_model_a", "winner_model_b", "winner_tie")
+
+
+def _is_empty(series: pd.Series) -> pd.Series:
+    return series.isin(EMPTY_RESPONSE_PATTERNS)
+
+
+def load_arena_csv(
+    path_or_df,
+    drop_identical: bool = True,
+    relabel_empty: bool = True,
+) -> pd.DataFrame:
+    """Load a Chatbot-Arena-style CSV (LMSYS competition format).
+
+    Expects JSON-encoded list columns ``prompt``/``response_a``/``response_b``
+    and one-hot winner columns.
+
+    Cleaning rules (from the gold-medal solution):
+
+    - rows where *both* responses are empty are dropped (no signal);
+    - rows where the two responses are byte-identical are dropped when
+      ``drop_identical`` — the label is noise there, and inference handles
+      that case with a guardrail instead (:func:`empty_and_identical_masks`);
+    - rows where exactly one response is empty are relabeled so the non-empty
+      side wins when ``relabel_empty`` — annotators almost always prefer *any*
+      answer over a blank one, and the few contrary labels are noise.
+    """
+    df = path_or_df if isinstance(path_or_df, pd.DataFrame) else pd.read_csv(path_or_df, encoding="utf-8")
+    df = df.copy()
+
+    a_empty, b_empty = _is_empty(df["response_a"]), _is_empty(df["response_b"])
+    df = df[~(a_empty & b_empty)]
+    if drop_identical:
+        df = df[~(df["response_a"] == df["response_b"])]
+
+    if relabel_empty:
+        a_empty, b_empty = _is_empty(df["response_a"]), _is_empty(df["response_b"])
+        df.loc[a_empty, list(_WINNER_COLUMNS)] = [0.0, 1.0, 0.0]
+        df.loc[b_empty, list(_WINNER_COLUMNS)] = [1.0, 0.0, 0.0]
+
+    for col in _LIST_COLUMNS:
+        df[col] = df[col].apply(json.loads)
+
+    df = df.reset_index(drop=True)
+    df["id"] = df["id"].astype(str)
+    return df
+
+
+def load_ultrafeedback(
+    path_or_df,
+    seed: int = 42,
+    dedup_by_prompt: bool = True,
+) -> pd.DataFrame:
+    """Convert an UltraFeedback-style chosen/rejected dataset to the canonical schema.
+
+    Each record has ``prompt`` (str) and ``chosen``/``rejected`` conversations
+    (list of ``{"role", "content"}`` messages, assistant reply at index 1).
+    The chosen/rejected pair is assigned to A/B *uniformly at random* (seeded)
+    so the resulting dataset is free of position bias by construction.
+    """
+    df = path_or_df if isinstance(path_or_df, pd.DataFrame) else pd.read_parquet(path_or_df)
+    rng = random.Random(seed)
+
+    records = []
+    for _, row in df.iterrows():
+        chosen = [row["chosen"][1]["content"]]
+        rejected = [row["rejected"][1]["content"]]
+        if rng.random() > 0.5:
+            response_a, response_b, winner = chosen, rejected, "a"
+        else:
+            response_a, response_b, winner = rejected, chosen, "b"
+        records.append(
+            {
+                "prompt": [row["prompt"]],
+                "response_a": response_a,
+                "response_b": response_b,
+                "winner_model_a": 1.0 if winner == "a" else 0.0,
+                "winner_model_b": 1.0 if winner == "b" else 0.0,
+                "winner_tie": 0.0,
+            }
+        )
+
+    out = pd.DataFrame(records)
+    if dedup_by_prompt:
+        out["_key"] = out["prompt"].apply(lambda x: x[0])
+        out = out.drop_duplicates(subset=["_key"], ignore_index=True)
+        out = out.drop(columns=["_key"])
+    out["id"] = out.index.astype(str)
+    return out
+
+
+def from_pairs(
+    prompts: Sequence[str],
+    responses_a: Sequence[str],
+    responses_b: Sequence[str],
+    winners: Optional[Iterable[str]] = None,
+) -> pd.DataFrame:
+    """Build a canonical dataframe from flat single-turn pairs.
+
+    ``winners`` entries are ``"a"``, ``"b"`` or ``"tie"``; omit for unlabeled
+    data (e.g. inference or pseudo-labeling inputs).
+    """
+    df = pd.DataFrame(
+        {
+            "prompt": [[p] for p in prompts],
+            "response_a": [[r] for r in responses_a],
+            "response_b": [[r] for r in responses_b],
+        }
+    )
+    if winners is not None:
+        winners = list(winners)
+        bad = sorted({w for w in winners} - {"a", "b", "tie"})
+        if bad:
+            raise ValueError(f"winners must be 'a', 'b' or 'tie', got {bad}")
+        df["winner_model_a"] = [1.0 if w == "a" else 0.0 for w in winners]
+        df["winner_model_b"] = [1.0 if w == "b" else 0.0 for w in winners]
+        df["winner_tie"] = [1.0 if w == "tie" else 0.0 for w in winners]
+    df["id"] = df.index.astype(str)
+    return df
+
+
+def empty_and_identical_masks(df: pd.DataFrame):
+    """Inference guardrail masks for degenerate pairs.
+
+    Returns boolean Series ``(a_empty, b_empty, identical)`` over raw (still
+    JSON-encoded) response columns. A judge should not be trusted on these
+    rows: an empty response loses against a non-empty one, and identical
+    responses are a tie. Overriding the model's prediction with fixed,
+    *calibrated* probabilities (not 0/1 — labels are noisy and log-loss
+    punishes overconfidence) is worth a measurable amount of log-loss; the
+    gold-medal solution used ``[0.04, 0.88, 0.08]`` for empty-vs-non-empty
+    and ``[0.06, 0.06, 0.88]`` for identical pairs.
+    """
+    return (
+        _is_empty(df["response_a"]),
+        _is_empty(df["response_b"]),
+        df["response_a"] == df["response_b"],
+    )