tribunal-kit 9.2.1 → 9.2.4

package/skills/train-sentence-transformers/references/base_model_selection.md

@@ -0,0 +1,79 @@
+# Base Model Selection
+
+Leaderboards rotate every few months. Don't trust any hardcoded "best" pick. Discover current options live. Run **both** sort orders since most-downloaded surfaces proven options and trending surfaces recent SOTA that may not have download volume yet.
+
+## Discovery commands
+
+**[BI]**:
+```bash
+hf models list --filter sentence-transformers --sort downloads --limit 20
+hf models list --filter sentence-transformers --sort trending  --limit 20
+```
+
+**[CE]**:
+```bash
+hf models list --filter sentence-transformers --filter text-ranking --sort downloads --limit 20
+hf models list --filter sentence-transformers --filter text-ranking --sort trending  --limit 20
+```
+
+**[SPARSE]**:
+```bash
+hf models list --filter sentence-transformers --filter sparse-encoder --sort downloads --limit 20
+hf models list --filter sentence-transformers --filter sparse-encoder --sort trending  --limit 20
+```
+
+Optional language narrowing (any type): add `--filter <language-code>`. Not all multilingual models tag each language, so missing matches doesn't mean the model can't handle that language. Re-run without the filter to compare.
+
+```bash
+hf models card <id> --text                        # confirm dimensions, max_seq_length, license, languages
+```
+
+Cross-check the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard) (pick the relevant tab) before committing to a multi-hour run.
+
+## [BI] Bi-Encoder
+
+Continue from an existing retriever beats fresh-start + 100k-500k pairs. Common namespaces as of 2026-Q2 (verify against discovery commands, since the field rotates):
+
+- **English encoder retrievers**: `sentence-transformers/all-*` (MiniLM-L6-v2, mpnet-base-v2 still the most-downloaded models on the Hub), `BAAI/bge-*-en-v1.5`, `nomic-ai/nomic-embed-text-v1.5`, `mixedbread-ai/mxbai-embed-large-v1`, `Alibaba-NLP/gte-*`, `Snowflake/snowflake-arctic-embed-*`, `jinaai/jina-embeddings-v5-text-small` / `-nano`, `microsoft/harrier-oss-v1-270m` / `-0.6b`.
+- **Multilingual encoder retrievers**: `sentence-transformers/paraphrase-multilingual-*`, `intfloat/multilingual-e5-*`, `ibm-granite/granite-embedding-*-multilingual-r2`, `google/embeddinggemma-300m`, `voyageai/voyage-4-nano`.
+- **Long documents (8k+)**: `nomic-ai/modernbert-embed-*`, `answerdotai/ModernBERT-large`.
+- **Decoder LLM retrievers** (multilingual, **need last-token pooling**): `Qwen/Qwen3-Embedding-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Embedding-*` (multimodal), `codefuse-ai/F2LLM-v2-*`.
+- **Fresh-start English** (≥500k pairs + domain-fit reason): `microsoft/mpnet-base`, `answerdotai/ModernBERT-base`, `google-bert/bert-base-uncased`, `jhu-clsp/ettin-encoder-*` (17m / 32m / 68m / 150m / 400m / 1b, paired ModernBERT encoder family).
+- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (MLM-only, needs contrastive training), `microsoft/mdeberta-v3-base`, `jhu-clsp/mmBERT-base` / `-small`.
+- **CPU / small footprint** (`StaticEmbedding`): `StaticEmbedding(tokenizer, embedding_dim=...)`. **Model size = `vocab_size × dim × 4 bytes`**. Pick a small-vocab tokenizer or you get a giant model: 30k-vocab `bert-base-uncased` × 128 dim ≈ 15 MB, **250k-vocab `paraphrase-multilingual-MiniLM-L12-v2` × 256 dim ≈ 256 MB**. Random init needs 1M+ pairs. Warm-start (`StaticEmbedding.from_distillation(...)`) helps under ~100k pairs.
+
+Architecture variants (encoder / decoder / static / Router), pooling rules, and decoder-vs-encoder setup paths: `model_architectures.md`.
+
+**ModernBERT-family bases default to `max_seq_length=8192`.** That allocates activation memory for 8192-token sequences regardless of your data length and silently drives Windows VRAM into "shared memory" spillover. After loading any ModernBERT / mmBERT / Ettin / gte-modernbert / nomic-modernbert base, **explicitly set `model.max_seq_length = 256` (or 512 for documents)** unless you actually need long context.
+
+## [CE] Cross-Encoder
+
+Continue from an existing reranker beats fresh-start + 100k-500k pairs in most domains. Default to this unless you have a strong reason otherwise. Common namespaces as of 2026-Q2:
+
+- **English encoder rerankers**: `cross-encoder/ms-marco-*`, `BAAI/bge-reranker-*`, `mixedbread-ai/mxbai-rerank-*-v1` / `-v2`, `Alibaba-NLP/gte-reranker-modernbert-*`, `ibm-granite/granite-embedding-reranker-english-*`.
+- **Multilingual encoder rerankers**: `cross-encoder/mmarco-*`, `BAAI/bge-reranker-v2-m3`, `Alibaba-NLP/gte-multilingual-reranker-*`, `ibm-granite/granite-embedding-reranker-multilingual-*`.
+- **Decoder LLM rerankers** (multilingual, `num_labels=1` last-token-style scoring): `Qwen/Qwen3-Reranker-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Reranker-*` (multimodal).
+- **Fresh-start**: `microsoft/MiniLM-L12-H384-uncased`, `answerdotai/ModernBERT-base` / `-large`, `jhu-clsp/ettin-encoder-*`, `FacebookAI/xlm-roberta-base` (multilingual), `microsoft/mdeberta-v3-base` (multilingual), `jhu-clsp/mmBERT-base` / `-small` (multilingual). Pass `num_labels >= 2` for classification cross-encoders.
+
+Encoder-only bases are still the latency-efficient default (bidirectional attention is well-suited to the reranking use case at small parameter counts), but decoder LLM rerankers are now competitive at the top of MTEB Reranking when latency / memory budget allows.
+
+**Minimum dataset:** 500k+ labeled `(query, passage, label)` tuples for production. 10k-100k labeled pairs for continue-training on domain data. Low-resource languages may have less than 10k labeled pairs. In that case, lean on a multilingual base's pretraining and accept a noisier signal.
+
+**"Small" multilingual is ~100M+ params**, not 17M-50M like the English small models. mMiniLMv2-L12-H384 (~117M) is roughly the small-end for usable multilingual rerankers.
+
+## [SPARSE] Sparse Encoder (SPLADE)
+
+SPLADE requires a fill-mask / `AutoModelForMaskedLM`-compatible checkpoint. Encoder-only MLM models work out of the box. **Decoder LLMs do not**.
+
+- **Continue from existing SPLADE (English)**: `naver/splade-*` (the canonical family), `opensearch-project/opensearch-neural-sparse-encoding-*` (incl. `-doc-v2-distill`, `-doc-v3-distill` / `-doc-v3-gte`), `prithivida/Splade_PP_en_v*`, `ibm-granite/granite-embedding-30m-sparse`.
+- **Continue from existing SPLADE (multilingual)**: `opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1`.
+- **Fresh-start English** (≥500k pairs): any encoder with an MLM head (`distilbert/distilbert-base-uncased`, `google-bert/bert-base-uncased`). Pure `AutoModel` checkpoints without MLM won't work. Discover MLM bases: `hf models list --filter fill-mask --sort downloads --limit 20`.
+- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (has MLM head). For other multilingual MLM bases: add `--filter <language-code>`.
+
+**Minimum dataset:** 500k+ triplets (with mined hard negatives) for a competitive SPLADE. 50k+ triplets for domain adaptation on existing SPLADE.
+
+## Cross-cutting tips
+
+- **Non-English retrieval starting points** (when language tag returns 0 results): check `intfloat/multilingual_e5_train_data` for parallel pair data, MIRACL via the `sentence-transformers/miracl` mirror for multilingual retrieval, mMARCO via `unicamp-dl/mmarco` (14 languages, parquet-backed).
+- **Avoid script-based dataset loaders.** `datasets >= 4` rejects them with `RuntimeError: Dataset scripts are no longer supported`. Look for parquet-backed mirrors (e.g. `sentence-transformers/miracl` instead of `miracl/miracl`).
+- **`hf datasets sql` requires DuckDB** (`pip install duckdb`). Without it, fall back to `python -c "from datasets import load_dataset; ds = load_dataset('<id>', ...); print(ds.column_names, ds[0])"`.

package/skills/train-sentence-transformers/references/dataset_formats.md

@@ -0,0 +1,128 @@
+# Dataset Formats
+
+This reference covers: how datasets map to losses, how to reshape data when it doesn't fit, and how to mine hard negatives.
+
+## The two rules
+
+From the sentence-transformers training overview:
+
+1. If the loss requires a label, the dataset must have a column named **`label`, `labels`, `score`, or `scores`**. Any column with one of these names is the label.
+2. All other columns are **inputs**. The loss defines how many input columns it expects. Column **names don't matter. Order does**.
+
+Example: `CoSENTLoss` expects 2 inputs + a float label. A dataset with columns `["premise", "hypothesis", "score"]` works. A dataset with `["score", "premise", "hypothesis"]` does not. Reorder first.
+
+## Per-loss data shapes
+
+The per-type loss references (`losses_sentence_transformer.md`, `losses_cross_encoder.md`, `losses_sparse_encoder.md`) are the canonical mappings from data shape to loss. Cross-cutting recipe bits those tables don't show:
+
+- **`CosineSimilarityLoss`** wants `score` normalized to `[0, 1]`. `CoSENTLoss` / `AnglELoss` are pairwise-ranking and ignore absolute scale, so on `stsb` (raw 0-5) divide by 5 only when using cosine-similarity.
+- **`BatchAllTripletLoss` / `BatchHardTripletLoss` / `BatchSemiHardTripletLoss`** need `batch_sampler=BatchSamplers.GROUP_BY_LABEL` so multiple samples per label appear in the same batch.
+- **`MSELoss` (distillation)** label is the teacher's full embedding vector (a list of floats), not a scalar score.
+- **`MarginMSELoss` (distillation)** label is `teacher_score(q, pos) - teacher_score(q, neg)`, precomputed per row.
+- **N-tuple shape** for MNRL `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed) is produced by `mine_hard_negatives(..., output_format="n-tuple")`. "labeled-list" output_format produces the CrossEncoder listwise shape.
+
+## Reshaping operations
+
+If your data doesn't fit the loss's expected shape:
+
+### Reorder columns
+
+```python
+# Columns are ["hypothesis", "premise", "score"] but CoSENTLoss expects premise first.
+dataset = dataset.select_columns(["premise", "hypothesis", "score"])
+```
+
+### Rename label column
+
+```python
+# Your label is called "relevance" but ST wants "label".
+dataset = dataset.rename_column("relevance", "label")
+```
+
+### Drop extra columns
+
+```python
+# ST will treat every non-label column as an input. Drop metadata.
+dataset = dataset.remove_columns(["source_id", "created_at", "language"])
+```
+
+### Convert dtypes
+
+```python
+# Label is str, need float for CoSENTLoss.
+dataset = dataset.map(lambda x: {"label": float(x["label"])})
+```
+
+## Hard-negative mining
+
+`mine_hard_negatives` (in `sentence_transformers.util`) produces a training dataset with mined negatives using a retriever. Hard negatives are the single highest-leverage lever for retrieval-model quality.
+
+### Basic usage
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.util import mine_hard_negatives
+
+retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+
+mined = mine_hard_negatives(
+    dataset=train_pairs,                # has (anchor, positive) or (q, a) columns
+    model=retriever,
+    num_negatives=5,
+    range_min=0, range_max=100,         # rank window to sample hard negatives from
+    sampling_strategy="top",            # "top" = rank-1 hardest; "random" = random in window
+    output_format="n-tuple",            # "triplet" | "n-tuple" | "labeled-pair" | "labeled-list"
+    use_faiss=True,
+)
+```
+
+### Output formats
+
+- `"triplet"`: `(anchor, positive, negative)` triplets. One row per `(query, negative)` pair.
+- `"n-tuple"`: `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed). One row per query.
+- `"labeled-pair"`: `(anchor, text, label)` with `label=1` for positives and `label=0` for negatives. Good for `BinaryCrossEntropyLoss`.
+- `"labeled-list"`: `(anchor, texts, labels)`. One row per query with a list of candidates. Good for listwise losses.
+
+### Filtering false negatives
+
+If the retriever returns "negatives" that are actually relevant, they become false negatives and hurt training. Filter them:
+
+```python
+mined = mine_hard_negatives(
+    dataset=train_pairs,
+    model=retriever,
+    cross_encoder=CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2"),   # score candidates
+    num_negatives=5,
+    max_score=0.9,                      # drop candidates scoring above 0.9
+    relative_margin=0.05,               # require neg_score <= pos_score - abs(pos_score) * 0.05
+    absolute_margin=0.2,                # require neg_score <= pos_score - 0.2
+    output_format="n-tuple",
+    use_faiss=True,
+)
+```
+
+Use **either** `relative_margin` or `absolute_margin`, usually not both. `max_score` is independently useful as a hard ceiling.
+
+### CLI
+
+`scripts/mine_hard_negatives.py` is a CLI wrapper. See it for a ready-to-run command.
+
+## Choosing the right `range_min` / `range_max`
+
+`range_max=None` is the default. Pass an integer to cap how far down the ranked list to sample from.
+
+- `range_min=0`, `range_max=100`: sample from the top-100 retrieved. Good default.
+- `range_min=10`, `range_max=100`: skip the top-10 (often contains true positives). Safer if you lack a cross-encoder.
+- `range_min=0`, `range_max=1000`: wider net, more diverse negatives, slower.
+- `sampling_strategy="top"`: always pick the rank-1 hardest. Maximum training signal per row.
+- `sampling_strategy="random"`: pick randomly within the range. More robust if your retriever is itself noisy.
+
+## Quick Hub-side dataset checks
+
+`hf datasets sql "SELECT * FROM 'hf://datasets/<id>/<split>' LIMIT 5"` streams rows via DuckDB without `load_dataset(...)`. This is the fastest way to confirm column names match your loss before a full validation run. `hf datasets info <id>` shows config / splits / size. `hf datasets card <id> --text` renders the README.
+
+## Gotchas
+
+- **`remove_unused_columns=True` (default)**: the trainer drops columns that aren't passed to the model's forward. Usually fine, but if you rely on a custom collator that uses metadata columns, set `remove_unused_columns=False`.
+- **Floats stored as strings after CSV load**: `load_dataset("csv", ...)` keeps columns as strings by default. Cast with `.map(lambda x: {"label": float(x["label"])})`.
+- **Mined hard negatives with `include_positives=True`** include the positive as a negative in the output list, only useful when you're building an evaluator or want to measure rank of the positive. For training, leave it `False`.

package/skills/train-sentence-transformers/references/evaluators_cross_encoder.md

@@ -0,0 +1,116 @@
+# Evaluators (Cross-Encoder)
+
+All cross-encoder evaluators live in `sentence_transformers.cross_encoder.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Rerank retrieval results (nDCG@k on BM25 top-N), fast default | `CrossEncoderNanoBEIREvaluator` |
+| Rerank with custom candidates per query | `CrossEncoderRerankingEvaluator` |
+| Binary / multi-class pair classification | `CrossEncoderClassificationEvaluator` |
+| Continuous pair scoring (STS-style) | `CrossEncoderCorrelationEvaluator` |
+
+Wrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`) to track them together:
+
+```python
+from sentence_transformers.base.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([nano_beir_eval, custom_rerank_eval])
+```
+
+## The default: `CrossEncoderNanoBEIREvaluator`
+
+Analog of `NanoBEIREvaluator` for rerankers. Takes BM25 top-100 for each NanoBEIR query and measures how well the cross-encoder re-ranks them.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderNanoBEIREvaluator
+
+evaluator = CrossEncoderNanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: 11 of 13 NanoBEIR datasets (excludes "arguana", "touche2020")
+    batch_size=64,
+    rerank_k=100,                                   # rerank the BM25 top-K
+)
+```
+
+Output key for `metric_for_best_model`: **`eval_NanoBEIR_R100_mean_ndcg@10`**. The `R100` signals "rerank top-100". If you change `rerank_k`, the prefix changes (e.g. `R50`).
+
+Each individual dataset contributes `eval_Nano{DatasetName}_R100_ndcg@10` (e.g. `eval_NanoMSMARCO_R100_ndcg@10`) too.
+
+## Custom reranking with your own candidates
+
+Use when you have query + positive + distractor candidates that aren't part of NanoBEIR:
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderRerankingEvaluator
+
+samples = [
+    {"query": "...", "positive": ["the gold answer"], "documents": ["...", "...", ...]}
+    for ...
+]
+
+evaluator = CrossEncoderRerankingEvaluator(
+    samples=samples,
+    batch_size=64,
+    name="my-rerank",
+    always_rerank_positives=False,   # default is True; override to False for realistic eval
+)
+```
+
+- `always_rerank_positives=True` (the library default) forces the positive into the candidate pool even when the retriever missed it. The reranker is graded only on candidates it can actually score, so the metric reflects pure reranker quality.
+- `always_rerank_positives=False`: the positive is only reranked if it's already in `documents`. If the retriever missed it, the rank counts as N+1. This reflects end-to-end retriever+reranker quality. A positive the retriever missed is lost, regardless of reranker skill.
+
+Output key: `eval_{name}_ndcg@10`, `eval_{name}_map`, `eval_{name}_mrr@10`.
+
+## Classification-style cross-encoders
+
+### `CrossEncoderClassificationEvaluator`
+
+Works for both binary (`num_labels=1`) and multi-class (`num_labels>=2`) cross-encoders. Branches internally:
+- `num_labels=1`: binary mode. Sweeps thresholds to report accuracy, F1, precision, recall, and **average_precision** (primary).
+- `num_labels>=2`: multi-class mode (e.g. NLI: entailment / neutral / contradiction). Reports **f1_macro** (primary), f1_micro, f1_weighted, and per-class precision / recall.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderClassificationEvaluator
+
+evaluator = CrossEncoderClassificationEvaluator(
+    sentence_pairs=[(premise, hypothesis), ...],
+    labels=[0, 1, 2, ...],
+    batch_size=64,
+    name="nli-dev",
+)
+```
+
+Output keys (binary, `num_labels=1`): `eval_{name}_accuracy`, `eval_{name}_f1`, `eval_{name}_average_precision` (primary).
+Output keys (multi-class, `num_labels>=2`): `eval_{name}_f1_macro` (primary), `eval_{name}_f1_micro`, `eval_{name}_f1_weighted`.
+
+### `CrossEncoderCorrelationEvaluator`
+
+For continuous-score cross-encoders (like an STS cross-encoder outputting a similarity score). Reports Pearson/Spearman vs. gold scores.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderCorrelationEvaluator
+
+evaluator = CrossEncoderCorrelationEvaluator(
+    sentence_pairs=[(a, b), ...],
+    scores=[0.4, 0.8, ...],
+    name="stsb-dev",
+)
+```
+
+Output keys: `eval_{name}_spearman`, `eval_{name}_pearson`.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_R100_mean_ndcg@10`: `CrossEncoderNanoBEIREvaluator` default
+- `eval_{name}_ndcg@10`: `CrossEncoderRerankingEvaluator`
+- `eval_{name}_average_precision`: `CrossEncoderClassificationEvaluator` (binary, `num_labels=1`)
+- `eval_{name}_f1_macro`: `CrossEncoderClassificationEvaluator` (multi-class, `num_labels>=2`)
+- `eval_{name}_spearman`: `CrossEncoderCorrelationEvaluator`
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**: pre-training baseline. Tiny post-training delta means the loss/data/base is wrong.
+- `CrossEncoderClassificationEvaluator` accepts both `num_labels=1` (binary, primary `average_precision`) and `num_labels>=2` (multi-class, primary `f1_macro`). `CrossEncoderCorrelationEvaluator` requires `num_labels=1`.
+- The default `dataset_names=None` excludes `arguana` and `touche2020` (Argument-Retrieval task differs from the rest). Pass `dataset_names=list(DATASET_NAME_TO_HUMAN_READABLE)` from `sentence_transformers.cross_encoder.evaluation.nano_beir` to actually run all 13.
+- Subset NanoBEIR datasets during training (3-4) to keep eval cheap. Run the broader set post-training.

package/skills/train-sentence-transformers/references/evaluators_multi_vector_encoder.md

@@ -0,0 +1,101 @@
+# Multi-Vector-Encoder Evaluators
+
+All evaluators live in `sentence_transformers.multi_vector_encoder.evaluation`. They mirror the bi-encoder evaluators but use MaxSim scoring end-to-end.
+
+## Top-line pick
+
+| Task | Evaluator |
+|---|---|
+| Retrieval on a suite of common English IR benchmarks | `MultiVectorNanoBEIREvaluator` |
+| Custom retrieval corpus (your own docs / queries / qrels) | `MultiVectorInformationRetrievalEvaluator` |
+| Distillation from a cross-encoder teacher | `MultiVectorDistillationEvaluator` |
+| Reranking a fixed candidate list per query | `MultiVectorRerankingEvaluator` |
+| Triplet accuracy (does anchor score positive > negative?) | `MultiVectorTripletEvaluator` |
+
+**Default recommendation for training**: `MultiVectorNanoBEIREvaluator` on a subset of NanoBEIR datasets (e.g. `["msmarco", "nq", "fiqa2018"]`) during training, and the full suite at end-of-run. Cheap, well-calibrated, and the metric key format is stable.
+
+## `MultiVectorNanoBEIREvaluator`
+
+Runs the 13 NanoBEIR sub-datasets and returns nDCG@10 averaged across them.
+
+```python
+from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator
+
+evaluator = MultiVectorNanoBEIREvaluator(
+    dataset_names=["msmarco", "nq", "fiqa2018"],  # subset for training-time evals
+    batch_size=16,
+)
+```
+
+**primary_metric**: `"NanoBEIR_mean_maxsim_ndcg@10"` (unless you pass `aggregate_key="..."`, in which case it becomes `"NanoBEIR_{aggregate_key}_maxsim_ndcg@10"`). Note that `NanoBEIREvaluator` composes its display name from `aggregate_key` (and optionally `truncate_dim`), it does not accept a `name` kwarg.
+
+**`metric_for_best_model` key**: `"eval_NanoBEIR_mean_maxsim_ndcg@10"` (add the `eval_` prefix, since Trainer adds it to evaluator metrics).
+
+- `dataset_names`: list of NanoBEIR sub-datasets. Full list: `msmarco`, `nq`, `fiqa2018`, `hotpotqa`, `nfcorpus`, `arguana`, `scidocs`, `climatefever`, `dbpedia`, `fever`, `quoraretrieval`, `scifact`, `touche2020`. Note it is `quoraretrieval`, not `quora`: an invalid name raises at construction.
+- `batch_size`: also drives corpus encoding, so scale to fit memory.
+- English-only. For non-English retrieval, use `MultiVectorInformationRetrievalEvaluator` with your own corpus.
+
+## `MultiVectorInformationRetrievalEvaluator`
+
+Full IR evaluator over a corpus + queries + qrels you supply.
+
+```python
+from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorInformationRetrievalEvaluator
+
+evaluator = MultiVectorInformationRetrievalEvaluator(
+    queries={qid: text, ...},                       # dict of query_id to query_text
+    corpus={did: text, ...},                        # dict of doc_id to doc_text
+    relevant_docs={qid: {did1, did2, ...}, ...},    # qid to set of relevant doc_ids
+    batch_size=16,
+    name="my-eval",                                 # optional, prefixes the metric key
+    write_csv=True,                                 # persists per-eval metrics
+    ndcg_at_k=[10],                                 # customize k-values if you need @20/@100 etc.
+)
+```
+
+**primary_metric**: `"maxsim_ndcg@10"` by default (built from `score_function="maxsim"` and `max(ndcg_at_k)`).
+
+**`metric_for_best_model` key** with `name="my-eval"`:  `"eval_my-eval_maxsim_ndcg@10"`.
+
+- `main_score_function` overrides the score used in `primary_metric` if you set a non-default. Default is `None`, which resolves to `maxsim` from the model's `similarity_fn_name` at call time. For MVE just leave it as `None`.
+- Reports MRR@k, nDCG@k, Recall@k, Precision@k, MAP@k for the k-values you pass (`mrr_at_k`, `ndcg_at_k`, etc.).
+- `chunk_elements` (default `None`) is the element budget for the padded `(chunk, d_tokens, dim)` documents plus the `(num_queries, chunk, q_tokens, d_tokens)` MaxSim scoring intermediate. Leave it as `None`: `maxsim` packs documents into chunks under a 100M-element budget (at most ~400 MB, half that in bf16 / fp16), adapting to the query count and document lengths. Lower it to cut evaluation memory further. It is a separate axis from `corpus_chunk_size`, which caps how many encoded documents are held at a time.
+
+## `MultiVectorDistillationEvaluator`
+
+Regression of student MaxSim scores against teacher scores over a held-out set of `(query, doc, teacher_score)` rows. Use it when you're distilling.
+
+**primary_metric**: `"spearman"` (correlation of student scores with teacher scores).
+
+**`metric_for_best_model` key** with `name="my-eval"`: `"eval_my-eval_spearman"`.
+
+- Pair with `MultiVectorMarginMSELoss` or `MultiVectorDistillKLDivLoss` during training.
+- Higher is better (student ranking matches teacher).
+
+## `MultiVectorRerankingEvaluator`
+
+Reranks a fixed list of candidates per query. Same shape as the bi-encoder `RerankingEvaluator`, MaxSim-scored.
+
+- **Data**: `samples = [{"query": ..., "positive": [...], "negative": [...]} , ...]`.
+- Reports MAP, MRR@10, nDCG@10 by default.
+
+## `MultiVectorTripletEvaluator`
+
+Fraction of `(anchor, positive, negative)` triplets where `sim(anchor, positive) > sim(anchor, negative)`. Cheap sanity signal, not what you want to gate a real IR release on.
+
+## Named-evaluator metric key format (universal)
+
+If you pass `name="..."` to any of these, the metric key format is:
+
+```
+eval_{name}_{primary_metric}
+```
+
+If you don't pass `name`, the key is just `eval_{primary_metric}`. The trainer's `metric_for_best_model=...` must match exactly, or `load_best_model_at_end` selects the wrong checkpoint. **Run the evaluator once before training** to observe the exact key format: `primary_metric` is `None` until the first `evaluator(model)` call, which composes it from the resolved score function and the `ndcg_at_k` you passed, then prefixes `name`. The production template does exactly this before building `TrainingArguments`.
+
+## Gotchas
+
+- **Metric key mismatch on `metric_for_best_model`**: silent failure. Training runs to completion and `load_best_model_at_end` picks a stale checkpoint. Always print `evaluator(model)` output once before starting the trainer and confirm the key format.
+- **Eval-time OOM**: MaxSim scoring builds `(num_queries, chunk, q_tokens, d_tokens)` intermediates, which `maxsim` bounds by default under its 100M-element budget. If you still OOM, lower `chunk_elements`, then drop `batch_size`. NanoBEIR corpora can be small enough that oversized `batch_size` bites before you'd expect.
+- **`MultiVectorNanoBEIREvaluator` with a Non-English base**: it's English-only. You'll get zero or garbage scores. Use `MultiVectorInformationRetrievalEvaluator` with an in-domain corpus.
+- **Distillation eval on the same data as training**: measures memorization, not generalization. Hold out a separate `(query, doc, teacher_score)` split.

package/skills/train-sentence-transformers/references/evaluators_sentence_transformer.md

@@ -0,0 +1,151 @@
+# Evaluators (Bi-Encoder)
+
+All bi-encoder evaluators live in `sentence_transformers.sentence_transformer.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Retrieval (nDCG, MRR, Recall), fast default | `NanoBEIREvaluator` |
+| Retrieval on your own corpus / qrels | `InformationRetrievalEvaluator` |
+| STS / continuous similarity | `EmbeddingSimilarityEvaluator` |
+| Binary classification | `BinaryClassificationEvaluator` |
+| Triplet accuracy | `TripletEvaluator` |
+| Reranking (from retrieval candidates) | `RerankingEvaluator` |
+| MSE vs. teacher (distillation) | `MSEEvaluator`, `MSEEvaluatorFromDataFrame` |
+| Paraphrase mining | `ParaphraseMiningEvaluator` |
+| Translation (cross-lingual alignment) | `TranslationEvaluator` |
+| Label accuracy (classification during training) | `LabelAccuracyEvaluator` |
+
+Wrap multiple evaluators in `SequentialEvaluator` to track all of them together:
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([evaluator1, evaluator2, evaluator3])
+```
+
+## The big three
+
+### `NanoBEIREvaluator` (retrieval)
+
+Small, fast subset of BEIR. Typically runs in <1 minute on a mid-range GPU. Default choice for retrieval training.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import NanoBEIREvaluator
+
+evaluator = NanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
+    batch_size=128,
+    show_progress_bar=False,
+)
+```
+
+- Default dataset list covers 13 tasks. Pick a subset for speed during training.
+- Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_cosine_ndcg@10`** (bi-encoder default = cosine similarity).
+
+### `EmbeddingSimilarityEvaluator` (STS-style)
+
+Computes Pearson/Spearman correlation between model cosine similarities and gold labels.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import EmbeddingSimilarityEvaluator
+from sentence_transformers.util.similarity import SimilarityFunction
+
+evaluator = EmbeddingSimilarityEvaluator(
+    sentences1=stsb["sentence1"],
+    sentences2=stsb["sentence2"],
+    scores=stsb["score"],
+    main_similarity=SimilarityFunction.COSINE,
+    name="sts-dev",
+)
+```
+
+- `main_similarity` can be `COSINE`, `DOT_PRODUCT`, `EUCLIDEAN`, `MANHATTAN`.
+- `name` is used in the output key: `eval_sts-dev_spearman_cosine`, `eval_sts-dev_pearson_cosine`, etc.
+
+### `InformationRetrievalEvaluator` (full retrieval)
+
+Use when you have your **own** corpus + queries + qrels (not one of the NanoBEIR tasks).
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import InformationRetrievalEvaluator
+
+evaluator = InformationRetrievalEvaluator(
+    queries={qid: query_text for qid, query_text in ...},
+    corpus={doc_id: doc_text for doc_id, doc_text in ...},
+    relevant_docs={qid: {doc_id, ...} for qid in ...},   # qid -> set of relevant doc_ids
+    name="my-retrieval",
+    mrr_at_k=[10],
+    ndcg_at_k=[10],
+    accuracy_at_k=[1, 5, 10],
+    precision_recall_at_k=[1, 5, 10],
+    map_at_k=[100],
+    show_progress_bar=False,
+    batch_size=64,
+)
+```
+
+Output keys: `eval_{name}_cosine_ndcg@10`, `eval_{name}_cosine_mrr@10`, etc.
+
+Heavy for large corpora: each eval encodes the full corpus. Don't run it every 100 steps. Use `NanoBEIREvaluator` for frequent evaluation during training and reserve full IR for milestones / post-training.
+
+## Other bi-encoder evaluators
+
+### `BinaryClassificationEvaluator`
+
+For labeled pair classification (e.g. duplicate detection, entailment as binary). Reports accuracy, F1, precision/recall, AP. Supports all distance metrics. Finds the best threshold per metric.
+
+### `TripletEvaluator`
+
+For `(anchor, positive, negative)` triplets. Reports the fraction of triplets where the positive is closer to the anchor than the negative.
+
+### `RerankingEvaluator`
+
+For custom re-ranking datasets: you provide candidates per query, the evaluator computes MAP and MRR. Good for measuring retrieval-quality on a held-out set.
+
+### `MSEEvaluator` / `MSEEvaluatorFromDataFrame`
+
+For distillation setups. Compares student embeddings against teacher embeddings (or teacher scores), reports MSE.
+
+### `ParaphraseMiningEvaluator`
+
+For paraphrase-mining tasks. Given a corpus of labeled paraphrase pairs, computes mining quality (F1 at various thresholds).
+
+### `TranslationEvaluator`
+
+For cross-lingual / `make_multilingual`-style alignment checking. Measures whether the student aligns sentences across languages.
+
+### `LabelAccuracyEvaluator`
+
+For a `SoftmaxLoss`-trained classifier head. Reports accuracy on held-out data.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_mean_cosine_ndcg@10`: `NanoBEIREvaluator`
+- `eval_sts-dev_spearman_cosine`: `EmbeddingSimilarityEvaluator(name="sts-dev")`
+- `eval_{name}_cosine_ndcg@10`: `InformationRetrievalEvaluator(name=...)`
+
+## Multi-dimensional evaluation (Matryoshka)
+
+For Matryoshka-trained models, evaluate at each target dimension:
+
+```python
+per_dim_evaluators = [
+    EmbeddingSimilarityEvaluator(
+        sentences1=..., sentences2=..., scores=...,
+        main_similarity=SimilarityFunction.COSINE,
+        name=f"sts-dev-{dim}",
+        truncate_dim=dim,
+    ) for dim in [768, 512, 256, 128, 64]
+]
+evaluator = SequentialEvaluator(per_dim_evaluators, main_score_function=lambda scores: scores[0])
+```
+
+The first evaluator's score drives `load_best_model_at_end`.
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**: pre-training baseline. If the post-training delta is tiny, the loss/data/base is wrong.
+- Don't run `InformationRetrievalEvaluator` with a large corpus (>100k docs) at frequent `eval_steps`. Use `NanoBEIREvaluator` during training, reserve full IR for end-of-training.
+- `greater_is_better=True` is the default, right for nDCG / MRR / accuracy.