tribunal-kit 9.2.3 → 9.2.4

package/.agent/skills/train-sentence-transformers/references/dataset_formats.md

@@ -0,0 +1,128 @@
+# Dataset Formats
+
+This reference covers: how datasets map to losses, how to reshape data when it doesn't fit, and how to mine hard negatives.
+
+## The two rules
+
+From the sentence-transformers training overview:
+
+1. If the loss requires a label, the dataset must have a column named **`label`, `labels`, `score`, or `scores`**. Any column with one of these names is the label.
+2. All other columns are **inputs**. The loss defines how many input columns it expects. Column **names don't matter. Order does**.
+
+Example: `CoSENTLoss` expects 2 inputs + a float label. A dataset with columns `["premise", "hypothesis", "score"]` works. A dataset with `["score", "premise", "hypothesis"]` does not. Reorder first.
+
+## Per-loss data shapes
+
+The per-type loss references (`losses_sentence_transformer.md`, `losses_cross_encoder.md`, `losses_sparse_encoder.md`) are the canonical mappings from data shape to loss. Cross-cutting recipe bits those tables don't show:
+
+- **`CosineSimilarityLoss`** wants `score` normalized to `[0, 1]`. `CoSENTLoss` / `AnglELoss` are pairwise-ranking and ignore absolute scale, so on `stsb` (raw 0-5) divide by 5 only when using cosine-similarity.
+- **`BatchAllTripletLoss` / `BatchHardTripletLoss` / `BatchSemiHardTripletLoss`** need `batch_sampler=BatchSamplers.GROUP_BY_LABEL` so multiple samples per label appear in the same batch.
+- **`MSELoss` (distillation)** label is the teacher's full embedding vector (a list of floats), not a scalar score.
+- **`MarginMSELoss` (distillation)** label is `teacher_score(q, pos) - teacher_score(q, neg)`, precomputed per row.
+- **N-tuple shape** for MNRL `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed) is produced by `mine_hard_negatives(..., output_format="n-tuple")`. "labeled-list" output_format produces the CrossEncoder listwise shape.
+
+## Reshaping operations
+
+If your data doesn't fit the loss's expected shape:
+
+### Reorder columns
+
+```python
+# Columns are ["hypothesis", "premise", "score"] but CoSENTLoss expects premise first.
+dataset = dataset.select_columns(["premise", "hypothesis", "score"])
+```
+
+### Rename label column
+
+```python
+# Your label is called "relevance" but ST wants "label".
+dataset = dataset.rename_column("relevance", "label")
+```
+
+### Drop extra columns
+
+```python
+# ST will treat every non-label column as an input. Drop metadata.
+dataset = dataset.remove_columns(["source_id", "created_at", "language"])
+```
+
+### Convert dtypes
+
+```python
+# Label is str, need float for CoSENTLoss.
+dataset = dataset.map(lambda x: {"label": float(x["label"])})
+```
+
+## Hard-negative mining
+
+`mine_hard_negatives` (in `sentence_transformers.util`) produces a training dataset with mined negatives using a retriever. Hard negatives are the single highest-leverage lever for retrieval-model quality.
+
+### Basic usage
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.util import mine_hard_negatives
+
+retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+
+mined = mine_hard_negatives(
+    dataset=train_pairs,                # has (anchor, positive) or (q, a) columns
+    model=retriever,
+    num_negatives=5,
+    range_min=0, range_max=100,         # rank window to sample hard negatives from
+    sampling_strategy="top",            # "top" = rank-1 hardest; "random" = random in window
+    output_format="n-tuple",            # "triplet" | "n-tuple" | "labeled-pair" | "labeled-list"
+    use_faiss=True,
+)
+```
+
+### Output formats
+
+- `"triplet"`: `(anchor, positive, negative)` triplets. One row per `(query, negative)` pair.
+- `"n-tuple"`: `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed). One row per query.
+- `"labeled-pair"`: `(anchor, text, label)` with `label=1` for positives and `label=0` for negatives. Good for `BinaryCrossEntropyLoss`.
+- `"labeled-list"`: `(anchor, texts, labels)`. One row per query with a list of candidates. Good for listwise losses.
+
+### Filtering false negatives
+
+If the retriever returns "negatives" that are actually relevant, they become false negatives and hurt training. Filter them:
+
+```python
+mined = mine_hard_negatives(
+    dataset=train_pairs,
+    model=retriever,
+    cross_encoder=CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2"),   # score candidates
+    num_negatives=5,
+    max_score=0.9,                      # drop candidates scoring above 0.9
+    relative_margin=0.05,               # require neg_score <= pos_score - abs(pos_score) * 0.05
+    absolute_margin=0.2,                # require neg_score <= pos_score - 0.2
+    output_format="n-tuple",
+    use_faiss=True,
+)
+```
+
+Use **either** `relative_margin` or `absolute_margin`, usually not both. `max_score` is independently useful as a hard ceiling.
+
+### CLI
+
+`scripts/mine_hard_negatives.py` is a CLI wrapper. See it for a ready-to-run command.
+
+## Choosing the right `range_min` / `range_max`
+
+`range_max=None` is the default. Pass an integer to cap how far down the ranked list to sample from.
+
+- `range_min=0`, `range_max=100`: sample from the top-100 retrieved. Good default.
+- `range_min=10`, `range_max=100`: skip the top-10 (often contains true positives). Safer if you lack a cross-encoder.
+- `range_min=0`, `range_max=1000`: wider net, more diverse negatives, slower.
+- `sampling_strategy="top"`: always pick the rank-1 hardest. Maximum training signal per row.
+- `sampling_strategy="random"`: pick randomly within the range. More robust if your retriever is itself noisy.
+
+## Quick Hub-side dataset checks
+
+`hf datasets sql "SELECT * FROM 'hf://datasets/<id>/<split>' LIMIT 5"` streams rows via DuckDB without `load_dataset(...)`. This is the fastest way to confirm column names match your loss before a full validation run. `hf datasets info <id>` shows config / splits / size. `hf datasets card <id> --text` renders the README.
+
+## Gotchas
+
+- **`remove_unused_columns=True` (default)**: the trainer drops columns that aren't passed to the model's forward. Usually fine, but if you rely on a custom collator that uses metadata columns, set `remove_unused_columns=False`.
+- **Floats stored as strings after CSV load**: `load_dataset("csv", ...)` keeps columns as strings by default. Cast with `.map(lambda x: {"label": float(x["label"])})`.
+- **Mined hard negatives with `include_positives=True`** include the positive as a negative in the output list, only useful when you're building an evaluator or want to measure rank of the positive. For training, leave it `False`.

package/.agent/skills/train-sentence-transformers/references/evaluators_cross_encoder.md

@@ -0,0 +1,116 @@
+# Evaluators (Cross-Encoder)
+
+All cross-encoder evaluators live in `sentence_transformers.cross_encoder.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Rerank retrieval results (nDCG@k on BM25 top-N), fast default | `CrossEncoderNanoBEIREvaluator` |
+| Rerank with custom candidates per query | `CrossEncoderRerankingEvaluator` |
+| Binary / multi-class pair classification | `CrossEncoderClassificationEvaluator` |
+| Continuous pair scoring (STS-style) | `CrossEncoderCorrelationEvaluator` |
+
+Wrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`) to track them together:
+
+```python
+from sentence_transformers.base.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([nano_beir_eval, custom_rerank_eval])
+```
+
+## The default: `CrossEncoderNanoBEIREvaluator`
+
+Analog of `NanoBEIREvaluator` for rerankers. Takes BM25 top-100 for each NanoBEIR query and measures how well the cross-encoder re-ranks them.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderNanoBEIREvaluator
+
+evaluator = CrossEncoderNanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: 11 of 13 NanoBEIR datasets (excludes "arguana", "touche2020")
+    batch_size=64,
+    rerank_k=100,                                   # rerank the BM25 top-K
+)
+```
+
+Output key for `metric_for_best_model`: **`eval_NanoBEIR_R100_mean_ndcg@10`**. The `R100` signals "rerank top-100". If you change `rerank_k`, the prefix changes (e.g. `R50`).
+
+Each individual dataset contributes `eval_Nano{DatasetName}_R100_ndcg@10` (e.g. `eval_NanoMSMARCO_R100_ndcg@10`) too.
+
+## Custom reranking with your own candidates
+
+Use when you have query + positive + distractor candidates that aren't part of NanoBEIR:
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderRerankingEvaluator
+
+samples = [
+    {"query": "...", "positive": ["the gold answer"], "documents": ["...", "...", ...]}
+    for ...
+]
+
+evaluator = CrossEncoderRerankingEvaluator(
+    samples=samples,
+    batch_size=64,
+    name="my-rerank",
+    always_rerank_positives=False,   # default is True; override to False for realistic eval
+)
+```
+
+- `always_rerank_positives=True` (the library default) forces the positive into the candidate pool even when the retriever missed it. The reranker is graded only on candidates it can actually score, so the metric reflects pure reranker quality.
+- `always_rerank_positives=False`: the positive is only reranked if it's already in `documents`. If the retriever missed it, the rank counts as N+1. This reflects end-to-end retriever+reranker quality. A positive the retriever missed is lost, regardless of reranker skill.
+
+Output key: `eval_{name}_ndcg@10`, `eval_{name}_map`, `eval_{name}_mrr@10`.
+
+## Classification-style cross-encoders
+
+### `CrossEncoderClassificationEvaluator`
+
+Works for both binary (`num_labels=1`) and multi-class (`num_labels>=2`) cross-encoders. Branches internally:
+- `num_labels=1`: binary mode. Sweeps thresholds to report accuracy, F1, precision, recall, and **average_precision** (primary).
+- `num_labels>=2`: multi-class mode (e.g. NLI: entailment / neutral / contradiction). Reports **f1_macro** (primary), f1_micro, f1_weighted, and per-class precision / recall.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderClassificationEvaluator
+
+evaluator = CrossEncoderClassificationEvaluator(
+    sentence_pairs=[(premise, hypothesis), ...],
+    labels=[0, 1, 2, ...],
+    batch_size=64,
+    name="nli-dev",
+)
+```
+
+Output keys (binary, `num_labels=1`): `eval_{name}_accuracy`, `eval_{name}_f1`, `eval_{name}_average_precision` (primary).
+Output keys (multi-class, `num_labels>=2`): `eval_{name}_f1_macro` (primary), `eval_{name}_f1_micro`, `eval_{name}_f1_weighted`.
+
+### `CrossEncoderCorrelationEvaluator`
+
+For continuous-score cross-encoders (like an STS cross-encoder outputting a similarity score). Reports Pearson/Spearman vs. gold scores.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderCorrelationEvaluator
+
+evaluator = CrossEncoderCorrelationEvaluator(
+    sentence_pairs=[(a, b), ...],
+    scores=[0.4, 0.8, ...],
+    name="stsb-dev",
+)
+```
+
+Output keys: `eval_{name}_spearman`, `eval_{name}_pearson`.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_R100_mean_ndcg@10`: `CrossEncoderNanoBEIREvaluator` default
+- `eval_{name}_ndcg@10`: `CrossEncoderRerankingEvaluator`
+- `eval_{name}_average_precision`: `CrossEncoderClassificationEvaluator` (binary, `num_labels=1`)
+- `eval_{name}_f1_macro`: `CrossEncoderClassificationEvaluator` (multi-class, `num_labels>=2`)
+- `eval_{name}_spearman`: `CrossEncoderCorrelationEvaluator`
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**: pre-training baseline. Tiny post-training delta means the loss/data/base is wrong.
+- `CrossEncoderClassificationEvaluator` accepts both `num_labels=1` (binary, primary `average_precision`) and `num_labels>=2` (multi-class, primary `f1_macro`). `CrossEncoderCorrelationEvaluator` requires `num_labels=1`.
+- The default `dataset_names=None` excludes `arguana` and `touche2020` (Argument-Retrieval task differs from the rest). Pass `dataset_names=list(DATASET_NAME_TO_HUMAN_READABLE)` from `sentence_transformers.cross_encoder.evaluation.nano_beir` to actually run all 13.
+- Subset NanoBEIR datasets during training (3-4) to keep eval cheap. Run the broader set post-training.

package/.agent/skills/train-sentence-transformers/references/evaluators_multi_vector_encoder.md

@@ -0,0 +1,101 @@
+# Multi-Vector-Encoder Evaluators
+
+All evaluators live in `sentence_transformers.multi_vector_encoder.evaluation`. They mirror the bi-encoder evaluators but use MaxSim scoring end-to-end.
+
+## Top-line pick
+
+| Task | Evaluator |
+|---|---|
+| Retrieval on a suite of common English IR benchmarks | `MultiVectorNanoBEIREvaluator` |
+| Custom retrieval corpus (your own docs / queries / qrels) | `MultiVectorInformationRetrievalEvaluator` |
+| Distillation from a cross-encoder teacher | `MultiVectorDistillationEvaluator` |
+| Reranking a fixed candidate list per query | `MultiVectorRerankingEvaluator` |
+| Triplet accuracy (does anchor score positive > negative?) | `MultiVectorTripletEvaluator` |
+
+**Default recommendation for training**: `MultiVectorNanoBEIREvaluator` on a subset of NanoBEIR datasets (e.g. `["msmarco", "nq", "fiqa2018"]`) during training, and the full suite at end-of-run. Cheap, well-calibrated, and the metric key format is stable.
+
+## `MultiVectorNanoBEIREvaluator`
+
+Runs the 13 NanoBEIR sub-datasets and returns nDCG@10 averaged across them.
+
+```python
+from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator
+
+evaluator = MultiVectorNanoBEIREvaluator(
+    dataset_names=["msmarco", "nq", "fiqa2018"],  # subset for training-time evals
+    batch_size=16,
+)
+```
+
+**primary_metric**: `"NanoBEIR_mean_maxsim_ndcg@10"` (unless you pass `aggregate_key="..."`, in which case it becomes `"NanoBEIR_{aggregate_key}_maxsim_ndcg@10"`). Note that `NanoBEIREvaluator` composes its display name from `aggregate_key` (and optionally `truncate_dim`), it does not accept a `name` kwarg.
+
+**`metric_for_best_model` key**: `"eval_NanoBEIR_mean_maxsim_ndcg@10"` (add the `eval_` prefix, since Trainer adds it to evaluator metrics).
+
+- `dataset_names`: list of NanoBEIR sub-datasets. Full list: `msmarco`, `nq`, `fiqa2018`, `hotpotqa`, `nfcorpus`, `arguana`, `scidocs`, `climatefever`, `dbpedia`, `fever`, `quoraretrieval`, `scifact`, `touche2020`. Note it is `quoraretrieval`, not `quora`: an invalid name raises at construction.
+- `batch_size`: also drives corpus encoding, so scale to fit memory.
+- English-only. For non-English retrieval, use `MultiVectorInformationRetrievalEvaluator` with your own corpus.
+
+## `MultiVectorInformationRetrievalEvaluator`
+
+Full IR evaluator over a corpus + queries + qrels you supply.
+
+```python
+from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorInformationRetrievalEvaluator
+
+evaluator = MultiVectorInformationRetrievalEvaluator(
+    queries={qid: text, ...},                       # dict of query_id to query_text
+    corpus={did: text, ...},                        # dict of doc_id to doc_text
+    relevant_docs={qid: {did1, did2, ...}, ...},    # qid to set of relevant doc_ids
+    batch_size=16,
+    name="my-eval",                                 # optional, prefixes the metric key
+    write_csv=True,                                 # persists per-eval metrics
+    ndcg_at_k=[10],                                 # customize k-values if you need @20/@100 etc.
+)
+```
+
+**primary_metric**: `"maxsim_ndcg@10"` by default (built from `score_function="maxsim"` and `max(ndcg_at_k)`).
+
+**`metric_for_best_model` key** with `name="my-eval"`:  `"eval_my-eval_maxsim_ndcg@10"`.
+
+- `main_score_function` overrides the score used in `primary_metric` if you set a non-default. Default is `None`, which resolves to `maxsim` from the model's `similarity_fn_name` at call time. For MVE just leave it as `None`.
+- Reports MRR@k, nDCG@k, Recall@k, Precision@k, MAP@k for the k-values you pass (`mrr_at_k`, `ndcg_at_k`, etc.).
+- `chunk_elements` (default `None`) is the element budget for the padded `(chunk, d_tokens, dim)` documents plus the `(num_queries, chunk, q_tokens, d_tokens)` MaxSim scoring intermediate. Leave it as `None`: `maxsim` packs documents into chunks under a 100M-element budget (at most ~400 MB, half that in bf16 / fp16), adapting to the query count and document lengths. Lower it to cut evaluation memory further. It is a separate axis from `corpus_chunk_size`, which caps how many encoded documents are held at a time.
+
+## `MultiVectorDistillationEvaluator`
+
+Regression of student MaxSim scores against teacher scores over a held-out set of `(query, doc, teacher_score)` rows. Use it when you're distilling.
+
+**primary_metric**: `"spearman"` (correlation of student scores with teacher scores).
+
+**`metric_for_best_model` key** with `name="my-eval"`: `"eval_my-eval_spearman"`.
+
+- Pair with `MultiVectorMarginMSELoss` or `MultiVectorDistillKLDivLoss` during training.
+- Higher is better (student ranking matches teacher).
+
+## `MultiVectorRerankingEvaluator`
+
+Reranks a fixed list of candidates per query. Same shape as the bi-encoder `RerankingEvaluator`, MaxSim-scored.
+
+- **Data**: `samples = [{"query": ..., "positive": [...], "negative": [...]} , ...]`.
+- Reports MAP, MRR@10, nDCG@10 by default.
+
+## `MultiVectorTripletEvaluator`
+
+Fraction of `(anchor, positive, negative)` triplets where `sim(anchor, positive) > sim(anchor, negative)`. Cheap sanity signal, not what you want to gate a real IR release on.
+
+## Named-evaluator metric key format (universal)
+
+If you pass `name="..."` to any of these, the metric key format is:
+
+```
+eval_{name}_{primary_metric}
+```
+
+If you don't pass `name`, the key is just `eval_{primary_metric}`. The trainer's `metric_for_best_model=...` must match exactly, or `load_best_model_at_end` selects the wrong checkpoint. **Run the evaluator once before training** to observe the exact key format: `primary_metric` is `None` until the first `evaluator(model)` call, which composes it from the resolved score function and the `ndcg_at_k` you passed, then prefixes `name`. The production template does exactly this before building `TrainingArguments`.
+
+## Gotchas
+
+- **Metric key mismatch on `metric_for_best_model`**: silent failure. Training runs to completion and `load_best_model_at_end` picks a stale checkpoint. Always print `evaluator(model)` output once before starting the trainer and confirm the key format.
+- **Eval-time OOM**: MaxSim scoring builds `(num_queries, chunk, q_tokens, d_tokens)` intermediates, which `maxsim` bounds by default under its 100M-element budget. If you still OOM, lower `chunk_elements`, then drop `batch_size`. NanoBEIR corpora can be small enough that oversized `batch_size` bites before you'd expect.
+- **`MultiVectorNanoBEIREvaluator` with a Non-English base**: it's English-only. You'll get zero or garbage scores. Use `MultiVectorInformationRetrievalEvaluator` with an in-domain corpus.
+- **Distillation eval on the same data as training**: measures memorization, not generalization. Hold out a separate `(query, doc, teacher_score)` split.

package/.agent/skills/train-sentence-transformers/references/evaluators_sentence_transformer.md

@@ -0,0 +1,151 @@
+# Evaluators (Bi-Encoder)
+
+All bi-encoder evaluators live in `sentence_transformers.sentence_transformer.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Retrieval (nDCG, MRR, Recall), fast default | `NanoBEIREvaluator` |
+| Retrieval on your own corpus / qrels | `InformationRetrievalEvaluator` |
+| STS / continuous similarity | `EmbeddingSimilarityEvaluator` |
+| Binary classification | `BinaryClassificationEvaluator` |
+| Triplet accuracy | `TripletEvaluator` |
+| Reranking (from retrieval candidates) | `RerankingEvaluator` |
+| MSE vs. teacher (distillation) | `MSEEvaluator`, `MSEEvaluatorFromDataFrame` |
+| Paraphrase mining | `ParaphraseMiningEvaluator` |
+| Translation (cross-lingual alignment) | `TranslationEvaluator` |
+| Label accuracy (classification during training) | `LabelAccuracyEvaluator` |
+
+Wrap multiple evaluators in `SequentialEvaluator` to track all of them together:
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([evaluator1, evaluator2, evaluator3])
+```
+
+## The big three
+
+### `NanoBEIREvaluator` (retrieval)
+
+Small, fast subset of BEIR. Typically runs in <1 minute on a mid-range GPU. Default choice for retrieval training.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import NanoBEIREvaluator
+
+evaluator = NanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
+    batch_size=128,
+    show_progress_bar=False,
+)
+```
+
+- Default dataset list covers 13 tasks. Pick a subset for speed during training.
+- Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_cosine_ndcg@10`** (bi-encoder default = cosine similarity).
+
+### `EmbeddingSimilarityEvaluator` (STS-style)
+
+Computes Pearson/Spearman correlation between model cosine similarities and gold labels.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import EmbeddingSimilarityEvaluator
+from sentence_transformers.util.similarity import SimilarityFunction
+
+evaluator = EmbeddingSimilarityEvaluator(
+    sentences1=stsb["sentence1"],
+    sentences2=stsb["sentence2"],
+    scores=stsb["score"],
+    main_similarity=SimilarityFunction.COSINE,
+    name="sts-dev",
+)
+```
+
+- `main_similarity` can be `COSINE`, `DOT_PRODUCT`, `EUCLIDEAN`, `MANHATTAN`.
+- `name` is used in the output key: `eval_sts-dev_spearman_cosine`, `eval_sts-dev_pearson_cosine`, etc.
+
+### `InformationRetrievalEvaluator` (full retrieval)
+
+Use when you have your **own** corpus + queries + qrels (not one of the NanoBEIR tasks).
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import InformationRetrievalEvaluator
+
+evaluator = InformationRetrievalEvaluator(
+    queries={qid: query_text for qid, query_text in ...},
+    corpus={doc_id: doc_text for doc_id, doc_text in ...},
+    relevant_docs={qid: {doc_id, ...} for qid in ...},   # qid -> set of relevant doc_ids
+    name="my-retrieval",
+    mrr_at_k=[10],
+    ndcg_at_k=[10],
+    accuracy_at_k=[1, 5, 10],
+    precision_recall_at_k=[1, 5, 10],
+    map_at_k=[100],
+    show_progress_bar=False,
+    batch_size=64,
+)
+```
+
+Output keys: `eval_{name}_cosine_ndcg@10`, `eval_{name}_cosine_mrr@10`, etc.
+
+Heavy for large corpora: each eval encodes the full corpus. Don't run it every 100 steps. Use `NanoBEIREvaluator` for frequent evaluation during training and reserve full IR for milestones / post-training.
+
+## Other bi-encoder evaluators
+
+### `BinaryClassificationEvaluator`
+
+For labeled pair classification (e.g. duplicate detection, entailment as binary). Reports accuracy, F1, precision/recall, AP. Supports all distance metrics. Finds the best threshold per metric.
+
+### `TripletEvaluator`
+
+For `(anchor, positive, negative)` triplets. Reports the fraction of triplets where the positive is closer to the anchor than the negative.
+
+### `RerankingEvaluator`
+
+For custom re-ranking datasets: you provide candidates per query, the evaluator computes MAP and MRR. Good for measuring retrieval-quality on a held-out set.
+
+### `MSEEvaluator` / `MSEEvaluatorFromDataFrame`
+
+For distillation setups. Compares student embeddings against teacher embeddings (or teacher scores), reports MSE.
+
+### `ParaphraseMiningEvaluator`
+
+For paraphrase-mining tasks. Given a corpus of labeled paraphrase pairs, computes mining quality (F1 at various thresholds).
+
+### `TranslationEvaluator`
+
+For cross-lingual / `make_multilingual`-style alignment checking. Measures whether the student aligns sentences across languages.
+
+### `LabelAccuracyEvaluator`
+
+For a `SoftmaxLoss`-trained classifier head. Reports accuracy on held-out data.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_mean_cosine_ndcg@10`: `NanoBEIREvaluator`
+- `eval_sts-dev_spearman_cosine`: `EmbeddingSimilarityEvaluator(name="sts-dev")`
+- `eval_{name}_cosine_ndcg@10`: `InformationRetrievalEvaluator(name=...)`
+
+## Multi-dimensional evaluation (Matryoshka)
+
+For Matryoshka-trained models, evaluate at each target dimension:
+
+```python
+per_dim_evaluators = [
+    EmbeddingSimilarityEvaluator(
+        sentences1=..., sentences2=..., scores=...,
+        main_similarity=SimilarityFunction.COSINE,
+        name=f"sts-dev-{dim}",
+        truncate_dim=dim,
+    ) for dim in [768, 512, 256, 128, 64]
+]
+evaluator = SequentialEvaluator(per_dim_evaluators, main_score_function=lambda scores: scores[0])
+```
+
+The first evaluator's score drives `load_best_model_at_end`.
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**: pre-training baseline. If the post-training delta is tiny, the loss/data/base is wrong.
+- Don't run `InformationRetrievalEvaluator` with a large corpus (>100k docs) at frequent `eval_steps`. Use `NanoBEIREvaluator` during training, reserve full IR for end-of-training.
+- `greater_is_better=True` is the default, right for nDCG / MRR / accuracy.

package/.agent/skills/train-sentence-transformers/references/evaluators_sparse_encoder.md

@@ -0,0 +1,121 @@
+# Evaluators (Sparse Encoder)
+
+All sparse-encoder evaluators live in `sentence_transformers.sparse_encoder.evaluation`. They mirror the bi-encoder versions with a `Sparse` prefix and default to **dot product** similarity (cosine on sparse vectors is less meaningful).
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Retrieval (nDCG, MRR, Recall), fast default | `SparseNanoBEIREvaluator` |
+| Retrieval on your own corpus / qrels | `SparseInformationRetrievalEvaluator` |
+| STS / continuous similarity | `SparseEmbeddingSimilarityEvaluator` |
+| Binary classification | `SparseBinaryClassificationEvaluator` |
+| Triplet accuracy | `SparseTripletEvaluator` |
+| Reranking (from retrieval candidates) | `SparseRerankingEvaluator` |
+| MSE vs. teacher (distillation) | `SparseMSEEvaluator` |
+| Translation (cross-lingual alignment) | `SparseTranslationEvaluator` |
+| Hybrid BM25 + sparse retrieval | `ReciprocalRankFusionEvaluator` |
+
+Wrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`):
+
+```python
+from sentence_transformers.base.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([sparse_nano_beir, my_custom_ir])
+```
+
+## The default: `SparseNanoBEIREvaluator`
+
+Small, fast subset of BEIR adapted for sparse retrieval. Typical runtime <1 minute on a mid-range GPU.
+
+```python
+from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator
+
+evaluator = SparseNanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
+    batch_size=32,
+    show_progress_bar=False,
+)
+```
+
+Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_dot_ndcg@10`** (sparse defaults to dot product).
+
+### Sparsity tracking
+
+Unlike the dense variant, the sparse evaluator also reports **active dimension counts** so you can monitor sparsity during training:
+
+- `query_active_dims`: non-zero entries per query vector
+- `document_active_dims`: non-zero entries per document vector
+
+A healthy SPLADE checkpoint typically shows ~30-50 active dims for queries and ~150-250 for documents. If these drift toward the vocab size (~30k), the FLOPS regularization isn't doing its job. Raise `query_regularizer_weight` / `document_regularizer_weight` in `SpladeLoss`.
+
+## Retrieval on your own corpus
+
+### `SparseInformationRetrievalEvaluator`
+
+Same shape as the dense version but operates on sparse vectors internally:
+
+```python
+from sentence_transformers.sparse_encoder.evaluation import SparseInformationRetrievalEvaluator
+
+evaluator = SparseInformationRetrievalEvaluator(
+    queries={qid: text for qid, text in ...},
+    corpus={doc_id: text for doc_id, text in ...},
+    relevant_docs={qid: {doc_id, ...} for qid in ...},
+    name="my-sparse-ir",
+    ndcg_at_k=[10],
+    mrr_at_k=[10],
+    accuracy_at_k=[1, 5, 10],
+    map_at_k=[100],
+    batch_size=32,
+)
+```
+
+Output keys: `eval_{name}_dot_ndcg@10`, `eval_{name}_dot_mrr@10`, etc. Also reports active-dims.
+
+Heavy for large corpora. Use `SparseNanoBEIREvaluator` during training. Reserve full IR for post-training.
+
+## Hybrid retrieval
+
+### `ReciprocalRankFusionEvaluator`
+
+Measures the performance of combining your sparse encoder with BM25 (or any other retriever) via reciprocal-rank fusion. Useful when shipping a hybrid system is the actual deployment target.
+
+## Other sparse evaluators
+
+### `SparseEmbeddingSimilarityEvaluator`
+
+STS-style. Computes Pearson/Spearman between sparse vector similarities and gold labels. Uses dot product by default.
+
+### `SparseBinaryClassificationEvaluator`
+
+For labeled pair classification with sparse embeddings.
+
+### `SparseTripletEvaluator`
+
+For `(anchor, positive, negative)` triplets. Reports fraction where the positive is closer than the negative (by dot product).
+
+### `SparseRerankingEvaluator`
+
+For custom re-ranking with sparse embeddings. Same semantics as the dense `RerankingEvaluator`.
+
+### `SparseMSEEvaluator`
+
+For distillation setups. Compares sparse student embeddings against teacher outputs.
+
+### `SparseTranslationEvaluator`
+
+For cross-lingual / `make_multilingual`-style alignment checking with sparse embeddings.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_mean_dot_ndcg@10`: `SparseNanoBEIREvaluator` default
+- `eval_{name}_dot_ndcg@10`: `SparseInformationRetrievalEvaluator`
+- `eval_{name}_spearman_dot`: `SparseEmbeddingSimilarityEvaluator`
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**. This confirms the pipeline works (a fill-mask base scores ~0 on retrieval until trained).
+- Sparse evaluators default to dot product. Cosine on sparse vectors isn't meaningful.
+- Don't compare dense and sparse metrics directly: different scales (cosine ∈ [-1, 1] vs. dot ∈ [0, ∞)).
+- Always check `query_active_dims` / `document_active_dims`: thousands of active dims per doc means the FLOPS regularizer is mistuned, even if nDCG looks OK.