tribunal-kit 9.2.3 → 9.2.5

package/.agent/skills/train-sentence-transformers/references/evaluators_cross_encoder.md

@@ -0,0 +1,116 @@
+# Evaluators (Cross-Encoder)
+
+All cross-encoder evaluators live in `sentence_transformers.cross_encoder.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Rerank retrieval results (nDCG@k on BM25 top-N), fast default | `CrossEncoderNanoBEIREvaluator` |
+| Rerank with custom candidates per query | `CrossEncoderRerankingEvaluator` |
+| Binary / multi-class pair classification | `CrossEncoderClassificationEvaluator` |
+| Continuous pair scoring (STS-style) | `CrossEncoderCorrelationEvaluator` |
+
+Wrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`) to track them together:
+
+```python
+from sentence_transformers.base.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([nano_beir_eval, custom_rerank_eval])
+```
+
+## The default: `CrossEncoderNanoBEIREvaluator`
+
+Analog of `NanoBEIREvaluator` for rerankers. Takes BM25 top-100 for each NanoBEIR query and measures how well the cross-encoder re-ranks them.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderNanoBEIREvaluator
+
+evaluator = CrossEncoderNanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: 11 of 13 NanoBEIR datasets (excludes "arguana", "touche2020")
+    batch_size=64,
+    rerank_k=100,                                   # rerank the BM25 top-K
+)
+```
+
+Output key for `metric_for_best_model`: **`eval_NanoBEIR_R100_mean_ndcg@10`**. The `R100` signals "rerank top-100". If you change `rerank_k`, the prefix changes (e.g. `R50`).
+
+Each individual dataset contributes `eval_Nano{DatasetName}_R100_ndcg@10` (e.g. `eval_NanoMSMARCO_R100_ndcg@10`) too.
+
+## Custom reranking with your own candidates
+
+Use when you have query + positive + distractor candidates that aren't part of NanoBEIR:
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderRerankingEvaluator
+
+samples = [
+    {"query": "...", "positive": ["the gold answer"], "documents": ["...", "...", ...]}
+    for ...
+]
+
+evaluator = CrossEncoderRerankingEvaluator(
+    samples=samples,
+    batch_size=64,
+    name="my-rerank",
+    always_rerank_positives=False,   # default is True; override to False for realistic eval
+)
+```
+
+- `always_rerank_positives=True` (the library default) forces the positive into the candidate pool even when the retriever missed it. The reranker is graded only on candidates it can actually score, so the metric reflects pure reranker quality.
+- `always_rerank_positives=False`: the positive is only reranked if it's already in `documents`. If the retriever missed it, the rank counts as N+1. This reflects end-to-end retriever+reranker quality. A positive the retriever missed is lost, regardless of reranker skill.
+
+Output key: `eval_{name}_ndcg@10`, `eval_{name}_map`, `eval_{name}_mrr@10`.
+
+## Classification-style cross-encoders
+
+### `CrossEncoderClassificationEvaluator`
+
+Works for both binary (`num_labels=1`) and multi-class (`num_labels>=2`) cross-encoders. Branches internally:
+- `num_labels=1`: binary mode. Sweeps thresholds to report accuracy, F1, precision, recall, and **average_precision** (primary).
+- `num_labels>=2`: multi-class mode (e.g. NLI: entailment / neutral / contradiction). Reports **f1_macro** (primary), f1_micro, f1_weighted, and per-class precision / recall.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderClassificationEvaluator
+
+evaluator = CrossEncoderClassificationEvaluator(
+    sentence_pairs=[(premise, hypothesis), ...],
+    labels=[0, 1, 2, ...],
+    batch_size=64,
+    name="nli-dev",
+)
+```
+
+Output keys (binary, `num_labels=1`): `eval_{name}_accuracy`, `eval_{name}_f1`, `eval_{name}_average_precision` (primary).
+Output keys (multi-class, `num_labels>=2`): `eval_{name}_f1_macro` (primary), `eval_{name}_f1_micro`, `eval_{name}_f1_weighted`.
+
+### `CrossEncoderCorrelationEvaluator`
+
+For continuous-score cross-encoders (like an STS cross-encoder outputting a similarity score). Reports Pearson/Spearman vs. gold scores.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderCorrelationEvaluator
+
+evaluator = CrossEncoderCorrelationEvaluator(
+    sentence_pairs=[(a, b), ...],
+    scores=[0.4, 0.8, ...],
+    name="stsb-dev",
+)
+```
+
+Output keys: `eval_{name}_spearman`, `eval_{name}_pearson`.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_R100_mean_ndcg@10`: `CrossEncoderNanoBEIREvaluator` default
+- `eval_{name}_ndcg@10`: `CrossEncoderRerankingEvaluator`
+- `eval_{name}_average_precision`: `CrossEncoderClassificationEvaluator` (binary, `num_labels=1`)
+- `eval_{name}_f1_macro`: `CrossEncoderClassificationEvaluator` (multi-class, `num_labels>=2`)
+- `eval_{name}_spearman`: `CrossEncoderCorrelationEvaluator`
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**: pre-training baseline. Tiny post-training delta means the loss/data/base is wrong.
+- `CrossEncoderClassificationEvaluator` accepts both `num_labels=1` (binary, primary `average_precision`) and `num_labels>=2` (multi-class, primary `f1_macro`). `CrossEncoderCorrelationEvaluator` requires `num_labels=1`.
+- The default `dataset_names=None` excludes `arguana` and `touche2020` (Argument-Retrieval task differs from the rest). Pass `dataset_names=list(DATASET_NAME_TO_HUMAN_READABLE)` from `sentence_transformers.cross_encoder.evaluation.nano_beir` to actually run all 13.
+- Subset NanoBEIR datasets during training (3-4) to keep eval cheap. Run the broader set post-training.

package/.agent/skills/train-sentence-transformers/references/evaluators_multi_vector_encoder.md

@@ -0,0 +1,101 @@
+# Multi-Vector-Encoder Evaluators
+
+All evaluators live in `sentence_transformers.multi_vector_encoder.evaluation`. They mirror the bi-encoder evaluators but use MaxSim scoring end-to-end.
+
+## Top-line pick
+
+| Task | Evaluator |
+|---|---|
+| Retrieval on a suite of common English IR benchmarks | `MultiVectorNanoBEIREvaluator` |
+| Custom retrieval corpus (your own docs / queries / qrels) | `MultiVectorInformationRetrievalEvaluator` |
+| Distillation from a cross-encoder teacher | `MultiVectorDistillationEvaluator` |
+| Reranking a fixed candidate list per query | `MultiVectorRerankingEvaluator` |
+| Triplet accuracy (does anchor score positive > negative?) | `MultiVectorTripletEvaluator` |
+
+**Default recommendation for training**: `MultiVectorNanoBEIREvaluator` on a subset of NanoBEIR datasets (e.g. `["msmarco", "nq", "fiqa2018"]`) during training, and the full suite at end-of-run. Cheap, well-calibrated, and the metric key format is stable.
+
+## `MultiVectorNanoBEIREvaluator`
+
+Runs the 13 NanoBEIR sub-datasets and returns nDCG@10 averaged across them.
+
+```python
+from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator
+
+evaluator = MultiVectorNanoBEIREvaluator(
+    dataset_names=["msmarco", "nq", "fiqa2018"],  # subset for training-time evals
+    batch_size=16,
+)
+```
+
+**primary_metric**: `"NanoBEIR_mean_maxsim_ndcg@10"` (unless you pass `aggregate_key="..."`, in which case it becomes `"NanoBEIR_{aggregate_key}_maxsim_ndcg@10"`). Note that `NanoBEIREvaluator` composes its display name from `aggregate_key` (and optionally `truncate_dim`), it does not accept a `name` kwarg.
+
+**`metric_for_best_model` key**: `"eval_NanoBEIR_mean_maxsim_ndcg@10"` (add the `eval_` prefix, since Trainer adds it to evaluator metrics).
+
+- `dataset_names`: list of NanoBEIR sub-datasets. Full list: `msmarco`, `nq`, `fiqa2018`, `hotpotqa`, `nfcorpus`, `arguana`, `scidocs`, `climatefever`, `dbpedia`, `fever`, `quoraretrieval`, `scifact`, `touche2020`. Note it is `quoraretrieval`, not `quora`: an invalid name raises at construction.
+- `batch_size`: also drives corpus encoding, so scale to fit memory.
+- English-only. For non-English retrieval, use `MultiVectorInformationRetrievalEvaluator` with your own corpus.
+
+## `MultiVectorInformationRetrievalEvaluator`
+
+Full IR evaluator over a corpus + queries + qrels you supply.
+
+```python
+from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorInformationRetrievalEvaluator
+
+evaluator = MultiVectorInformationRetrievalEvaluator(
+    queries={qid: text, ...},                       # dict of query_id to query_text
+    corpus={did: text, ...},                        # dict of doc_id to doc_text
+    relevant_docs={qid: {did1, did2, ...}, ...},    # qid to set of relevant doc_ids
+    batch_size=16,
+    name="my-eval",                                 # optional, prefixes the metric key
+    write_csv=True,                                 # persists per-eval metrics
+    ndcg_at_k=[10],                                 # customize k-values if you need @20/@100 etc.
+)
+```
+
+**primary_metric**: `"maxsim_ndcg@10"` by default (built from `score_function="maxsim"` and `max(ndcg_at_k)`).
+
+**`metric_for_best_model` key** with `name="my-eval"`:  `"eval_my-eval_maxsim_ndcg@10"`.
+
+- `main_score_function` overrides the score used in `primary_metric` if you set a non-default. Default is `None`, which resolves to `maxsim` from the model's `similarity_fn_name` at call time. For MVE just leave it as `None`.
+- Reports MRR@k, nDCG@k, Recall@k, Precision@k, MAP@k for the k-values you pass (`mrr_at_k`, `ndcg_at_k`, etc.).
+- `chunk_elements` (default `None`) is the element budget for the padded `(chunk, d_tokens, dim)` documents plus the `(num_queries, chunk, q_tokens, d_tokens)` MaxSim scoring intermediate. Leave it as `None`: `maxsim` packs documents into chunks under a 100M-element budget (at most ~400 MB, half that in bf16 / fp16), adapting to the query count and document lengths. Lower it to cut evaluation memory further. It is a separate axis from `corpus_chunk_size`, which caps how many encoded documents are held at a time.
+
+## `MultiVectorDistillationEvaluator`
+
+Regression of student MaxSim scores against teacher scores over a held-out set of `(query, doc, teacher_score)` rows. Use it when you're distilling.
+
+**primary_metric**: `"spearman"` (correlation of student scores with teacher scores).
+
+**`metric_for_best_model` key** with `name="my-eval"`: `"eval_my-eval_spearman"`.
+
+- Pair with `MultiVectorMarginMSELoss` or `MultiVectorDistillKLDivLoss` during training.
+- Higher is better (student ranking matches teacher).
+
+## `MultiVectorRerankingEvaluator`
+
+Reranks a fixed list of candidates per query. Same shape as the bi-encoder `RerankingEvaluator`, MaxSim-scored.
+
+- **Data**: `samples = [{"query": ..., "positive": [...], "negative": [...]} , ...]`.
+- Reports MAP, MRR@10, nDCG@10 by default.
+
+## `MultiVectorTripletEvaluator`
+
+Fraction of `(anchor, positive, negative)` triplets where `sim(anchor, positive) > sim(anchor, negative)`. Cheap sanity signal, not what you want to gate a real IR release on.
+
+## Named-evaluator metric key format (universal)
+
+If you pass `name="..."` to any of these, the metric key format is:
+
+```
+eval_{name}_{primary_metric}
+```
+
+If you don't pass `name`, the key is just `eval_{primary_metric}`. The trainer's `metric_for_best_model=...` must match exactly, or `load_best_model_at_end` selects the wrong checkpoint. **Run the evaluator once before training** to observe the exact key format: `primary_metric` is `None` until the first `evaluator(model)` call, which composes it from the resolved score function and the `ndcg_at_k` you passed, then prefixes `name`. The production template does exactly this before building `TrainingArguments`.
+
+## Gotchas
+
+- **Metric key mismatch on `metric_for_best_model`**: silent failure. Training runs to completion and `load_best_model_at_end` picks a stale checkpoint. Always print `evaluator(model)` output once before starting the trainer and confirm the key format.
+- **Eval-time OOM**: MaxSim scoring builds `(num_queries, chunk, q_tokens, d_tokens)` intermediates, which `maxsim` bounds by default under its 100M-element budget. If you still OOM, lower `chunk_elements`, then drop `batch_size`. NanoBEIR corpora can be small enough that oversized `batch_size` bites before you'd expect.
+- **`MultiVectorNanoBEIREvaluator` with a Non-English base**: it's English-only. You'll get zero or garbage scores. Use `MultiVectorInformationRetrievalEvaluator` with an in-domain corpus.
+- **Distillation eval on the same data as training**: measures memorization, not generalization. Hold out a separate `(query, doc, teacher_score)` split.

package/.agent/skills/train-sentence-transformers/references/evaluators_sentence_transformer.md

@@ -0,0 +1,151 @@
+# Evaluators (Bi-Encoder)
+
+All bi-encoder evaluators live in `sentence_transformers.sentence_transformer.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Retrieval (nDCG, MRR, Recall), fast default | `NanoBEIREvaluator` |
+| Retrieval on your own corpus / qrels | `InformationRetrievalEvaluator` |
+| STS / continuous similarity | `EmbeddingSimilarityEvaluator` |
+| Binary classification | `BinaryClassificationEvaluator` |
+| Triplet accuracy | `TripletEvaluator` |
+| Reranking (from retrieval candidates) | `RerankingEvaluator` |
+| MSE vs. teacher (distillation) | `MSEEvaluator`, `MSEEvaluatorFromDataFrame` |
+| Paraphrase mining | `ParaphraseMiningEvaluator` |
+| Translation (cross-lingual alignment) | `TranslationEvaluator` |
+| Label accuracy (classification during training) | `LabelAccuracyEvaluator` |
+
+Wrap multiple evaluators in `SequentialEvaluator` to track all of them together:
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([evaluator1, evaluator2, evaluator3])
+```
+
+## The big three
+
+### `NanoBEIREvaluator` (retrieval)
+
+Small, fast subset of BEIR. Typically runs in <1 minute on a mid-range GPU. Default choice for retrieval training.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import NanoBEIREvaluator
+
+evaluator = NanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
+    batch_size=128,
+    show_progress_bar=False,
+)
+```
+
+- Default dataset list covers 13 tasks. Pick a subset for speed during training.
+- Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_cosine_ndcg@10`** (bi-encoder default = cosine similarity).
+
+### `EmbeddingSimilarityEvaluator` (STS-style)
+
+Computes Pearson/Spearman correlation between model cosine similarities and gold labels.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import EmbeddingSimilarityEvaluator
+from sentence_transformers.util.similarity import SimilarityFunction
+
+evaluator = EmbeddingSimilarityEvaluator(
+    sentences1=stsb["sentence1"],
+    sentences2=stsb["sentence2"],
+    scores=stsb["score"],
+    main_similarity=SimilarityFunction.COSINE,
+    name="sts-dev",
+)
+```
+
+- `main_similarity` can be `COSINE`, `DOT_PRODUCT`, `EUCLIDEAN`, `MANHATTAN`.
+- `name` is used in the output key: `eval_sts-dev_spearman_cosine`, `eval_sts-dev_pearson_cosine`, etc.
+
+### `InformationRetrievalEvaluator` (full retrieval)
+
+Use when you have your **own** corpus + queries + qrels (not one of the NanoBEIR tasks).
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import InformationRetrievalEvaluator
+
+evaluator = InformationRetrievalEvaluator(
+    queries={qid: query_text for qid, query_text in ...},
+    corpus={doc_id: doc_text for doc_id, doc_text in ...},
+    relevant_docs={qid: {doc_id, ...} for qid in ...},   # qid -> set of relevant doc_ids
+    name="my-retrieval",
+    mrr_at_k=[10],
+    ndcg_at_k=[10],
+    accuracy_at_k=[1, 5, 10],
+    precision_recall_at_k=[1, 5, 10],
+    map_at_k=[100],
+    show_progress_bar=False,
+    batch_size=64,
+)
+```
+
+Output keys: `eval_{name}_cosine_ndcg@10`, `eval_{name}_cosine_mrr@10`, etc.
+
+Heavy for large corpora: each eval encodes the full corpus. Don't run it every 100 steps. Use `NanoBEIREvaluator` for frequent evaluation during training and reserve full IR for milestones / post-training.
+
+## Other bi-encoder evaluators
+
+### `BinaryClassificationEvaluator`
+
+For labeled pair classification (e.g. duplicate detection, entailment as binary). Reports accuracy, F1, precision/recall, AP. Supports all distance metrics. Finds the best threshold per metric.
+
+### `TripletEvaluator`
+
+For `(anchor, positive, negative)` triplets. Reports the fraction of triplets where the positive is closer to the anchor than the negative.
+
+### `RerankingEvaluator`
+
+For custom re-ranking datasets: you provide candidates per query, the evaluator computes MAP and MRR. Good for measuring retrieval-quality on a held-out set.
+
+### `MSEEvaluator` / `MSEEvaluatorFromDataFrame`
+
+For distillation setups. Compares student embeddings against teacher embeddings (or teacher scores), reports MSE.
+
+### `ParaphraseMiningEvaluator`
+
+For paraphrase-mining tasks. Given a corpus of labeled paraphrase pairs, computes mining quality (F1 at various thresholds).
+
+### `TranslationEvaluator`
+
+For cross-lingual / `make_multilingual`-style alignment checking. Measures whether the student aligns sentences across languages.
+
+### `LabelAccuracyEvaluator`
+
+For a `SoftmaxLoss`-trained classifier head. Reports accuracy on held-out data.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_mean_cosine_ndcg@10`: `NanoBEIREvaluator`
+- `eval_sts-dev_spearman_cosine`: `EmbeddingSimilarityEvaluator(name="sts-dev")`
+- `eval_{name}_cosine_ndcg@10`: `InformationRetrievalEvaluator(name=...)`
+
+## Multi-dimensional evaluation (Matryoshka)
+
+For Matryoshka-trained models, evaluate at each target dimension:
+
+```python
+per_dim_evaluators = [
+    EmbeddingSimilarityEvaluator(
+        sentences1=..., sentences2=..., scores=...,
+        main_similarity=SimilarityFunction.COSINE,
+        name=f"sts-dev-{dim}",
+        truncate_dim=dim,
+    ) for dim in [768, 512, 256, 128, 64]
+]
+evaluator = SequentialEvaluator(per_dim_evaluators, main_score_function=lambda scores: scores[0])
+```
+
+The first evaluator's score drives `load_best_model_at_end`.
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**: pre-training baseline. If the post-training delta is tiny, the loss/data/base is wrong.
+- Don't run `InformationRetrievalEvaluator` with a large corpus (>100k docs) at frequent `eval_steps`. Use `NanoBEIREvaluator` during training, reserve full IR for end-of-training.
+- `greater_is_better=True` is the default, right for nDCG / MRR / accuracy.

package/.agent/skills/train-sentence-transformers/references/evaluators_sparse_encoder.md

@@ -0,0 +1,121 @@
+# Evaluators (Sparse Encoder)
+
+All sparse-encoder evaluators live in `sentence_transformers.sparse_encoder.evaluation`. They mirror the bi-encoder versions with a `Sparse` prefix and default to **dot product** similarity (cosine on sparse vectors is less meaningful).
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Retrieval (nDCG, MRR, Recall), fast default | `SparseNanoBEIREvaluator` |
+| Retrieval on your own corpus / qrels | `SparseInformationRetrievalEvaluator` |
+| STS / continuous similarity | `SparseEmbeddingSimilarityEvaluator` |
+| Binary classification | `SparseBinaryClassificationEvaluator` |
+| Triplet accuracy | `SparseTripletEvaluator` |
+| Reranking (from retrieval candidates) | `SparseRerankingEvaluator` |
+| MSE vs. teacher (distillation) | `SparseMSEEvaluator` |
+| Translation (cross-lingual alignment) | `SparseTranslationEvaluator` |
+| Hybrid BM25 + sparse retrieval | `ReciprocalRankFusionEvaluator` |
+
+Wrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`):
+
+```python
+from sentence_transformers.base.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([sparse_nano_beir, my_custom_ir])
+```
+
+## The default: `SparseNanoBEIREvaluator`
+
+Small, fast subset of BEIR adapted for sparse retrieval. Typical runtime <1 minute on a mid-range GPU.
+
+```python
+from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator
+
+evaluator = SparseNanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
+    batch_size=32,
+    show_progress_bar=False,
+)
+```
+
+Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_dot_ndcg@10`** (sparse defaults to dot product).
+
+### Sparsity tracking
+
+Unlike the dense variant, the sparse evaluator also reports **active dimension counts** so you can monitor sparsity during training:
+
+- `query_active_dims`: non-zero entries per query vector
+- `document_active_dims`: non-zero entries per document vector
+
+A healthy SPLADE checkpoint typically shows ~30-50 active dims for queries and ~150-250 for documents. If these drift toward the vocab size (~30k), the FLOPS regularization isn't doing its job. Raise `query_regularizer_weight` / `document_regularizer_weight` in `SpladeLoss`.
+
+## Retrieval on your own corpus
+
+### `SparseInformationRetrievalEvaluator`
+
+Same shape as the dense version but operates on sparse vectors internally:
+
+```python
+from sentence_transformers.sparse_encoder.evaluation import SparseInformationRetrievalEvaluator
+
+evaluator = SparseInformationRetrievalEvaluator(
+    queries={qid: text for qid, text in ...},
+    corpus={doc_id: text for doc_id, text in ...},
+    relevant_docs={qid: {doc_id, ...} for qid in ...},
+    name="my-sparse-ir",
+    ndcg_at_k=[10],
+    mrr_at_k=[10],
+    accuracy_at_k=[1, 5, 10],
+    map_at_k=[100],
+    batch_size=32,
+)
+```
+
+Output keys: `eval_{name}_dot_ndcg@10`, `eval_{name}_dot_mrr@10`, etc. Also reports active-dims.
+
+Heavy for large corpora. Use `SparseNanoBEIREvaluator` during training. Reserve full IR for post-training.
+
+## Hybrid retrieval
+
+### `ReciprocalRankFusionEvaluator`
+
+Measures the performance of combining your sparse encoder with BM25 (or any other retriever) via reciprocal-rank fusion. Useful when shipping a hybrid system is the actual deployment target.
+
+## Other sparse evaluators
+
+### `SparseEmbeddingSimilarityEvaluator`
+
+STS-style. Computes Pearson/Spearman between sparse vector similarities and gold labels. Uses dot product by default.
+
+### `SparseBinaryClassificationEvaluator`
+
+For labeled pair classification with sparse embeddings.
+
+### `SparseTripletEvaluator`
+
+For `(anchor, positive, negative)` triplets. Reports fraction where the positive is closer than the negative (by dot product).
+
+### `SparseRerankingEvaluator`
+
+For custom re-ranking with sparse embeddings. Same semantics as the dense `RerankingEvaluator`.
+
+### `SparseMSEEvaluator`
+
+For distillation setups. Compares sparse student embeddings against teacher outputs.
+
+### `SparseTranslationEvaluator`
+
+For cross-lingual / `make_multilingual`-style alignment checking with sparse embeddings.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_mean_dot_ndcg@10`: `SparseNanoBEIREvaluator` default
+- `eval_{name}_dot_ndcg@10`: `SparseInformationRetrievalEvaluator`
+- `eval_{name}_spearman_dot`: `SparseEmbeddingSimilarityEvaluator`
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training**. This confirms the pipeline works (a fill-mask base scores ~0 on retrieval until trained).
+- Sparse evaluators default to dot product. Cosine on sparse vectors isn't meaningful.
+- Don't compare dense and sparse metrics directly: different scales (cosine ∈ [-1, 1] vs. dot ∈ [0, ∞)).
+- Always check `query_active_dims` / `document_active_dims`: thousands of active dims per doc means the FLOPS regularizer is mistuned, even if nDCG looks OK.

package/.agent/skills/train-sentence-transformers/references/hardware_guide.md

@@ -0,0 +1,105 @@
+# Hardware Guide
+
+Training embedding models is memory-bound more often than compute-bound.
+
+## If you hit OOM
+
+Try in this order:
+
+1. **Reduce `per_device_train_batch_size`**. Raise `gradient_accumulation_steps` to keep the effective batch size for regression losses. (For MNRL, effective batch via grad-accum is **not** equivalent. See point 3.)
+2. **Enable `gradient_checkpointing=True`**. ~30% slower, ~40% less activation memory. Incompatible with `Cached*` losses.
+3. **Switch to a `Cached*` loss**:
+   - `CachedMultipleNegativesRankingLoss(model, mini_batch_size=32)`: forwards in mini-batches, accumulates the contrastive loss over the full batch. Can simulate batch sizes of 1024+ on a 24GB GPU.
+   - `CachedSpladeLoss(model, loss=..., mini_batch_size=16)`: same trick for sparse.
+   - `CachedGISTEmbedLoss(model, guide_model, mini_batch_size=32)`: GIST variant.
+4. **Enable PEFT / LoRA** for decoder models >1B. `LoraConfig(r=64, lora_alpha=128, task_type="FEATURE_EXTRACTION")`. See `../scripts/train_sentence_transformer_with_lora_example.py` (docstring covers when to use, hyperparams, QLoRA, sharing).
+5. **Move to multi-GPU**. See below.
+6. **Shorten sequences**. If truncating to 128 is already sufficient for your task, set `max_seq_length` on the transformer module.
+
+## Multi-GPU
+
+`sentence-transformers` uses `accelerate` under the hood. Distributed training works without code changes.
+
+### Data parallel (DDP)
+
+Launch:
+
+```bash
+accelerate launch train.py
+
+# or explicitly:
+accelerate launch --multi_gpu --num_processes=4 train.py
+```
+
+`per_device_train_batch_size` stays per-GPU. Effective batch size scales linearly. MNRL's in-batch negatives remain **per-device**, not global, unless you pass `gather_across_devices=True` to losses that support it (`MultipleNegativesRankingLoss`, `CachedMultipleNegativesRankingLoss`, the symmetric variants, `GISTEmbedLoss`, `CachedGISTEmbedLoss`, `SparseMultipleNegativesRankingLoss`).
+
+### FSDP / DeepSpeed
+
+For models >3B, use `accelerate config` to enable FSDP or DeepSpeed ZeRO. Both are supported: `sentence-transformers` doesn't require any code changes, only the launch config.
+
+```bash
+accelerate config                   # interactive; choose FSDP or DeepSpeed
+accelerate launch train.py
+```
+
+With FSDP full-shard: a 7B model trains on 4×24GB GPUs that would OOM on any single one of them.
+
+**FSDP caveats** (from the [distributed training docs](https://sbert.net/docs/sentence_transformer/training/distributed.html)):
+- **Evaluators don't run under FSDP** as of writing: the eval hooks call `model.encode()` which FSDP-wrapped modules can't service mid-training. Plan to evaluate post-training on a single-GPU load of the final checkpoint instead, or train with DDP if you need mid-training evaluation.
+- **Layer wrapping must be specified**, e.g. `fsdp_config={"transformer_layer_cls_to_wrap": "BertLayer"}` (substitute the right layer class for your model: `BertLayer`, `LlamaDecoderLayer`, `Qwen2DecoderLayer`, etc.). Without this FSDP sharding can silently misbehave.
+- **Slower than DDP** for models that fit on a single GPU. Only reach for FSDP when you actually need the memory savings.
+
+DeepSpeed ZeRO-2/3 is an alternative with its own config. It works identically at the `accelerate config` level.
+
+## Effective batch size for contrastive losses
+
+For `MultipleNegativesRankingLoss` and its variants, **batch size is a quality knob**, not just a speed knob. Bigger batches = more in-batch negatives = richer gradients.
+
+The effective pool of in-batch negatives per anchor:
+
+| Setup | In-batch negatives per anchor |
+|---|---|
+| Single GPU, batch 64 | 63 |
+| 4× DDP, per-device batch 64 | 63 local only by default. 255 with `MultipleNegativesRankingLoss(model, gather_across_devices=True)` |
+| Single GPU, CachedMNRL, mini_batch 32, batch 256 | 255 |
+| 4× DDP, CachedMNRL, per-device 256 | 255 local. 1023 with `gather_across_devices=True` |
+
+For large corpora (retrieval), push toward 512+ effective negatives. For small, clean datasets (STS), 64 is plenty.
+
+## Precision choice by GPU
+
+| GPU generation | Recommended |
+|---|---|
+| T4, V100, GTX 1xxx, RTX 2xxx | `fp16=True` |
+| RTX 3xxx, A10G, A100, L4 | `bf16=True` |
+| RTX 4xxx, H100, B200 | `bf16=True` (or fp8 on H100 via specific kernels, not default) |
+| Apple M-series / ROCm | MPS/ROCm support is variable. `fp16` or `fp32` most reliable |
+
+bf16 is more numerically stable and almost always preferred when available.
+
+## Hugging Face Jobs flavor guide
+
+Hugging Face Jobs requires a Pro/Team/Enterprise plan. Pricing is approximate and subject to change. See the [Jobs pricing page](https://huggingface.co/docs/huggingface_hub/guides/jobs).
+
+| Flavor | Memory | Typical use | Est. $/hr |
+|---|---|---|---|
+| `cpu-basic` | ~2 GB | Dataset prep, validation, hard-neg mining (small) | <$0.10 |
+| `cpu-upgrade` | ~4 GB | Same, slightly bigger | $0.10 |
+| `t4-small` | 16 GB | Demos, MiniLM/DistilBERT with small batches | ~$0.75 |
+| `t4-medium` | 16 GB | MiniLM / DistilBERT with larger batch | ~$1.50 |
+| `l4x1` | 24 GB | BERT-base, MPNet, ModernBERT-base | ~$2.50 |
+| `a10g-small` | 24 GB | BERT-base to BERT-large | ~$3.50 |
+| `a10g-large` | 48 GB | ModernBERT-large, Qwen3-0.6B | ~$5.00 |
+| `a10g-largex2` | 96 GB (2× 48GB) | Mid-size multi-GPU | ~$10 |
+| `a100-large` | 80 GB | Large models or big contrastive batches | ~$10-12 |
+| `h100` | 80 GB | Biggest single-GPU | ~$12 |
+| `h100x8` | 640 GB | LLM-scale distributed | ~$96 |
+
+Defaults by base model:
+- MiniLM / DistilBERT -> `t4-small`
+- BERT-base / MPNet / ModernBERT-base -> `a10g-small` or `l4x1`
+- BERT-large / ModernBERT-large -> `a10g-large`
+- Qwen3-0.6B decoder base -> `a10g-large`
+- 1B+ decoder bases with LoRA -> `a10g-large` or `a100-large`
+
+Always start one flavor **smaller** than you think you need: OOM on Jobs is cheap ($0.50-$5 for a failed run). Underprovisioned is better than overprovisioned for the first attempt. When budgeting `timeout`, add **20-30% buffer** for model loading, checkpoint saving, and Hub push.