opencode-skills-collection 3.1.12 → 3.1.14

package/bundled-skills/train-sentence-transformers/references/losses_cross_encoder.md

@@ -0,0 +1,137 @@
+# Cross-Encoder Losses (reranker)
+
+All losses live in `sentence_transformers.cross_encoder.losses`.
+
+Cross-encoder losses fall into three families: **pointwise** (score each pair independently), **pairwise** (score pairs of pairs), and **listwise** (score a whole ranked list at once). Plus contrastive/distillation variants for specific setups.
+
+## Top-line decision table
+
+| You have | Use | Family |
+|---|---|---|
+| `(query, passage, label)` with `label ∈ {0, 1}` or `[0, 1]` | `BinaryCrossEntropyLoss` | Pointwise |
+| `(query, passage, class_id)` multi-class | `CrossEntropyLoss` | Pointwise |
+| `(query, passage)` with implicit positives, want contrastive | `CachedMultipleNegativesRankingLoss` | Contrastive |
+| `(query, passages, labels)` (one row per query, with parallel lists of candidate passages and their relevance scores) | `LambdaLoss` | Listwise |
+| Same listwise shape, want well-tested simpler loss | `ListNetLoss` or `ListMLELoss` | Listwise |
+| `(query, positive, negative)` pairwise | `RankNetLoss` | Pairwise |
+| `(query, passage, teacher_score)` distillation from stronger reranker | `MSELoss` or `MarginMSELoss` | Distillation |
+
+## Pointwise losses
+
+### `BinaryCrossEntropyLoss`
+
+**The default cross-encoder loss.** For `(query, passage, label)` where label is 0/1 or a continuous [0, 1] relevance score.
+
+```python
+loss = BinaryCrossEntropyLoss(model, pos_weight=torch.tensor(5.0))
+```
+
+- `pos_weight`: upweight positives when your dataset is imbalanced (e.g. 1 positive + N hard negatives). A good default is `pos_weight = num_hard_negatives`.
+- Handles both binary labels and continuous relevance scores — if you have graded relevance (0.0, 0.5, 1.0) BCE still works.
+
+### `CrossEntropyLoss`
+
+Multi-class classification over passages. Use with `CrossEncoder("...", num_labels=N)` where N is the number of classes (e.g. 3 for NLI: entailment / neutral / contradiction).
+
+## Contrastive losses (for reranker training)
+
+### `MultipleNegativesRankingLoss` (cross-encoder)
+
+Contrastive training mirroring bi-encoder MNRL. Applies in-batch negatives: for each `(query, positive)`, every other `positive` in the batch is used as a negative.
+
+- **Data**: `(query, positive)` or `(query, positive, negative_0, negative_1, ...)`.
+- For CrossEncoder training, **prefer `BinaryCrossEntropyLoss` with mined hard negatives** as the default; MNRL is the fallback when you only have `(query, positive)` pairs and want to avoid a separate mining pass.
+
+### `CachedMultipleNegativesRankingLoss`
+
+Cached variant (GradCache) — decouples per-device batch size from effective in-batch negatives, same as the bi-encoder cached version.
+
+```python
+loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16)
+```
+
+- Incompatible with `gradient_checkpointing=True`.
+- Key choice for training rerankers with effective batch 256+ on a single GPU.
+
+## Distillation losses
+
+> **`activation_fn=nn.Identity()` is mandatory** for all distillation, listwise, and pairwise losses below — only `BinaryCrossEntropyLoss` and `CrossEntropyLoss` tolerate the default `Sigmoid`. The loss sees raw logits during training, but the model's `activation_fn` is applied at evaluation via `predict()`; the default `Sigmoid` (with `num_labels=1`) saturates raw logits >5 to ~1.0 inside `predict()`, silently collapsing eval ranking (training loss looks healthy while nDCG crashes from ~0.59 to ~0.14). Construct as `CrossEncoder("...", num_labels=1, activation_fn=torch.nn.Identity())`. See `troubleshooting.md` ("CrossEncoder eval nDCG crashes after distillation / listwise / pairwise training") for the failure-mode walkthrough.
+
+### `MSELoss` (cross-encoder)
+
+Regress the student's output score to the teacher's score. Construct the model with `activation_fn=nn.Identity()` (see callout above).
+
+- **Data**: `(query, passage, teacher_score)`.
+- The teacher is typically a larger/stronger cross-encoder. Precompute its scores once, store as the label.
+
+### `MarginMSELoss` (cross-encoder)
+
+Regress the **difference** between positive and negative scores against the teacher's margin. Typically gives better distillation than plain MSE.
+
+- **Data**: `(query, positive, negative, score_diff)` where `score_diff = teacher_score(query, positive) - teacher_score(query, negative)`.
+- Popular recipe for MS MARCO-style distillation.
+
+## Listwise losses
+
+All listwise losses expect the dataset to have per-query **lists** of candidate passages and their scores — typically via a collator that groups rows by query.
+
+> Listwise and pairwise losses below also require `activation_fn=nn.Identity()` (see the Distillation-section callout above).
+
+### `LambdaLoss`
+
+The state-of-the-art listwise ranking loss. Optimizes a surrogate of nDCG via weighted pairwise comparisons.
+
+**Data shape**: `(query, [doc1...docN], [score1...scoreN])` per row; One query, a list of candidate documents, and a parallel list of relevance scores. Use `mine_hard_negatives(..., output_format="labeled-list", ...)` to build this from `(query, positive)` pairs.
+
+```python
+import torch.nn as nn
+from sentence_transformers.cross_encoder.losses import LambdaLoss, NDCGLoss2PPScheme
+
+model = CrossEncoder("...", num_labels=1, activation_fn=nn.Identity())
+loss = LambdaLoss(model, weighting_scheme=NDCGLoss2PPScheme())
+```
+
+- Strong default when you have multiple candidates per query and graded relevance.
+- `weighting_scheme`: `NDCGLoss2PPScheme` (default), `NDCGLoss2Scheme`, `LambdaRankScheme`. The `NDCGLoss2PPScheme` default was shown to reach the strongest performance in the original LambdaLoss paper.
+
+#### LambdaLoss-specific operational notes
+
+- **OOM recovery order**: drop `mini_batch_size` first (chunking inside the loss preserves the K-list semantic), then `per_device_train_batch_size` paired with `gradient_accumulation_steps`, then reduce K (the per-query candidate-list length) only as a last resort. Lowering K changes the experiment; the others don't.
+- **Tiny loss at large K is expected.** With `NDCGLoss2PPScheme`, the loss normalizes by the discount-weighted pair count — at K=128 the loss can scale to ~`1e-4` numerically. That's not "training broken"; read `eval_NanoBEIR_R100_mean_ndcg@10` (or your equivalent) instead of the training loss to judge progress.
+- For very large K (>=128), the O(K²) weight buffer per query is materialized outside of the forward chunking, so even small `mini_batch_size` may not be enough. Consider scoring strategy: top-K hard negatives often beat random K.
+
+### `ListNetLoss`
+
+Treats the ranking as a probability distribution (via softmax) and minimizes cross-entropy against the teacher distribution.
+
+### `ListMLELoss`
+
+Maximum-likelihood estimation over permutations. Simpler than LambdaLoss; decent default.
+
+### `PListMLELoss`
+
+Position-aware ListMLE — weights higher-ranked items more heavily. Often outperforms plain ListMLE on top-k metrics.
+
+### `RankNetLoss`
+
+Pairwise classification: for each pair of candidates, predict which is ranked higher via cross-entropy.
+
+- Simpler and faster than LambdaLoss.
+- Scales quadratically with list length; not great for long candidate lists (>20).
+
+### `ADRMSELoss`
+
+Alternative listwise formulation (Approx Discounted Rank MSE) from the Rank-DistiLLM paper. Same data shape as `LambdaLoss`. In practice LambdaLoss is the stronger default; `RankNetLoss` was reported as marginally more effective (~0.002 nDCG@10) than ADRMSELoss on the paper's LLM-distillation setup, and LambdaLoss generally beats both.
+
+Hard-negative mining is essential for any contrastive reranker (random negatives teach nothing). See `dataset_formats.md` (Hard-negative mining section) and `../scripts/mine_hard_negatives.py`.
+
+## Gotchas
+
+- **`BinaryCrossEntropyLoss` without `pos_weight`** when you have 5+ hard negatives per positive: the loss under-weights the positive signal. Set `pos_weight=num_hard_negatives`.
+- **`CachedMultipleNegativesRankingLoss` + `gradient_checkpointing=True`**: crash. Pick one.
+- **Listwise losses with wildly different list lengths per query**: some losses don't handle ragged lists well. Pad or truncate to a fixed length.
+- **`MarginMSELoss` without precomputed teacher score diffs**: this loss needs the `score_diff` label column populated from a teacher pass; it does not compute the teacher internally.
+- **Training a cross-encoder with `num_labels=1` then using `CrossEntropyLoss`**: mismatch — BCE is for num_labels=1, CE is for num_labels>=2.
+- **Default `Sigmoid` activation under any non-BCE loss**: silently destroys eval ranking. Pass `activation_fn=torch.nn.Identity()` to `CrossEncoder(...)` for distillation, listwise, and pairwise losses (everything except BCE/CE). See the callouts in the Distillation and Listwise sections.
+- **Custom CE head writing to the wrong feature key**: a custom scoring head must populate `features["scores"]` (not `features["sentence_embedding"]`). Otherwise `CrossEncoder.predict()` raises `KeyError: 'scores'` at inference time, even though training succeeds.
+- **Loading a CE with a custom-class head fails from a different script**: if you defined a `class ClassifierHead(nn.Module)` inline in `train.py` and saved the model, `modules.json` records `__main__.ClassifierHead`. Loading via `CrossEncoder("path")` from any other entry point raises `ImportError: Module '__main__' does not define a 'ClassifierHead' attribute`. Either move the class into an importable file (`my_pkg/heads.py`), build the same shape from stock ST modules (`Dense + LayerNorm + Dense`), or document that the model is only loadable from the same script.

package/bundled-skills/train-sentence-transformers/references/losses_sentence_transformer.md

@@ -0,0 +1,246 @@
+# Bi-Encoder Losses (SentenceTransformer)
+
+All losses live in `sentence_transformers.sentence_transformer.losses`.
+
+Losses are grouped by data shape. The #1 rule: **pick a loss that matches your data**, not the other way around.
+
+## Top-line decision table
+
+| You have | Use |
+|---|---|
+| `(anchor, positive)` pairs | `MultipleNegativesRankingLoss` (or Cached variant for large batches) |
+| `(anchor, positive, negative)` triplets | `MultipleNegativesRankingLoss` — it handles triplets natively |
+| `(text1, text2, score)` with `score ∈ [-1, 1]` or `[0, 1]` | `CoSENTLoss` (strongly recommended) |
+| `(text1, text2, label)` with `label ∈ {0, 1}` | `OnlineContrastiveLoss` |
+| `(text, class_id)` single-column with integer class | `BatchAllTripletLoss` |
+| `(query, positive, negative, score_diff)` | `MarginMSELoss` (distillation) |
+| `(text, teacher_embedding)` | `MSELoss` (embedding distillation) |
+| Want multiple output dims from one training | Wrap any of the above in `MatryoshkaLoss` |
+| No labels at all, just sentences | `DenoisingAutoEncoderLoss` or `ContrastiveTensionLossInBatchNegatives` |
+
+## Contrastive losses (pairs + triplets, no labels)
+
+### `MultipleNegativesRankingLoss` (MNRL)
+
+The default bi-encoder loss. Uses **in-batch negatives**: every other `positive` in the batch acts as a negative for the current `anchor`.
+
+```python
+loss = MultipleNegativesRankingLoss(model, scale=20.0)  # similarity_fct defaults to cos_sim
+```
+
+- **Data**: `(anchor, positive)` or `(anchor, positive, negative)`. More columns = more explicit hard negatives per row.
+- **Scale**: temperature. Default `scale=20.0` multiplies similarities by 20 (equivalent to softmax temperature 0.05). Tune only if cosine similarities end up saturated.
+- **Critical**: set `batch_sampler=BatchSamplers.NO_DUPLICATES` on training args. Otherwise duplicate anchors create false negatives.
+- **Tip**: batch size matters a lot — more in-batch negatives = better gradients.
+
+### `CachedMultipleNegativesRankingLoss`
+
+Same loss, but with gradient caching (GradCache): forwards in mini-batches but computes the contrastive loss over the full batch. Use this when you want effective batch size of 256+ but your GPU can only fit 32 forwards.
+
+```python
+loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=32)
+```
+
+- **Incompatible with `gradient_checkpointing=True`**.
+- Set `mini_batch_size` to whatever `per_device_train_batch_size` would be if you couldn't use this. Then crank the actual `per_device_train_batch_size` to what you want the effective batch to be (256+, 1024+).
+
+### `MultipleNegativesSymmetricRankingLoss`
+
+MNRL computed bidirectionally — scores positives from both (anchor -> positive) and (positive -> anchor) directions. Slightly better on retrieval tasks where the "anchor" and "positive" distinctions are soft (paraphrase, deduplication).
+
+### `CachedMultipleNegativesSymmetricRankingLoss`
+
+Cached variant of the above.
+
+### `GISTEmbedLoss`
+
+Like MNRL, but uses a **guide model** (a separate pretrained Sentence Transformer) to **filter out false negatives** before computing the contrastive loss. The guide model scores each potential negative; if it looks too similar to the positive, it's excluded.
+
+```python
+guide = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+loss = GISTEmbedLoss(model, guide=guide)
+```
+
+- Expensive: guide model does a forward pass per batch.
+- Strong when your in-batch negatives are noisy (e.g. small corpus with many near-duplicates).
+
+### `CachedGISTEmbedLoss`
+
+Cached + GIST.
+
+### `MegaBatchMarginLoss`
+
+In-batch margin-based triplet: for each anchor, find the hardest negative in the batch and apply a margin loss. Older pattern, usually outperformed by MNRL.
+
+### `TripletLoss`
+
+Classic triplet margin loss on explicit `(anchor, positive, negative)`. Uses a fixed margin and the hardest in-batch is not considered — only the provided triplet.
+
+```python
+loss = TripletLoss(model, distance_metric=TripletDistanceMetric.EUCLIDEAN, triplet_margin=5)
+```
+
+- Simpler than MNRL; less powerful when the batch has useful negatives.
+- Good for cases where you have *trusted* pre-mined triplets and want to avoid in-batch noise.
+
+## Batch-triplet losses (single column + integer label)
+
+These mine triplets *within* the batch from samples sharing a label.
+
+### `BatchAllTripletLoss`
+
+For each anchor, form triplets with all positive/negative combinations in the batch. Max signal per batch.
+
+```python
+loss = BatchAllTripletLoss(model, margin=5)
+```
+
+- **Data**: single text column + integer `label`. Needs multiple samples per label in each batch (set `batch_sampler=BatchSamplers.GROUP_BY_LABEL`).
+
+### `BatchHardTripletLoss`
+
+Same, but only the single hardest positive + hardest negative per anchor.
+
+### `BatchSemiHardTripletLoss`
+
+Semi-hard mining: negatives harder than the positive but easier than the margin. Often more stable than fully-hard.
+
+### `BatchHardSoftMarginTripletLoss`
+
+Variant with a soft margin (log-sum-exp) instead of a fixed margin hinge.
+
+**When to use batch-triplet losses**: classification-style datasets (labels are class IDs, not pair annotations). E.g. "train an embedder where samples from the same class are close."
+
+## Scored regression losses (labeled pairs, float score)
+
+### `CoSENTLoss`
+
+**The recommended regression loss** for `(text1, text2, score)`. Trains on pairwise ranking: for any two pairs `(a, b)` and `(c, d)` with `score(a,b) > score(c,d)`, the model should score `(a, b)` higher. Much better than squared error.
+
+```python
+loss = CoSENTLoss(model, scale=20.0)
+```
+
+- **Data**: `(text1, text2, float_score)`. Labels can be `[0, 1]` or `[-1, 1]`.
+- Works well with small datasets (STS-B, 5k pairs).
+
+### `AnglELoss`
+
+Similar to CoSENT but uses angle-based optimization in complex space. Sometimes outperforms CoSENT on tasks with fine-grained similarity gradations. Strong alternative.
+
+### `CosineSimilarityLoss`
+
+Squared-error loss on cosine similarity: `mse(cos(text1, text2), label)`. Simpler than CoSENT, usually worse. Keep for legacy / reproducibility.
+
+## Contrastive labeled losses (labeled pairs, binary label)
+
+### `ContrastiveLoss`
+
+For `(text1, text2, label)` where `label ∈ {0, 1}`. Minimizes distance for positives; pushes negatives past a margin.
+
+```python
+loss = ContrastiveLoss(model, margin=0.5, distance_metric=SiameseDistanceMetric.COSINE_DISTANCE)
+```
+
+### `OnlineContrastiveLoss`
+
+Same setup but **ignores "easy" pairs** (positives already close, negatives already far) and only optimizes hard ones. Much more robust to label noise.
+
+```python
+loss = OnlineContrastiveLoss(model, margin=0.5)
+```
+
+**Preferred over `ContrastiveLoss`** for most practical labeled pair datasets.
+
+### `SoftmaxLoss`
+
+A classifier head on concatenated `(u, v, |u-v|)` embeddings, trained with cross-entropy. Useful when you have NLI-style multi-class labels (entailment / neutral / contradiction) and want a categorical loss. Historically important (it trained the first popular sentence embedding models) but **generally outperformed by MNRL**.
+
+## Distillation losses
+
+### `MSELoss`
+
+Regress the student's embedding to match a teacher's embedding.
+
+- **Data**: `(text, teacher_embedding)`. The teacher embedding is a fixed vector per row.
+- Use when distilling a smaller bi-encoder from a larger one.
+
+### `MarginMSELoss`
+
+For `(query, positive, negative, score_diff)`: minimize `mse(student_score_diff, teacher_score_diff)`. The teacher is typically a cross-encoder that produced the score differences.
+
+- **Data**: 3 text columns + 1 float column.
+- Workhorse of dense retriever training from cross-encoder teachers (ms-marco distillation).
+
+### `DistillKLDivLoss`
+
+KL-divergence distillation: student's softmax distribution over candidates should match teacher's.
+
+- **Data**: `(query, passages[], teacher_scores[])`.
+- Good for list-wise distillation when you have multiple candidates per query.
+
+See `../scripts/train_sentence_transformer_distillation_example.py` for the end-to-end pattern (its docstring covers Embedding MSE / Margin MSE / Listwise KL with full recipes).
+
+## Regularizer / wrapper losses
+
+These don't have their own data shape — they wrap another loss and add a regularization objective.
+
+### `MatryoshkaLoss`
+
+Train **once**, deploy at any of several dimensions. Wraps any loss and computes it at multiple truncated dimensions, adding them weighted.
+
+```python
+base_loss = MultipleNegativesRankingLoss(model)
+loss = MatryoshkaLoss(
+    model,
+    base_loss,
+    matryoshka_dims=[768, 512, 256, 128, 64],
+    matryoshka_weights=[1, 1, 1, 1, 1],   # relative weighting per dim
+)
+```
+
+- At inference, `SentenceTransformer(..., truncate_dim=128)` gives 128-dim output with ~95% of full quality.
+- Default matryoshka_dims: pick the dims you want to deploy at. Smaller dims improve compression at the cost of slight quality drop.
+
+### `Matryoshka2dLoss`
+
+2D-Matryoshka: reduce dimension **and** number of transformer layers in a single wrapper. Internally composes `MatryoshkaLoss` + `AdaptiveLayerLoss`, so you only need this one (don't wrap it in AdaptiveLayerLoss yourself). Deploy at any (dim, layer) pair at inference.
+
+### `AdaptiveLayerLoss`
+
+Wrap any loss; adds a term that trains each of the transformer's layers to be a valid exit point. Deploy with fewer layers at inference for faster encoding.
+
+```python
+loss = AdaptiveLayerLoss(
+    model,
+    base_loss,
+    n_layers_per_step=1,
+    last_layer_weight=1.0,
+    prior_layers_weight=1.0,
+)
+```
+
+### `GlobalOrthogonalRegularizationLoss` (GOR)
+
+Stand-alone regularizer (not a wrapper, despite living in this section). Penalizes embedding pairs whose dot product deviates from orthogonality, encouraging the model to spread embeddings across the full vector space. Use it alongside a primary contrastive loss by summing the two outputs in your own training step; can help with downstream retrieval diversity.
+
+## Unsupervised losses
+
+### `DenoisingAutoEncoderLoss` (TSDAE)
+
+Sentence-level denoising autoencoder: corrupt a sentence (drop tokens), force the model to reconstruct it. Pretraining-style — useful for domain adaptation when you have unlabeled in-domain sentences.
+
+### `ContrastiveTensionLoss`
+
+Unsupervised contrastive: two copies of the model encode the same sentence; they should agree. Pure self-supervised.
+
+### `ContrastiveTensionLossInBatchNegatives`
+
+CT with in-batch negatives. Stronger than vanilla CT.
+
+## Gotchas
+
+- **`MultipleNegativesRankingLoss` without `BatchSamplers.NO_DUPLICATES`** will include duplicate anchors in the same batch, destroying training signal. Always set the sampler.
+- **Any `Cached*` loss + `gradient_checkpointing=True` = crash.** Pick one.
+- **`TripletLoss` with bad negatives** (too easy) = loss hits zero fast and model stops learning. Mine hard negatives first.
+- **Matryoshka wrapping `CachedMultipleNegativesRankingLoss`**: supported, but the cached-loss's mini-batch semantics apply to the base loss only. Think twice before combining.

package/bundled-skills/train-sentence-transformers/references/losses_sparse_encoder.md

@@ -0,0 +1,106 @@
+# Sparse-Encoder Losses (SPLADE)
+
+All losses live in `sentence_transformers.sparse_encoder.losses`.
+
+This reference targets the **SPLADE** architecture (Transformer + SpladePooling). The sparse-encoder package also exports `CSRLoss` and `CSRReconstructionLoss` for the CSR architecture (Transformer + Pooling + SparseAutoEncoder); those are out of scope here — see the sbert.net docs if you're training a CSR model.
+
+Choosing a loss means (a) pick a base loss (contrastive, regression, distillation) and (b) wrap it in `SpladeLoss` to add FLOPS regularization.
+
+## Top-line decision table
+
+| You have | Use |
+|---|---|
+| `(anchor, positive)` or triplet, SPLADE architecture | `SpladeLoss(loss=SparseMultipleNegativesRankingLoss(model), ...)` |
+| Same, want effective batch size of 256+ | `CachedSpladeLoss(...)` |
+| `(text1, text2, score)` labeled pairs | `SparseCoSENTLoss` or `SparseCosineSimilarityLoss` |
+| Distillation from cross-encoder teacher | `SparseMarginMSELoss` |
+| Listwise distillation | `SparseDistillKLDivLoss` |
+| Explicit triplet | `SparseTripletLoss` |
+
+## The core wrapper: `SpladeLoss`
+
+`SpladeLoss` adds **FLOPS regularization** on top of another sparse loss. FLOPS regularization penalizes non-zero activations, keeping embeddings genuinely sparse.
+
+```python
+loss = SpladeLoss(
+    model=model,
+    loss=SparseMultipleNegativesRankingLoss(model=model),
+    query_regularizer_weight=5e-5,
+    document_regularizer_weight=3e-5,
+)
+```
+
+- `query_regularizer_weight`: how much to penalize non-zero terms in query embeddings.
+- `document_regularizer_weight`: same for documents.
+- Typical range: 1e-5 to 1e-4. Higher = sparser embeddings, lower recall; lower = denser, possibly better recall.
+- `SparseEncoderTrainer` automatically registers a `SpladeRegularizerWeightSchedulerCallback` whenever the loss is a `SpladeLoss`. The callback ramps the weights from 0 up to the target over the first ~33% of training; the default shape is `SchedulerType.QUADRATIC` (not linear). The ramp length and shape are configured on the callback (`SpladeRegularizerWeightSchedulerCallback(loss=..., warmup_ratio=..., scheduler_type=...)`), not on `SpladeLoss`; to override, instantiate the callback yourself and pass it via `callbacks=[...]`. This ramp is important; starting with full regularization from step 0 kills learning.
+
+Use `CachedSpladeLoss` for the GradCache variant.
+
+## Contrastive losses (no labels)
+
+### `SparseMultipleNegativesRankingLoss`
+
+Sparse analog of bi-encoder MNRL. In-batch contrastive.
+
+```python
+inner = SparseMultipleNegativesRankingLoss(model=model)
+loss = SpladeLoss(model=model, loss=inner, query_regularizer_weight=5e-5, document_regularizer_weight=3e-5)
+```
+
+- **Always wrap in `SpladeLoss`** for SPLADE architectures.
+- Set `batch_sampler=BatchSamplers.NO_DUPLICATES` on training args.
+
+### `SparseTripletLoss`
+
+Classic triplet margin loss on explicit `(anchor, positive, negative)`.
+
+## Labeled regression losses
+
+### `SparseCoSENTLoss`
+
+Pairwise ranking loss for `(text1, text2, score)`. Mirrors bi-encoder `CoSENTLoss`.
+
+### `SparseCosineSimilarityLoss`
+
+MSE on cosine similarity. Simpler, usually worse than CoSENT.
+
+### `SparseAnglELoss`
+
+Angle-based loss in complex space. Alternative to CoSENT.
+
+## Distillation losses
+
+### `SparseMSELoss`
+
+Embedding MSE. Student sparse embedding should match teacher embedding.
+
+- **Data**: `(text, teacher_embedding)`.
+- Teacher can be a dense bi-encoder or another sparse model.
+
+### `SparseMarginMSELoss`
+
+Margin MSE from a cross-encoder teacher.
+
+- **Data**: `(query, positive, negative, score_diff)` where `score_diff = teacher_score(query, positive) - teacher_score(query, negative)`.
+- Typical recipe for training SPLADE from cross-encoder labels (ms-marco distillation).
+- Wrap in `SpladeLoss(model, loss=SparseMarginMSELoss(model), ...)` for SPLADE.
+
+### `SparseDistillKLDivLoss`
+
+Listwise KL-div distillation — student's softmax distribution over candidates should match teacher's.
+
+## Independent regularizer
+
+### `FlopsLoss`
+
+Standalone FLOPS regularizer. Usually you use this via `SpladeLoss`, not directly.
+
+For regularizer-weight tuning and dense-output recovery, see `troubleshooting.md` ("SPLADE embeddings are dense"). MLM-head requirement: `base_model_selection.md` (SPARSE section). Active-dim sparsity targets and how to monitor them: `evaluators_sparse_encoder.md` (Sparsity tracking).
+
+## Gotchas
+
+- **`SparseMultipleNegativesRankingLoss` without `SpladeLoss` wrapping on a SPLADE model**: no FLOPS regularization -> dense outputs defeating the purpose of SPLADE. Always wrap.
+- **`CachedSpladeLoss` + `gradient_checkpointing=True`**: crash. Pick one.
+- **Starting training with full FLOPS regularization at step 0**: the model outputs zero everywhere and gets stuck. The built-in scheduler avoids this — don't override it unless you know why.
+- **`query_regularizer_weight` == `document_regularizer_weight`**: usually wrong. Queries should be sparser than documents (fewer terms per query). Since higher regularization drives more zeros, give the query weight the larger value. `query_regularizer_weight=5e-5`, `document_regularizer_weight=3e-5` is a good starting ratio.

package/bundled-skills/train-sentence-transformers/references/model_architectures.md

@@ -0,0 +1,178 @@
+# Model Architectures (SentenceTransformer)
+
+The `SentenceTransformer` class is a `torch.nn.Sequential` of modules. The common shape is `Transformer` + `Pooling` (+ optional `Normalize` / `Dense`), but four distinct architecture families are supported and the right choice depends on the task.
+
+## The four architecture families
+
+| Family | Backbone | Pooling | Use case |
+|---|---|---|---|
+| **Encoder (bidirectional)** | BERT, RoBERTa, DeBERTa, MPNet, ModernBERT, XLM-R | `mean` (default) or `cls` | Short/medium text, general default |
+| **Decoder (causal LLM)** | Qwen, Llama, Mistral, Gemma | `lasttoken` | Long context, instruction-tunable, larger quality ceiling |
+| **Static embeddings** | `StaticEmbedding` module | N/A | CPU-only, <10MB, extremely fast |
+| **Multimodal / Router** | VLM backbones or composed encoders | depends | Text + image / audio / video |
+
+Below, each family with concrete setup.
+
+## Encoder models (the default)
+
+The historical default and still usually the right choice for text embeddings.
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("microsoft/mpnet-base")
+# Auto-constructs: Transformer(feature-extraction) -> Pooling(mean).
+```
+
+When `SentenceTransformer("<checkpoint>")` is called with a raw HF encoder, it auto-wraps the transformer and adds `Pooling(..., pooling_mode="mean")`.
+
+To customize pooling or add modules:
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import Normalize, Pooling, Transformer
+
+transformer = Transformer("answerdotai/ModernBERT-base")
+pooling = Pooling(transformer.get_embedding_dimension(), pooling_mode="cls")    # or "mean", "lasttoken", ...
+model = SentenceTransformer(modules=[transformer, pooling, Normalize()])
+```
+
+**Pooling modes**:
+- `mean` (default) — average of token embeddings, masked to the attention mask. Strongest default.
+- `cls` — embedding of the `[CLS]` token. Works if the base was CLS-pretrained.
+- `max` — element-wise max across tokens. Rare.
+- `mean_sqrt_len_tokens` — mean scaled by √seq_len. Empirically helps on some tasks.
+- `weightedmean` — token-position-weighted mean. Useful for decoder bases as a non-last-token alternative.
+- `lasttoken` — embedding of the last token. Required for causal-LM bases (see decoder section below).
+
+Don't switch pooling mid-training. Pick once.
+
+## Decoder / causal LLM models
+
+Strong at long context, instruction following, multilingual. Memory-hungry — typically LoRA-trained rather than full fine-tuned.
+
+**Two setup paths depending on whether the model was already adapted for embeddings:**
+
+```python
+# Path A: already-adapted embedding checkpoint (ships with the right modules):
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")   # just works
+
+# Path B: raw decoder LLM, build the pipeline manually:
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import Normalize, Pooling, Transformer
+
+transformer = Transformer(
+    "Qwen/Qwen2.5-0.5B",
+    transformer_task="text-generation",         # critical: causal attention, no bidirectional
+    processor_kwargs={"padding_side": "left"},  # last-token pooling wants left-padding
+)
+pooling = Pooling(transformer.get_embedding_dimension(), pooling_mode="lasttoken")
+model = SentenceTransformer(modules=[transformer, pooling, Normalize()])
+```
+
+Skipping `transformer_task="text-generation"` or `pooling_mode="lasttoken"` on a raw decoder gives embeddings that look plausible until you benchmark.
+
+**Why last-token pooling:** causal attention means only the last token has seen the full sequence. Mean-pooling a causal model averages embeddings that only saw prefixes — the result doesn't represent the whole input.
+
+For training decoder bases:
+- Learning rate: typically `1e-4` or higher (not `2e-5` like encoders).
+- LoRA is almost always the right choice for >1B-param bases; see `../scripts/train_sentence_transformer_with_lora_example.py` (its docstring covers when to use, hyperparams, QLoRA for 7B+, and adapter sharing).
+
+## Static embeddings
+
+`StaticEmbedding` skips the transformer entirely — each token maps to a pre-computed vector via a lookup table. No attention, no contextualization.
+
+**When to use:**
+- CPU inference, no GPU, browser / edge / on-device deployment.
+- Need <10MB model size.
+- Latency budget <1ms per embedding.
+- Have >1M training pairs (contextualization is replaced by per-token optimization; this takes data).
+
+**When NOT to use:**
+- Task needs contextual understanding (polysemy, syntax, long-range dependencies).
+- You have <100k training pairs — the model won't learn enough.
+
+**Setup:**
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import StaticEmbedding
+from tokenizers import Tokenizer
+
+tokenizer = Tokenizer.from_pretrained("google-bert/bert-base-uncased")
+static_embedding = StaticEmbedding(tokenizer, embedding_dim=512)
+model = SentenceTransformer(modules=[static_embedding])
+```
+
+Train with `MultipleNegativesRankingLoss` on a large contrastive dataset (1M+ pairs).
+
+**Warm starts vs. random init** — with **>1M training samples**, random-init beats `StaticEmbedding.from_model2vec(...)` or `.from_distillation(...)` warm starts. With smaller datasets, warm starts help.
+
+```python
+# For smaller datasets (<100k), warm-start:
+static_embedding = StaticEmbedding.from_model2vec("minishlab/potion-base-8M")
+# or:
+static_embedding = StaticEmbedding.from_distillation("sentence-transformers/all-MiniLM-L6-v2", vocabulary=list(tokenizer.get_vocab().keys()))
+```
+
+See `../scripts/train_sentence_transformer_static_embedding_example.py` for a runnable end-to-end recipe (random init + MNRL + Matryoshka + bf16 + lr=2e-1) and the [Static Embeddings blog post](https://huggingface.co/blog/static-embeddings) for benchmarks.
+
+## Multimodal via VLM backbone
+
+Modern vision-language models can be loaded directly and produce joint text+image embeddings:
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer(
+    "Qwen/Qwen3-VL-Embedding-2B",
+    model_kwargs={"attn_implementation": "flash_attention_2"},  # do NOT set torch_dtype here; see training_args.md
+    processor_kwargs={"min_pixels": 28 * 28, "max_pixels": 600 * 600},
+)
+
+# Check which modalities this model supports:
+print(model.modalities)
+# ['text', 'image', 'video', 'message']
+```
+
+Training data can mix text, PIL images, image paths/URLs, audio, and mixed-modality dicts like `{"image": <PIL>, "text": "describe this"}`. The data collator handles preprocessing via the model's `preprocess` method.
+
+Install multimodal extras: `pip install "sentence-transformers[image]"` (or `[audio]`, `[video]`).
+
+**Precision**: load in fp32 and pass `bf16=True` (or `fp16=True`) to TrainingArguments — autocast handles the inference path. Don't set `torch_dtype="bfloat16"` in `model_kwargs`: it puts Adam state in bf16 and silently degrades quality (see `training_args.md`).
+
+## Multimodal via Router
+
+Instead of one VLM backbone, compose separate encoders per modality:
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import Dense, Pooling, Router, Transformer
+
+# Text encoder
+text_encoder = Transformer("sentence-transformers/all-MiniLM-L6-v2")
+text_pooling = Pooling(text_encoder.get_embedding_dimension(), pooling_mode="mean")
+# Project text to match image encoder's dim
+text_projection = Dense(text_encoder.get_embedding_dimension(), 768)
+
+# Image encoder (SigLIP outputs pooled embeddings directly)
+image_encoder = Transformer("google/siglip2-base-patch16-224")
+
+router = Router(
+    sub_modules={
+        "text": [text_encoder, text_pooling, text_projection],
+        "image": [image_encoder],
+    },
+)
+model = SentenceTransformer(modules=[router])
+```
+
+**Warning**: Router-based models have unaligned embedding spaces at init — you must train to align them. Use a `Dense` projection layer when dimensions differ. Task-based routing (different encoders for queries vs. documents) is also supported via `route_mappings`; see the `Router` docstring.
+
+## Gotchas
+
+- **Decoder base with mean pooling**: silently produces garbage embeddings. Always use `lasttoken`.
+- **Router multimodal without training**: the separate encoders' embedding spaces are unaligned at init. Don't expect useful cross-modal similarity until you've trained with a loss that aligns the spaces.
+- **StaticEmbedding with fewer than 100k pairs**: the model won't learn enough. Either warm-start via `from_model2vec` / `from_distillation`, or use a regular encoder.
+- **Large VLM backbones on consumer GPUs**: combine LoRA + `attn_implementation="flash_attention_2"`. With LoRA only, you can additionally pass `torch_dtype="bfloat16"` — the bf16 base weights are frozen, so the Adam-state precision concern from the precision rule above doesn't apply (the LoRA adapter stays fp32, so its optimizer state stays fp32). Without LoRA, follow the precision rule: keep weights fp32 and rely on `bf16=True` autocast.