tribunal-kit 9.2.3 → 9.2.5

package/skills/train-sentence-transformers/references/losses_sparse_encoder.md

@@ -0,0 +1,106 @@
+# Sparse-Encoder Losses (SPLADE)
+
+All losses live in `sentence_transformers.sparse_encoder.losses`.
+
+This reference targets the **SPLADE** architecture (Transformer + SpladePooling). The sparse-encoder package also exports `CSRLoss` and `CSRReconstructionLoss` for the CSR architecture (Transformer + Pooling + SparseAutoEncoder). Those are out of scope here. See the sbert.net docs if you're training a CSR model.
+
+Choosing a loss means (a) pick a base loss (contrastive, regression, distillation) and (b) wrap it in `SpladeLoss` to add FLOPS regularization.
+
+## Top-line decision table
+
+| You have | Use |
+|---|---|
+| `(anchor, positive)` or triplet, SPLADE architecture | `SpladeLoss(loss=SparseMultipleNegativesRankingLoss(model), ...)` |
+| Same, want effective batch size of 256+ | `CachedSpladeLoss(...)` |
+| `(text1, text2, score)` labeled pairs | `SparseCoSENTLoss` or `SparseCosineSimilarityLoss` |
+| Distillation from cross-encoder teacher | `SparseMarginMSELoss` |
+| Listwise distillation | `SparseDistillKLDivLoss` |
+| Explicit triplet | `SparseTripletLoss` |
+
+## The core wrapper: `SpladeLoss`
+
+`SpladeLoss` adds **FLOPS regularization** on top of another sparse loss. FLOPS regularization penalizes non-zero activations, keeping embeddings genuinely sparse.
+
+```python
+loss = SpladeLoss(
+    model=model,
+    loss=SparseMultipleNegativesRankingLoss(model=model),
+    query_regularizer_weight=5e-5,
+    document_regularizer_weight=3e-5,
+)
+```
+
+- `query_regularizer_weight`: how much to penalize non-zero terms in query embeddings.
+- `document_regularizer_weight`: same for documents.
+- Typical range: 1e-5 to 1e-4. Higher = sparser embeddings, lower recall. Lower = denser, possibly better recall.
+- `SparseEncoderTrainer` automatically registers a `SpladeRegularizerWeightSchedulerCallback` whenever the loss is a `SpladeLoss`. The callback ramps the weights from 0 up to the target over the first ~33% of training. The default shape is `SchedulerType.QUADRATIC` (not linear). The ramp length and shape are configured on the callback (`SpladeRegularizerWeightSchedulerCallback(loss=..., warmup_ratio=..., scheduler_type=...)`), not on `SpladeLoss`. To override, instantiate the callback yourself and pass it via `callbacks=[...]`. This ramp is important. Starting with full regularization from step 0 kills learning.
+
+Use `CachedSpladeLoss` for the GradCache variant.
+
+## Contrastive losses (no labels)
+
+### `SparseMultipleNegativesRankingLoss`
+
+Sparse analog of bi-encoder MNRL. In-batch contrastive.
+
+```python
+inner = SparseMultipleNegativesRankingLoss(model=model)
+loss = SpladeLoss(model=model, loss=inner, query_regularizer_weight=5e-5, document_regularizer_weight=3e-5)
+```
+
+- **Always wrap in `SpladeLoss`** for SPLADE architectures.
+- Set `batch_sampler=BatchSamplers.NO_DUPLICATES` on training args.
+
+### `SparseTripletLoss`
+
+Classic triplet margin loss on explicit `(anchor, positive, negative)`.
+
+## Labeled regression losses
+
+### `SparseCoSENTLoss`
+
+Pairwise ranking loss for `(text1, text2, score)`. Mirrors bi-encoder `CoSENTLoss`.
+
+### `SparseCosineSimilarityLoss`
+
+MSE on cosine similarity. Simpler, usually worse than CoSENT.
+
+### `SparseAnglELoss`
+
+Angle-based loss in complex space. Alternative to CoSENT.
+
+## Distillation losses
+
+### `SparseMSELoss`
+
+Embedding MSE. Student sparse embedding should match teacher embedding.
+
+- **Data**: `(text, teacher_embedding)`.
+- Teacher can be a dense bi-encoder or another sparse model.
+
+### `SparseMarginMSELoss`
+
+Margin MSE from a cross-encoder teacher.
+
+- **Data**: `(query, positive, negative, score_diff)` where `score_diff = teacher_score(query, positive) - teacher_score(query, negative)`.
+- Typical recipe for training SPLADE from cross-encoder labels (ms-marco distillation).
+- Wrap in `SpladeLoss(model, loss=SparseMarginMSELoss(model), ...)` for SPLADE.
+
+### `SparseDistillKLDivLoss`
+
+Listwise KL-div distillation. Student's softmax distribution over candidates should match teacher's.
+
+## Independent regularizer
+
+### `FlopsLoss`
+
+Standalone FLOPS regularizer. Usually you use this via `SpladeLoss`, not directly.
+
+For regularizer-weight tuning and dense-output recovery, see `troubleshooting.md` ("SPLADE embeddings are dense"). MLM-head requirement: `base_model_selection.md` (SPARSE section). Active-dim sparsity targets and how to monitor them: `evaluators_sparse_encoder.md` (Sparsity tracking).
+
+## Gotchas
+
+- **`SparseMultipleNegativesRankingLoss` without `SpladeLoss` wrapping on a SPLADE model**: no FLOPS regularization -> dense outputs defeating the purpose of SPLADE. Always wrap.
+- **`CachedSpladeLoss` + `gradient_checkpointing=True`**: crash. Pick one.
+- **Starting training with full FLOPS regularization at step 0**: the model outputs zero everywhere and gets stuck. The built-in scheduler avoids this. Don't override it unless you know why.
+- **`query_regularizer_weight` == `document_regularizer_weight`**: usually wrong. Queries should be sparser than documents (fewer terms per query). Since higher regularization drives more zeros, give the query weight the larger value. `query_regularizer_weight=5e-5`, `document_regularizer_weight=3e-5` is a good starting ratio.

package/skills/train-sentence-transformers/references/model_architectures.md

@@ -0,0 +1,178 @@
+# Model Architectures (SentenceTransformer)
+
+The `SentenceTransformer` class is a `torch.nn.Sequential` of modules. The common shape is `Transformer` + `Pooling` (+ optional `Normalize` / `Dense`), but four distinct architecture families are supported and the right choice depends on the task.
+
+## The four architecture families
+
+| Family | Backbone | Pooling | Use case |
+|---|---|---|---|
+| **Encoder (bidirectional)** | BERT, RoBERTa, DeBERTa, MPNet, ModernBERT, XLM-R | `mean` (default) or `cls` | Short/medium text, general default |
+| **Decoder (causal LLM)** | Qwen, Llama, Mistral, Gemma | `lasttoken` | Long context, instruction-tunable, larger quality ceiling |
+| **Static embeddings** | `StaticEmbedding` module | N/A | CPU-only, <10MB, extremely fast |
+| **Multimodal / Router** | VLM backbones or composed encoders | depends | Text + image / audio / video |
+
+Below, each family with concrete setup.
+
+## Encoder models (the default)
+
+The historical default and still usually the right choice for text embeddings.
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("microsoft/mpnet-base")
+# Auto-constructs: Transformer(feature-extraction) -> Pooling(mean).
+```
+
+When `SentenceTransformer("<checkpoint>")` is called with a raw HF encoder, it auto-wraps the transformer and adds `Pooling(..., pooling_mode="mean")`.
+
+To customize pooling or add modules:
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import Normalize, Pooling, Transformer
+
+transformer = Transformer("answerdotai/ModernBERT-base")
+pooling = Pooling(transformer.get_embedding_dimension(), pooling_mode="cls")    # or "mean", "lasttoken", ...
+model = SentenceTransformer(modules=[transformer, pooling, Normalize()])
+```
+
+**Pooling modes**:
+- `mean` (default): average of token embeddings, masked to the attention mask. Strongest default.
+- `cls`: embedding of the `[CLS]` token. Works if the base was CLS-pretrained.
+- `max`: element-wise max across tokens. Rare.
+- `mean_sqrt_len_tokens`: mean scaled by √seq_len. Empirically helps on some tasks.
+- `weightedmean`: token-position-weighted mean. Useful for decoder bases as a non-last-token alternative.
+- `lasttoken`: embedding of the last token. Required for causal-LM bases (see decoder section below).
+
+Don't switch pooling mid-training. Pick once.
+
+## Decoder / causal LLM models
+
+Strong at long context, instruction following, multilingual. Memory-hungry, typically LoRA-trained rather than full fine-tuned.
+
+**Two setup paths depending on whether the model was already adapted for embeddings:**
+
+```python
+# Path A: already-adapted embedding checkpoint (ships with the right modules):
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")   # just works
+
+# Path B: raw decoder LLM, build the pipeline manually:
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import Normalize, Pooling, Transformer
+
+transformer = Transformer(
+    "Qwen/Qwen2.5-0.5B",
+    transformer_task="text-generation",         # critical: causal attention, no bidirectional
+    processor_kwargs={"padding_side": "left"},  # last-token pooling wants left-padding
+)
+pooling = Pooling(transformer.get_embedding_dimension(), pooling_mode="lasttoken")
+model = SentenceTransformer(modules=[transformer, pooling, Normalize()])
+```
+
+Skipping `transformer_task="text-generation"` or `pooling_mode="lasttoken"` on a raw decoder gives embeddings that look plausible until you benchmark.
+
+**Why last-token pooling:** causal attention means only the last token has seen the full sequence. Mean-pooling a causal model averages embeddings that only saw prefixes. The result doesn't represent the whole input.
+
+For training decoder bases:
+- Learning rate: typically `1e-4` or higher (not `2e-5` like encoders).
+- LoRA is almost always the right choice for >1B-param bases. See `../scripts/train_sentence_transformer_with_lora_example.py` (its docstring covers when to use, hyperparams, QLoRA for 7B+, and adapter sharing).
+
+## Static embeddings
+
+`StaticEmbedding` skips the transformer entirely. Each token maps to a pre-computed vector via a lookup table. No attention, no contextualization.
+
+**When to use:**
+- CPU inference, no GPU, browser / edge / on-device deployment.
+- Need <10MB model size.
+- Latency budget <1ms per embedding.
+- Have >1M training pairs (contextualization is replaced by per-token optimization, which takes data).
+
+**When NOT to use:**
+- Task needs contextual understanding (polysemy, syntax, long-range dependencies).
+- You have <100k training pairs. The model won't learn enough.
+
+**Setup:**
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import StaticEmbedding
+from tokenizers import Tokenizer
+
+tokenizer = Tokenizer.from_pretrained("google-bert/bert-base-uncased")
+static_embedding = StaticEmbedding(tokenizer, embedding_dim=512)
+model = SentenceTransformer(modules=[static_embedding])
+```
+
+Train with `MultipleNegativesRankingLoss` on a large contrastive dataset (1M+ pairs).
+
+**Warm starts vs. random init**: with **>1M training samples**, random-init beats `StaticEmbedding.from_model2vec(...)` or `.from_distillation(...)` warm starts. With smaller datasets, warm starts help.
+
+```python
+# For smaller datasets (<100k), warm-start:
+static_embedding = StaticEmbedding.from_model2vec("minishlab/potion-base-8M")
+# or:
+static_embedding = StaticEmbedding.from_distillation("sentence-transformers/all-MiniLM-L6-v2", vocabulary=list(tokenizer.get_vocab().keys()))
+```
+
+See `../scripts/train_sentence_transformer_static_embedding_example.py` for a runnable end-to-end recipe (random init + MNRL + Matryoshka + bf16 + lr=2e-1) and the [Static Embeddings blog post](https://huggingface.co/blog/static-embeddings) for benchmarks.
+
+## Multimodal via VLM backbone
+
+Modern vision-language models can be loaded directly and produce joint text+image embeddings:
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer(
+    "Qwen/Qwen3-VL-Embedding-2B",
+    model_kwargs={"attn_implementation": "flash_attention_2"},  # do NOT set torch_dtype here; see training_args.md
+    processor_kwargs={"min_pixels": 28 * 28, "max_pixels": 600 * 600},
+)
+
+# Check which modalities this model supports:
+print(model.modalities)
+# ['text', 'image', 'video', 'message']
+```
+
+Training data can mix text, PIL images, image paths/URLs, audio, and mixed-modality dicts like `{"image": <PIL>, "text": "describe this"}`. The data collator handles preprocessing via the model's `preprocess` method.
+
+Install multimodal extras: `pip install "sentence-transformers[image]"` (or `[audio]`, `[video]`).
+
+**Precision**: load in fp32 and pass `bf16=True` (or `fp16=True`) to TrainingArguments. Autocast handles the inference path. Don't set `torch_dtype="bfloat16"` in `model_kwargs`: it puts Adam state in bf16 and silently degrades quality (see `training_args.md`).
+
+## Multimodal via Router
+
+Instead of one VLM backbone, compose separate encoders per modality:
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.sentence_transformer.modules import Dense, Pooling, Router, Transformer
+
+# Text encoder
+text_encoder = Transformer("sentence-transformers/all-MiniLM-L6-v2")
+text_pooling = Pooling(text_encoder.get_embedding_dimension(), pooling_mode="mean")
+# Project text to match image encoder's dim
+text_projection = Dense(text_encoder.get_embedding_dimension(), 768)
+
+# Image encoder (SigLIP outputs pooled embeddings directly)
+image_encoder = Transformer("google/siglip2-base-patch16-224")
+
+router = Router(
+    sub_modules={
+        "text": [text_encoder, text_pooling, text_projection],
+        "image": [image_encoder],
+    },
+)
+model = SentenceTransformer(modules=[router])
+```
+
+**Warning**: Router-based models have unaligned embedding spaces at init. You must train to align them. Use a `Dense` projection layer when dimensions differ. Task-based routing (different encoders for queries vs. documents) is also supported via `route_mappings`. See the `Router` docstring.
+
+## Gotchas
+
+- **Decoder base with mean pooling**: silently produces garbage embeddings. Always use `lasttoken`.
+- **Router multimodal without training**: the separate encoders' embedding spaces are unaligned at init. Don't expect useful cross-modal similarity until you've trained with a loss that aligns the spaces.
+- **StaticEmbedding with fewer than 100k pairs**: the model won't learn enough. Either warm-start via `from_model2vec` / `from_distillation`, or use a regular encoder.
+- **Large VLM backbones on consumer GPUs**: combine LoRA + `attn_implementation="flash_attention_2"`. With LoRA only, you can additionally pass `torch_dtype="bfloat16"`. The bf16 base weights are frozen, so the Adam-state precision concern from the precision rule above doesn't apply (the LoRA adapter stays fp32, so its optimizer state stays fp32). Without LoRA, follow the precision rule: keep weights fp32 and rely on `bf16=True` autocast.

package/skills/train-sentence-transformers/references/prompts_and_instructions.md

@@ -0,0 +1,160 @@
+# Prompts and Instructions
+
+Modern embedding models (E5, BGE, GTE, Qwen3-Embedding, Nomic, Instructor, etc.) use **prompts** / **instructions** at encode time (short prefixes like `"query: "`, `"passage: "`, or `"Represent this sentence for retrieval:"`) to signal task intent.
+
+In sentence-transformers, prompts work during both **training** and **inference**, and the library keeps them aligned automatically.
+
+The prompt is literally prepended to the input text before tokenization.
+
+## Model-level prompts
+
+Set prompts on the model so `encode()`, `encode_query()`, `encode_document()` use them automatically:
+
+```python
+model = SentenceTransformer("your-base-model")
+model.prompts = {
+    "query":    "query: ",
+    "document": "passage: ",
+}
+model.default_prompt_name = "document"        # used when none is specified
+```
+
+At inference:
+
+```python
+q_emb = model.encode_query("What is the capital of France?")
+# Internally: encodes "query: What is the capital of France?"
+
+d_emb = model.encode_document(["Paris is the capital of France.", "Berlin is the capital of Germany."])
+# Internally: encodes "passage: Paris..." etc.
+
+# Explicit prompt_name:
+emb = model.encode(["some text"], prompt_name="query")
+```
+
+## Training with prompts
+
+Set prompts on the training args so they're applied automatically to the right columns during training. Four shapes, increasingly specific:
+
+### 1. Single prompt (applied everywhere)
+
+```python
+args = SentenceTransformerTrainingArguments(
+    ...,
+    prompts="Represent this sentence for similarity: ",
+)
+```
+
+Every input column gets the prefix. Simplest and often sufficient for single-task training.
+
+### 2. Per-column prompts
+
+```python
+prompts = {
+    "anchor":   "query: ",
+    "positive": "passage: ",
+    "negative": "passage: ",
+}
+```
+
+Keys are **column names**. Different prefixes for different roles.
+
+### 3. Per-dataset prompts (multi-dataset)
+
+```python
+prompts = {
+    "all-nli": "Classify the entailment relationship: ",
+    "stsb":    "Score semantic similarity: ",
+}
+```
+
+Keys are **dataset names** (matching the `train_dataset` dict keys).
+
+### 4. Per-dataset + per-column
+
+```python
+prompts = {
+    "all-nli": "",                                          # all-nli: no prompt
+    "msmarco": {                                            # msmarco: per-column
+        "query":    "query: ",
+        "positive": "passage: ",
+        "negative": "passage: ",
+    },
+}
+```
+
+## Cross-encoder specifics
+
+Cross-encoders restrict prompts to single-value or per-dataset only (no per-column). This is because cross-encoders take text pairs, not separate columns.
+
+```python
+args = CrossEncoderTrainingArguments(
+    ...,
+    prompts="Rank this passage for the query: ",   # single prompt
+)
+# or
+args = CrossEncoderTrainingArguments(
+    ...,
+    prompts={"msmarco": "...", "gooaq": "..."},    # per-dataset
+)
+```
+
+## Sparse encoders (SPLADE)
+
+Sparse encoders support prompts with the same four shapes as bi-encoders (added in v5.x). Same `model.prompts = {...}` API.
+
+## Pooling and prompt tokens (bi-encoder)
+
+When using mean/last-token pooling with prompt prefixes, the prompt tokens themselves can either:
+- **Be included** in the pooled embedding (default behavior). Simpler, what most models do.
+- **Be excluded**: only the actual content tokens are pooled.
+
+To exclude, flip `include_prompt` on the Pooling module, found via `isinstance` rather than a fixed index, since multimodal / Router pipelines don't always put Pooling at position 1:
+
+```python
+from sentence_transformers.sentence_transformer.modules import Pooling
+
+for module in model:
+    if isinstance(module, Pooling):
+        module.include_prompt = False
+```
+
+Or use the helper if your model exposes one (e.g. `SparseEncoder.set_pooling_include_prompt(False)`).
+
+**When to exclude**: when the prompt is purely a task signal and you don't want it to dilute the semantic representation. Most E5 / BGE / GTE models leave prompts included.
+
+During training with `prompts=`, the `include_prompt` setting is respected. The trainer also tracks `include_prompt_lengths` internally so pooling skips the prompt tokens correctly when `include_prompt=False`.
+
+## Inference alignment
+
+If you trained with `prompts={"query": "query: ", ...}`, you must:
+
+1. **Save the prompts on the model**: `model.prompts = args.prompts` before `save_pretrained`, OR let the library do this automatically via the model card data.
+2. **Use the same prompts at inference**: call `encode_query()` / `encode_document()` and the saved prompts are applied.
+
+If the saved model's `config_sentence_transformers.json` has `prompts` + `default_prompt_name`, anyone who loads it gets the correct prompts for free.
+
+## Instruction tuning (different from prompts)
+
+Some models (Instructor, Qwen3-Embedding) use **instructions**, longer descriptions like `"Represent the biomedical query for retrieving relevant passages:"`. In sentence-transformers these are modeled the same way: just set them as prompts.
+
+The model-card convention is to list available instructions/prompts in a dict:
+
+```python
+model.prompts = {
+    "msmarco-query":    "Represent the query for MS MARCO retrieval: ",
+    "msmarco-doc":      "Represent the passage for MS MARCO retrieval: ",
+    "sts":              "Represent the sentence for semantic similarity: ",
+    "classification":   "Represent the sentence for topic classification: ",
+}
+```
+
+Users call `model.encode(["..."], prompt_name="msmarco-query")`.
+
+## Gotchas
+
+- **Training with prompts but forgetting to set them on the model before `save_pretrained`**: the saved model won't know about the prompts. Users will encode *without* the prefix and get bad results. Fix: `model.prompts = args.prompts` before saving, or use `SentenceTransformerModelCardData(...)` with the prompt info.
+- **Using `encode_query()` at inference but didn't train with a "query" prompt**: the method will just call regular `encode` (no prefix applied), which is fine. But the documentation implies you use it, and users may be confused.
+- **Trailing spaces matter**: `"query: "` vs `"query:"`. The former has a trailing space before the real text. Inspect your training data to know which one your model was trained with.
+- **Mixing prompted and non-prompted data in multi-dataset training** is fine as long as your `prompts` dict covers it per-dataset (e.g. `{"dataset_a": "query: ", "dataset_b": ""}`).
+- **`include_prompt=False` with small datasets**: the model may under-fit because you've effectively shortened every input. Usually leave as True.