tribunal-kit 9.2.3 → 9.2.5

package/.agent/skills/train-sentence-transformers/references/training_args.md

@@ -0,0 +1,270 @@
+# Training Arguments
+
+`SentenceTransformerTrainingArguments`, `CrossEncoderTrainingArguments`, and `SparseEncoderTrainingArguments` all inherit from Hugging Face's `TrainingArguments`, so 95% of the arguments are the same.
+
+This reference covers the arguments that actually matter for embedding-model training.
+
+## The recommended default set
+
+Start with this and adjust only what you have a reason to change:
+
+```python
+from sentence_transformers import SentenceTransformerTrainingArguments
+from sentence_transformers.base.sampler import BatchSamplers
+
+args = SentenceTransformerTrainingArguments(
+    output_dir="models/my-model",
+
+    # Duration
+    num_train_epochs=1,
+    # max_steps=10_000,                    # alternative to epochs
+
+    # Batch size
+    per_device_train_batch_size=64,
+    per_device_eval_batch_size=64,
+    gradient_accumulation_steps=1,
+
+    # Optimizer
+    learning_rate=2e-5,
+    warmup_steps=0.1,                      # transformers v5.2 deprecated `warmup_ratio`; pass the ratio as a float directly to `warmup_steps`
+    lr_scheduler_type="linear",
+    weight_decay=0.0,
+
+    # Precision
+    bf16=True,                             # fp16=True on older GPUs (T4, V100)
+
+    # Sampler (bi-encoder + sparse-encoder)
+    batch_sampler=BatchSamplers.NO_DUPLICATES,
+
+    # Eval + checkpointing
+    eval_strategy="steps",
+    eval_steps=0.1,                        # fraction: 10 evals/epoch, scales with dataset size
+    save_strategy="steps",
+    save_steps=0.1,                        # keep aligned with eval_steps for load_best_model_at_end
+    save_total_limit=2,
+    load_best_model_at_end=True,
+    metric_for_best_model="eval_NanoBEIR_mean_cosine_ndcg@10",
+    greater_is_better=True,
+
+    # Logging
+    logging_steps=0.01,                    # fraction: ~100 log lines/epoch
+    logging_first_step=True,
+    run_name="my-model",
+    report_to="trackio",                   # or "wandb", "tensorboard", "mlflow", "none"
+)
+```
+
+## Duration
+
+- `num_train_epochs`: most common. 1 for large datasets (>500k), 3-10 for small.
+- `max_steps`: use instead of epochs when you want a fixed compute budget. Overrides `num_train_epochs`.
+- For huge datasets where 1 epoch is overkill, pick `max_steps` matching your compute plan.
+
+## Batch size
+
+Effective batch size = `per_device_train_batch_size × num_gpus × gradient_accumulation_steps`.
+
+Rules of thumb:
+- **Contrastive losses (MNRL, GIST, SMNRL)**: push `per_device_train_batch_size` as high as VRAM allows. Larger in-batch negatives = better gradients. 64-256 typical.
+- **Regression losses (CoSENTLoss, CosineSimilarityLoss, etc.)**: batch size matters less. 16-64 works.
+- **Cross-encoders**: batch size is less critical for quality. 32-128 typical.
+- If you can't fit the desired per-device batch, use `gradient_accumulation_steps` to simulate it, but for MNRL-family losses this **does not** provide the same benefit as a real batch (in-batch negatives are still only per-device). Use `CachedMultipleNegativesRankingLoss` instead.
+
+## Learning rate and schedule
+
+- `2e-5` is the safe default for full fine-tuning of BERT-family encoders.
+- `1e-4` to `5e-4` for LoRA / PEFT adapters.
+- `2e-1` for training a `StaticEmbedding` model from scratch (much higher than transformers because each token is a free-floating vector with no upstream gradients).
+- `lr_scheduler_type="linear"` with `warmup_steps=0.1` is standard (a float `< 1` is interpreted as a fraction of total steps). `"cosine"` works equally well. `"constant_with_warmup"` is fine for very short runs. The legacy `warmup_ratio` was deprecated in transformers v5.2 in favor of `warmup_steps` accepting floats. Passing `warmup_ratio=...` still works but emits a DeprecationWarning.
+- If loss goes NaN, **drop LR first** before anything else.
+
+## Precision
+
+**The non-negotiable rule:** load the model in **fp32** (the default). Don't pass `torch_dtype=torch.bfloat16` to the model constructor or `model_kwargs`. Use the `bf16=True` / `fp16=True` flags below to enable **autocast**, not a weight cast. The trainer keeps the model and optimizer state in fp32 and autocasts activations to bf16/fp16 at forward/backward time. This preserves Adam's full-precision moments while giving you most of the bf16 throughput.
+
+Casting weights to bf16 *before* the optimizer is created puts the Adam state (`exp_avg`, `exp_avg_sq`) in bf16 too: bf16's 7-bit mantissa is too coarse for small gradient moments and you get silent quality regressions across runs.
+
+| Flag | When to use |
+|---|---|
+| `bf16=True` | Ampere (A10G, A100, 3090) and newer (Hopper, Ada). Preferred when supported. More numerically stable than fp16. Activations only. Weights stay fp32. |
+| `fp16=True` | Older GPUs (T4, V100, 2080, Titan V). Be prepared to drop LR or enable loss scaling if you see NaN. Activations only. Weights stay fp32. |
+| Neither | fp32 throughout. Slow. Only useful for debugging numerical issues. |
+
+Do not set both `bf16=True` and `fp16=True`.
+
+**Evaluator calls outside the trainer** (typically the pre-training baseline + a final post-training pass) don't get the trainer's autocast. Wrap them manually for the speedup, and note that the wrap is **only strictly required when the model uses `attn_implementation="flash_attention_2"`**, since FA2 kernels need bf16/fp16 inputs to function. Without FA2 the wrap is a throughput optimization, not a correctness requirement:
+
+```python
+import torch
+from contextlib import nullcontext
+
+def autocast_ctx():
+    if not torch.cuda.is_available():
+        return nullcontext()
+    dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
+    return torch.autocast("cuda", dtype=dtype)
+
+with autocast_ctx():
+    evaluator(model)            # baseline
+trainer.train()
+with autocast_ctx():
+    evaluator(model)            # post-training
+```
+
+**FlashAttention 2** wants bf16/fp16 inputs but doesn't require bf16 weights. Pass `model_kwargs={"attn_implementation": "flash_attention_2"}` *without* `torch_dtype`, and let `bf16=True` autocast feed bf16 activations to FA2. Weights stay fp32, optimizer state stays fp32.
+
+## Batch sampler (bi-encoder + sparse-encoder)
+
+`batch_sampler=BatchSamplers.NO_DUPLICATES` is critical for contrastive losses. Without it, the same (anchor, positive) can appear multiple times in one batch, turning legitimate positives into false negatives.
+
+Use `BatchSamplers.NO_DUPLICATES` for MNRL / SMNRL / CachedMNRL / GIST (the default recommendation). Use `BatchSamplers.GROUP_BY_LABEL` for batch-triplet losses (`BatchAllTripletLoss`, `BatchHardTripletLoss`). Use `BatchSamplers.NO_DUPLICATES_HASHED` only for very large datasets where per-batch string comparison gets slow.
+
+For multi-dataset training, the analogous `MultiDatasetBatchSamplers` class controls how to draw from each dataset (`ROUND_ROBIN`, `PROPORTIONAL`). Under DDP, each dataset is sharded automatically per process, no extra config needed. Set `multi_dataset_batch_sampler=...` once and it works for 1-GPU and N-GPU runs identically.
+
+## Evaluation and checkpointing
+
+```python
+eval_strategy="steps",
+eval_steps=0.1,                        # evaluate every 10% of training
+save_strategy="steps",
+save_steps=0.1,                        # save at the same cadence (required for load_best_model_at_end)
+save_total_limit=2,
+load_best_model_at_end=True,
+metric_for_best_model="eval_<EvaluatorName>_<metric>",
+greater_is_better=True,
+```
+
+**Prefer fractional values over absolute step counts.** `eval_steps=0.1` / `save_steps=0.1` / `logging_steps=0.01` are interpreted as fractions of total training steps (10 evals per epoch, 100 log lines per epoch) and auto-scale when the dataset size or epoch count changes. The HF Trainer converts a `float < 1` to `int(total_steps * fraction)` at init time, so the same config works whether you're training on 10k or 10M rows. No need to recompute absolute step counts each time.
+
+Use an absolute integer (e.g. `eval_steps=500`) only when you have a specific reason: comparing runs at known step counts, or when `max_steps` is set to an unusual value that makes fractions awkward.
+
+Non-negotiable rules:
+1. `save_steps` must be a multiple of `eval_steps` (or equal) when `load_best_model_at_end=True`, so the best-eval checkpoint is actually on disk. Matching them is the simplest path (e.g. both at `0.1`).
+2. If `eval_strategy="steps"` and you don't pass `eval_dataset`, training hangs. Either provide an eval dataset or set `eval_strategy="no"`.
+3. `metric_for_best_model` must match the exact key the evaluator writes. The pattern is usually `f"eval_{evaluator.primary_metric}"`. Common values:
+   - `NanoBEIREvaluator` (bi-encoder, cosine): `eval_NanoBEIR_mean_cosine_ndcg@10`
+   - `SparseNanoBEIREvaluator` (sparse, dot): `eval_NanoBEIR_mean_dot_ndcg@10`
+   - `CrossEncoderNanoBEIREvaluator` (rerank from BM25 top-100): `eval_NanoBEIR_R100_mean_ndcg@10`
+   - `EmbeddingSimilarityEvaluator(name="sts-dev")`: `eval_sts-dev_spearman_cosine`
+
+## Early stopping
+
+Add `EarlyStoppingCallback` via `callbacks=[...]`:
+
+```python
+from transformers import EarlyStoppingCallback
+
+trainer = SentenceTransformerTrainer(
+    ..., callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
+)
+```
+
+This requires `load_best_model_at_end=True` and `metric_for_best_model=...` to be set.
+
+`early_stopping_patience=3` means "stop if the best metric hasn't improved in 3 consecutive eval rounds." Use `early_stopping_threshold=0.001` to require a minimum improvement.
+
+**When it actually matters:**
+- **Cross-encoders**: strongly recommended. CE rerankers typically peak mid-training and then degrade. The best checkpoint is rarely the last. Early stopping saves compute *and* guards against quality regression.
+- **Bi-encoders and sparse encoders**: usually plateau rather than regress, so early stopping fires much less often. `load_best_model_at_end=True` alone gives you the right final model. Adding the callback is a belt-and-suspenders safety net.
+
+## Resuming training
+
+`trainer.train(resume_from_checkpoint=True)` resumes from the latest checkpoint in `output_dir`. Pass a specific path to resume from a specific step: `resume_from_checkpoint="models/my-model/checkpoint-500"`.
+
+State that persists across resumption: optimizer, scheduler, random seeds, trainer step counter. State that does **not**: dataset iteration order for `IterableDataset`. If you use streaming datasets, you must handle resumption yourself.
+
+## Hub push
+
+`push_to_hub=True` + `hub_model_id="your-username/my-model"` + `hub_strategy="every_save"` is the standard pattern. On HF Jobs, also pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` on the job submission. The four `hub_strategy` values: `"every_save"` (each checkpoint, mandatory for HF Jobs), `"end"` (final only), `"checkpoint"` (latest, overwrite), `"all_checkpoints"` (each as a separate commit).
+
+## Logging
+
+```python
+logging_steps=0.01,             # fraction: ~100 log lines per epoch (use an int for a fixed cadence)
+logging_first_step=True,        # log before any training; useful sanity check
+logging_dir=None,               # defaults to output_dir/runs
+report_to="trackio",            # or ["trackio", "tensorboard"] for multiple; "none" disables all
+run_name="meaningful-name",     # shown in the tracker UI
+```
+
+**Tracker recommendation:**
+- **Trackio** (default) for solo / small-team work: zero friction beyond `HF_TOKEN`. Auto-creates a Space at `https://huggingface.co/spaces/<your-username>/trackio` on the first run. Subsequent runs append and group by `run_name`.
+- **W&B** for larger teams or sweep / report features. `pip install wandb && wandb login` (or set `WANDB_API_KEY`).
+- **TensorBoard** for air-gapped environments. No remote dashboard.
+- **MLflow** when it's already the org standard.
+
+For trackio sweeps / ablations, use `trackio.init(project="...", name="...", group="v1", config={...})` before training to group related runs side-by-side. Without `trackio.init()`, defaults are derived from `run_name` and HF username.
+
+**Tracker gotchas:**
+- `report_to="all"` enables every installed integration (usually more than you want). `"none"` disables everything (the current `transformers` default). Always set explicitly.
+- Trackio on HF Jobs without `secrets={"HF_TOKEN": "$HF_TOKEN"}` fails silently. W&B on HF Jobs needs `WANDB_API_KEY` in `secrets`.
+- The HF Trainer logs only on rank 0 under DDP. Custom logging in your script may need explicit rank checks to avoid duplicate writes.
+
+## Memory-saving arguments
+
+```python
+gradient_checkpointing=True,    # trades compute for memory. ~30% slowdown, ~40% less memory.
+gradient_checkpointing_kwargs={"use_reentrant": False},
+torch_empty_cache_steps=1000,   # periodically clear PyTorch allocator cache
+dataloader_num_workers=2,       # parallel data loading, past 2 adds startup cost but no throughput
+dataloader_persistent_workers=True,  # recommended on Windows/macOS, can be False on Linux
+dataloader_pin_memory=True,
+```
+
+Do **not** combine `gradient_checkpointing=True` with any `Cached*` loss. They conflict.
+
+## Hyperparameter search
+
+`trainer.hyperparameter_search(...)` is supported for all three trainers via Hugging Face's `Trainer` API (uses Optuna, Ray Tune, Sigopt, or W&B as backends).
+
+Minimal example:
+
+```python
+def model_init(trial):
+    return SentenceTransformer("microsoft/mpnet-base")
+
+def hp_space(trial):
+    return {
+        "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
+        "num_train_epochs": trial.suggest_int("num_train_epochs", 1, 3),
+        "per_device_train_batch_size": trial.suggest_categorical(
+            "per_device_train_batch_size", [32, 64, 128]
+        ),
+    }
+
+trainer = SentenceTransformerTrainer(
+    model=None, model_init=model_init,
+    args=args, train_dataset=train_dataset, eval_dataset=eval_dataset,
+    loss=lambda model: MultipleNegativesRankingLoss(model),   # function that takes model -> loss
+    evaluator=evaluator,
+)
+
+best_run = trainer.hyperparameter_search(
+    hp_space=hp_space,
+    direction="maximize",
+    n_trials=10,
+    backend="optuna",
+)
+print(best_run)
+```
+
+Install a backend: `pip install optuna` (or `ray[tune]`).
+
+HPO is expensive. Don't reach for it until a single manually-tuned run is working end-to-end. For most production models, picking a reasonable LR from the range above and tuning batch size is enough.
+
+## Multi-task training args (brief)
+
+When training on a dict of datasets with a dict of losses, add:
+
+```python
+multi_dataset_batch_sampler=MultiDatasetBatchSamplers.PROPORTIONAL,  # or ROUND_ROBIN
+```
+
+See `../scripts/train_sentence_transformer_multi_dataset_example.py` (docstring covers per-dataset losses, single-loss + DatasetDict variant, samplers, gotchas).
+
+## Don't
+
+- Don't set `eval_strategy="epoch"` without setting `save_strategy="epoch"`. Checkpoint/eval alignment matters for `load_best_model_at_end`.
+- Don't set `remove_unused_columns=False` unless you have a custom collator that consumes metadata columns the loss doesn't see. The default (True) is safer. It drops unused columns automatically.
+- Don't set `seed` to verify reproducibility and then expect bit-for-bit identical runs on different GPUs or across PyTorch versions. Exact reproducibility across hardware is not guaranteed.
+- Don't tune `adam_beta1` / `adam_beta2` / `adam_epsilon` unless you have a specific reason. The defaults are fine for 99% of cases.

package/.agent/skills/train-sentence-transformers/references/troubleshooting.md

@@ -0,0 +1,225 @@
+# Troubleshooting
+
+Common failures in sentence-transformers training, with root causes and fixes. Organized by symptom.
+
+## Out of memory (OOM)
+
+**Symptom:** `torch.cuda.OutOfMemoryError` or `CUDA out of memory`.
+
+**Fixes in order:**
+
+1. Reduce `per_device_train_batch_size`. For MNRL, compensate via `CachedMultipleNegativesRankingLoss(model, mini_batch_size=...)` with a large outer batch.
+2. Set `gradient_accumulation_steps` to accumulate gradients over multiple smaller batches (for non-contrastive losses).
+3. Enable `gradient_checkpointing=True`. ~30% slower, ~40% less activation memory. **Incompatible with `Cached*` losses.**
+4. Use `bf16=True` instead of fp32 (if your GPU supports Ampere+).
+5. Reduce `max_seq_length` on the transformer module: `model[0].max_seq_length = 128`.
+6. Use LoRA (`peft`) for large decoder bases. See `../scripts/train_sentence_transformer_with_lora_example.py` (docstring covers when to use, hyperparams, QLoRA, gotchas).
+7. Move to a bigger GPU or multi-GPU (`accelerate launch`).
+
+**See also**: `hardware_guide.md` for VRAM estimates by batch size.
+
+## Loss is NaN or Inf
+
+**Symptom:** training loss printed as `nan` or `inf`, or suddenly jumps to a huge value.
+
+**Fixes:**
+
+1. **Drop learning rate** first. Try `5e-6` or `1e-6`.
+2. Enable `warmup_steps=0.1` (a float `< 1` is interpreted as a fraction of total steps), or set an absolute `warmup_steps=500`.
+3. Switch fp16 -> bf16 (more numerically stable). If stuck on fp16 (older GPU), try disabling fp16 for a sanity run.
+4. Check the dataset for broken rows: empty strings, NaN label values, mismatched column counts. Inspect a few rows with `print(dataset[:5])`.
+5. For custom losses or unusual base models (very long context), consider adding `max_grad_norm=1.0` to the training args.
+6. Check tokenization: some tokenizers emit no tokens for certain unicode / whitespace-only inputs, which can cause downstream NaN in mean pooling.
+
+## Metrics don't improve / are at baseline
+
+**Symptom:** eval metrics after training are the same as before.
+
+**Fixes:**
+
+1. **Re-run dataset inspector** with `--loss <your-loss>`. Most likely cause: column order wrong, label column not detected, or loss expects a different shape.
+2. For retrieval: **your negatives are too easy**. Mine hard negatives (`scripts/mine_hard_negatives.py`).
+3. Check `metric_for_best_model`. If the key doesn't match what the evaluator writes, the trainer silently uses the final checkpoint (not the best).
+4. Confirm `BatchSamplers.NO_DUPLICATES` is set for MNRL-style losses. Without it, the in-batch negative signal is corrupted.
+5. Base model is wrong for the task (e.g. decoder-only LLM for short-text STS). Try a different base.
+6. Learning rate too low. Default `2e-5` works for encoders. LoRA wants `1e-4+`. Static embeddings want about `2e-1` (much higher than transformers).
+7. Dataset too small for the chosen loss. Contrastive losses need >10k pairs to be meaningful.
+
+## Training hangs at the first eval
+
+**Symptom:** training starts, then hangs indefinitely at the first evaluation step.
+
+**Fix:** you set `eval_strategy="steps"` or `"epoch"` but either didn't pass `eval_dataset` or passed an empty one. Either provide an `eval_dataset` or set `eval_strategy="no"`.
+
+## Training hangs at startup (multi-GPU)
+
+**Symptom:** `accelerate launch` runs, prints "Found X GPUs" + model-loading messages, then hangs.
+
+**Fixes:**
+
+1. If using a custom dataset class, ensure `__len__` is implemented and returns a consistent length across processes.
+2. If using `batch_sampler=BatchSamplers.NO_DUPLICATES` with a very small dataset relative to the world size, batches may not form. Use a larger dataset or smaller per-device batch.
+3. Check for mismatched PyTorch / CUDA versions across nodes. `nvidia-smi` + `python -c "import torch; print(torch.version.cuda)"` on each node.
+4. NCCL timeout. Set `NCCL_TIMEOUT=300` (seconds) env var.
+
+## `CachedMultipleNegativesRankingLoss` crashes
+
+**Symptom:** error like "element 0 of tensors does not require grad", or a cryptic autograd error.
+
+**Fix:** you have `gradient_checkpointing=True`. The cached losses do their own forward/backward orchestration. Gradient checkpointing conflicts with it. Disable `gradient_checkpointing`.
+
+Same applies to `CachedSpladeLoss`, `CachedGISTEmbedLoss`, and any other `Cached*` loss.
+
+## Hub push fails
+
+**Symptom:** `HTTPError: 401` or `403` during `push_to_hub`.
+
+**Fixes:**
+
+1. Run `hf auth whoami`. If it fails, run `hf auth login`.
+2. Token needs **write** permission. Regenerate from https://huggingface.co/settings/tokens.
+3. Repo either must exist and you have write access, or `hub_private_repo=True`/`False` is set so the library can create it.
+4. On HF Jobs: pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` on the job submission.
+
+## Tracker (Trackio / W&B / TensorBoard) not logging
+
+**Symptom:** training runs but no metrics appear in the tracker UI.
+
+**Fixes:**
+
+- Trackio: confirm `pip install trackio` succeeded. No login step. Trackio uses your `HF_TOKEN` (set by `hf auth login` or the `HF_TOKEN` env var). On HF Jobs, `HF_TOKEN` must be in `secrets`.
+- W&B: confirm `wandb login` succeeded, or `WANDB_API_KEY` env var is set. On HF Jobs, `WANDB_API_KEY` must be in `secrets`.
+- `report_to` not set to the right tracker: `report_to="trackio"` (or `"wandb"`, or a list like `["trackio", "tensorboard"]`).
+- TensorBoard: check `logging_dir` (defaults to `output_dir/runs/<timestamp>`). Point TB at the parent.
+
+## Base model loads but `encode` produces garbage
+
+**Symptom:** `model.encode(["test"])` returns constant vectors, or all-zeros, or NaN.
+
+**Fixes:**
+
+1. You loaded a classification fine-tune as a base (e.g. a BERT fine-tuned on SQuAD). The CLS head is a QA head, not a usable pooling layer. Use the underlying pretrained encoder (`bert-base-uncased`), not the task-specific checkpoint.
+2. For decoder models: you're using mean pooling on causal attention. Switch to last-token pooling.
+3. For SPLADE: the model's MLM head isn't initialized properly. Make sure the base has `AutoModelForMaskedLM` compatibility.
+
+## Dataset loads but eval is trivially correct
+
+**Symptom:** eval metric is 1.0 (perfect) from the first evaluation step.
+
+**Fix:** your eval set overlaps the train set. Check `dataset.train_test_split(test_size=...)` was called correctly, or that the Hub dataset's `train` vs. `dev` splits are actually disjoint.
+
+## Model-card generation fails
+
+**Symptom:** warning about model-card generation, or `README.md` is missing after `save_pretrained`.
+
+**Fixes:**
+
+- `codecarbon` may be attempting to write emissions and failing. Set `CODECARBON_LOG_LEVEL=error` or uninstall codecarbon.
+- Some training state couldn't be serialized (custom objects with unusual types). Pass `model_card_data=SentenceTransformerModelCardData(...)` with explicit fields to bypass inference.
+
+## `num_labels` mismatch on CrossEncoder
+
+**Symptom:** `BinaryCrossEntropyLoss` raises about mismatched dims, or `CrossEntropyLoss` complains.
+
+**Fix:** `num_labels=1` goes with `BinaryCrossEntropyLoss`. `num_labels>=2` goes with `CrossEntropyLoss`. Set `CrossEncoder("...", num_labels=1)` for BCE.
+
+## CrossEncoder eval nDCG crashes after distillation / listwise / pairwise training
+
+**Symptom:** training loss looks healthy, baseline eval looked fine, but post-training eval nDCG drops a lot (e.g. 0.59 → 0.14). Every checkpoint after the first eval is below baseline.
+
+**Root cause:** the default `Sigmoid` activation function on `CrossEncoder(num_labels=1, ...)` saturates raw logits >5 to ~1.0. Distillation/listwise/pairwise losses (`MSELoss`, `MarginMSELoss`, `LambdaLoss`, `RankNetLoss`, `ListNetLoss`, `ListMLELoss`, `PListMLELoss`, `ADRMSELoss`) operate on raw logits. Once the model learns to push positives past the saturation point, ranking information is lost.
+
+**Fix:** construct the model with an `Identity` activation:
+
+```python
+import torch.nn as nn
+model = CrossEncoder("...", num_labels=1, activation_fn=nn.Identity())
+```
+
+Keep the default `Sigmoid` only for `BinaryCrossEntropyLoss` (which uses BCE-with-logits internally and wants sigmoidable inputs).
+
+## LambdaLoss training loss is tiny (e.g. 1e-4): is training broken?
+
+**Symptom:** with `LambdaLoss(model, weighting_scheme=NDCGLoss2PPScheme())` and large `K` (>=64), training loss prints in the 1e-3 to 1e-5 range.
+
+**Root cause:** `NDCGLoss2PPScheme` normalizes by the discount-weighted pair count, which scales roughly with K. The numeric magnitude of the loss isn't the signal you should track.
+
+**Fix:** ignore training loss for LambdaLoss. Watch the eval metric (`eval_NanoBEIR_R100_mean_ndcg@10` or your `metric_for_best_model`) instead. If eval is moving in the right direction and loss is "too small", that's expected.
+
+## LambdaLoss OOM: what to drop first
+
+**Symptom:** `CUDA out of memory` during a `LambdaLoss` training step, especially at K>=64.
+
+**Recovery order:**
+
+1. **Lower `mini_batch_size`** on the loss first. The internal forward chunking preserves the K-list semantic. This is the cheapest knob and doesn't change the experiment.
+2. Lower `per_device_train_batch_size` and compensate with `gradient_accumulation_steps` to keep total batch fixed.
+3. **Reduce K (the candidate-list length per query) only as a last resort.** Lowering K changes what the loss is computing. This is an experimental change, not a memory tweak.
+
+For very large K (>=128), `NDCGLoss2PPScheme` materializes O(K²) weight buffers per query that aren't covered by the forward chunking, so even small `mini_batch_size` may not be enough. At that point K is the right knob.
+
+## Loading a model with a custom inline `nn.Module` raises `ImportError`
+
+**Symptom:** training and saving via `train.py` works fine. Loading the saved model from any other script (`predict.py`, a notebook, or another package) fails with:
+
+```
+ImportError: Module "__main__" does not define a "ClassifierHead" attribute
+```
+
+**Root cause:** when a custom `nn.Module` is defined inline in a script, Python records its qualname as `__main__.ClassifierHead`. ST writes that qualname into `modules.json` at save time. Loading from a different entry point can't resolve `__main__.ClassifierHead`. The class doesn't exist in the loader's `__main__`.
+
+**Fixes (any one):**
+
+1. Move the class to an importable module: `from my_pkg.heads import ClassifierHead`. Save again so `modules.json` records `my_pkg.heads.ClassifierHead`.
+2. Build the same shape using stock ST modules (`Dense + LayerNorm + Dense`) instead of a custom class. Those are always loadable.
+3. Document that the model is only loadable from the same script (acceptable for one-off experiments, not for shipping models).
+
+## SPLADE embeddings are dense (not sparse)
+
+**Symptom:** after training, `(embedding != 0).sum(dim=-1)` is in the thousands, not ~30-250.
+
+**Fixes:**
+
+1. You're missing the `SpladeLoss` wrapper. `SparseMultipleNegativesRankingLoss` alone doesn't add FLOPS regularization. Wrap: `SpladeLoss(model, loss=inner_loss, query_regularizer_weight=5e-5, document_regularizer_weight=3e-5)`.
+2. Regularizer weights too low. Increase to 1e-4 or higher.
+3. Scheduler wiped out early. `SpladeRegularizerWeightSchedulerCallback` ramps weights from 0 to target over the first ~33% of training (`warmup_ratio=1/3` by default, configurable). On very short runs this never reaches target. Either train longer or set `query_regularizer_weight` / `document_regularizer_weight` higher to compensate.
+
+## `ValueError: The dataset has ... columns but the loss expects N`
+
+**Fix:** column count mismatch. Drop extra columns (`dataset.remove_columns([...])`) or reorder with `select_columns`. Names don't matter. Count and order do.
+
+## Encoding on CPU is painfully slow
+
+**Symptom:** a few dozen sentences per second instead of 1000s.
+
+**Fixes:**
+
+- Ensure the model is on GPU: `model.to("cuda")` (or `device_map="auto"` when loading).
+- For genuine CPU inference (no GPU available), consider switching to a `StaticEmbedding`-based model, ~1000x faster on CPU than a transformer.
+
+## Hub model loads, but `model.encode(prompt_name="query")` acts like no prompt was applied
+
+**Fix:** the saved model doesn't have `prompts` in `config_sentence_transformers.json`. When training, set `model.prompts = args.prompts` before `save_pretrained`, or use `SentenceTransformerModelCardData(prompts=...)`.
+
+## `accelerate launch` runs only on one GPU
+
+**Fix:** run `accelerate config` first, set number of GPUs and precision. Or pass explicitly: `accelerate launch --multi_gpu --num_processes=4 train.py`.
+
+## Cached loss + PEFT adapter fails to backprop
+
+**Symptom:** "None of the inputs have requires_grad=True" when using Cached* loss with a LoRA adapter.
+
+**Fix:** after `add_adapter`, call:
+
+```python
+model.transformers_model.enable_input_require_grads()
+```
+
+This ensures gradient flow through the frozen base + trainable adapter.
+
+## Related reference docs
+
+- `training_args.md` (shared): the args that affect all of the above.
+- `hardware_guide.md` (shared): VRAM sizing and multi-GPU.
+- `dataset_formats.md` (shared): column/loss validation.
+- `losses_sentence_transformer.md` / `losses_cross_encoder.md` / `losses_sparse_encoder.md` (per-model-type catalogs): loss-specific quirks.