tribunal-kit 9.2.3 → 9.2.5

package/skills/thermo-nuclear-code-quality-review/SKILL.md

@@ -1,62 +1,62 @@
----
-name: thermo-nuclear-code-quality-review
-description: "Use when Run an extremely strict maintainability review for abstraction quality, giant files, spaghetti-condition growth, and architectural debt."
-version: 5.0.0
-last-updated: 2026-09-13
-skills:
-  - clean-code
-  - codebase-design
-  - code-review-checklist
-tools: Read, Grep, Glob, Bash, Edit, Write
-scripts-binding:
-  - .agent/scripts/lint_runner.js
-  - .agent/scripts/verify_all.js
----
-
-# Thermo-Nuclear Code Quality Review — Zero-Tolerance Maintainability Audit
-
----
-
-## 🛠️ Technical Architecture & Reference Recipes
-
----
-
----
-
-## 5 Zero-Tolerance Audit Rules
-
-### 1. File Size Hard Limit (Max 300 Lines)
-
-- Flag ANY single file exceeding 300 lines of code. Demand decomposition into modular helper components or sub-packages.
-
-### 2. Cyclomatic Complexity & Nested Conditionals (Max 3 Levels)
-
-- Flag any function with nested conditionals deeper than 3 levels (`if -> if -> if -> for`). Require early guard clause returns.
-
-```typescript
-// ❌ REJECTED: Deep nested spaghetti
-function processOrder(order: Order) {
-  if (order) {
-    if (order.isValid) {
-      if (order.items.length > 0) {
-        // Business logic hidden 4 levels deep
-      }
-    }
-  }
-}
-
-// ✅ APPROVED: Early guard returns
-function processOrder(order: Order) {
-  if (!order || !order.isValid) return;
-  if (order.items.length === 0) return;
-  // Business logic at top indentation level
-}
-```
-
-### 3. Magic Values & String Literals
-
-- Hardcoded status strings (`"PENDING_APPROVAL_V2"`) or raw numbers (`86400000`) outside central `const` or `enum` declarations trigger immediate failure.
-
-### 4. Direct Dependency Coupling
-
-- Modules importing concrete database models or vendor SDKs directly inside domain business logic instead of interface adapters are rejected.
+---
+name: thermo-nuclear-code-quality-review
+description: "Use when Run an extremely strict maintainability review for abstraction quality, giant files, spaghetti-condition growth, and architectural debt."
+version: 5.0.0
+last-updated: 2026-09-13
+skills:
+  - clean-code
+  - codebase-design
+  - code-review-checklist
+tools: Read, Grep, Glob, Bash, Edit, Write
+scripts-binding:
+  - .agent/scripts/lint_runner.js
+  - .agent/scripts/verify_all.js
+---
+
+# Thermo-Nuclear Code Quality Review — Zero-Tolerance Maintainability Audit
+
+---
+
+## 🛠️ Technical Architecture & Reference Recipes
+
+---
+
+---
+
+## 5 Zero-Tolerance Audit Rules
+
+### 1. File Size Hard Limit (Max 300 Lines)
+
+- Flag ANY single file exceeding 300 lines of code. Demand decomposition into modular helper components or sub-packages.
+
+### 2. Cyclomatic Complexity & Nested Conditionals (Max 3 Levels)
+
+- Flag any function with nested conditionals deeper than 3 levels (`if -> if -> if -> for`). Require early guard clause returns.
+
+```typescript
+// ❌ REJECTED: Deep nested spaghetti
+function processOrder(order: Order) {
+  if (order) {
+    if (order.isValid) {
+      if (order.items.length > 0) {
+        // Business logic hidden 4 levels deep
+      }
+    }
+  }
+}
+
+// ✅ APPROVED: Early guard returns
+function processOrder(order: Order) {
+  if (!order || !order.isValid) return;
+  if (order.items.length === 0) return;
+  // Business logic at top indentation level
+}
+```
+
+### 3. Magic Values & String Literals
+
+- Hardcoded status strings (`"PENDING_APPROVAL_V2"`) or raw numbers (`86400000`) outside central `const` or `enum` declarations trigger immediate failure.
+
+### 4. Direct Dependency Coupling
+
+- Modules importing concrete database models or vendor SDKs directly inside domain business logic instead of interface adapters are rejected.

package/skills/thinking-protocol/SKILL.md

@@ -1,21 +1,21 @@
----
-name: thinking-protocol
-description: "Use when Tribunal Agent Kit thinking and cognitive reasoning rules. Helps agents structure their thoughts and follow protocols."
-version: 5.0.0
-last-updated: 2026-09-13
-skills:
-  - fabel-protocol
-  - behavioral-modes
-  - clean-code
-tools: Read, Grep, Glob, Bash, Edit, Write
-scripts-binding:
-  - .agent/scripts/checklist.js
-  - .agent/scripts/verify_all.js
-  - .agent/scripts/lint_runner.js
----
-
-# Thinking Protocol — Cognitive Reasoning Rules
-
----
-
-## 🛠️ Technical Architecture & Reference Recipes
+---
+name: thinking-protocol
+description: "Use when Tribunal Agent Kit thinking and cognitive reasoning rules. Helps agents structure their thoughts and follow protocols."
+version: 5.0.0
+last-updated: 2026-09-13
+skills:
+  - fabel-protocol
+  - behavioral-modes
+  - clean-code
+tools: Read, Grep, Glob, Bash, Edit, Write
+scripts-binding:
+  - .agent/scripts/checklist.js
+  - .agent/scripts/verify_all.js
+  - .agent/scripts/lint_runner.js
+---
+
+# Thinking Protocol — Cognitive Reasoning Rules
+
+---
+
+## 🛠️ Technical Architecture & Reference Recipes

package/skills/to-spring-or-not-to-spring/SKILL.md

@@ -1,43 +1,43 @@
----
-name: to-spring-or-not-to-spring
-description: "Use when Audit and decide when to use physics-based spring animations (stiffness, damping, mass) vs duration-based cubic-bezier easing curves."
-version: 5.0.0
-last-updated: 2026-09-13
-skills:
-  - motion-engineering
-  - framer-motion-expert
-  - 60fps-animation
-tools: Read, Grep, Glob, Bash, Edit, Write
-scripts-binding:
-  - .agent/scripts/lint_runner.js
-  - .agent/scripts/verify_all.js
----
-
-# To Spring or Not to Spring — Motion Physics Decision Matrix
-
----
-
-## 🛠️ Technical Architecture & Reference Recipes
-
----
-
----
-
-## Spring vs Duration Decision Matrix
-
-| Motion Scenario                                      | Use Spring Physics?   | Recommended Parameters / Curve                                               |
-| ---------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------- |
-| **Interruptible Gestures** (Drag, Swipe, Sheet pull) | ✅ **ALWAYS**         | `type: "spring", stiffness: 300, damping: 30` (adapts to drag velocity)      |
-| **Button Press Feedback** (`:active`)                | ❌ **NO (Use Curve)** | `transition: transform 120ms cubic-bezier(0.2, 0, 0, 1)`                     |
-| **Modal / Dialog Entrance**                          | ❌ **NO (Use Curve)** | `transition: all 220ms cubic-bezier(0.16, 1, 0.3, 1)`                        |
-| **Badge Bouncing / Celebration**                     | ✅ **YES**            | `type: "spring", stiffness: 400, damping: 15` (intentional overshoot bounce) |
-| **Page / Route Transitions**                         | ❌ **NO (Use Curve)** | `transition: opacity 200ms ease-out`                                         |
-| **Toggle Switch Flip**                               | ✅ **YES**            | `type: "spring", stiffness: 500, damping: 35` (crisp snap without wobble)    |
-
----
-
-## 3 Core Spring Parameters
-
-1. **Stiffness** (Rigidity): High stiffness ($400+$) = snappy and tight; Low stiffness ($100$) = slow and lazy.
-2. **Damping** (Friction): High damping ($30+$) = zero overshoot bounce; Low damping ($10$) = heavy oscillating bounce.
-3. **Mass** (Weight): Higher mass ($2.0$) = feels heavy with momentum inertia.
+---
+name: to-spring-or-not-to-spring
+description: "Use when Audit and decide when to use physics-based spring animations (stiffness, damping, mass) vs duration-based cubic-bezier easing curves."
+version: 5.0.0
+last-updated: 2026-09-13
+skills:
+  - motion-engineering
+  - framer-motion-expert
+  - 60fps-animation
+tools: Read, Grep, Glob, Bash, Edit, Write
+scripts-binding:
+  - .agent/scripts/lint_runner.js
+  - .agent/scripts/verify_all.js
+---
+
+# To Spring or Not to Spring — Motion Physics Decision Matrix
+
+---
+
+## 🛠️ Technical Architecture & Reference Recipes
+
+---
+
+---
+
+## Spring vs Duration Decision Matrix
+
+| Motion Scenario                                      | Use Spring Physics?   | Recommended Parameters / Curve                                               |
+| ---------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------- |
+| **Interruptible Gestures** (Drag, Swipe, Sheet pull) | ✅ **ALWAYS**         | `type: "spring", stiffness: 300, damping: 30` (adapts to drag velocity)      |
+| **Button Press Feedback** (`:active`)                | ❌ **NO (Use Curve)** | `transition: transform 120ms cubic-bezier(0.2, 0, 0, 1)`                     |
+| **Modal / Dialog Entrance**                          | ❌ **NO (Use Curve)** | `transition: all 220ms cubic-bezier(0.16, 1, 0.3, 1)`                        |
+| **Badge Bouncing / Celebration**                     | ✅ **YES**            | `type: "spring", stiffness: 400, damping: 15` (intentional overshoot bounce) |
+| **Page / Route Transitions**                         | ❌ **NO (Use Curve)** | `transition: opacity 200ms ease-out`                                         |
+| **Toggle Switch Flip**                               | ✅ **YES**            | `type: "spring", stiffness: 500, damping: 35` (crisp snap without wobble)    |
+
+---
+
+## 3 Core Spring Parameters
+
+1. **Stiffness** (Rigidity): High stiffness ($400+$) = snappy and tight; Low stiffness ($100$) = slow and lazy.
+2. **Damping** (Friction): High damping ($30+$) = zero overshoot bounce; Low damping ($10$) = heavy oscillating bounce.
+3. **Mass** (Weight): Higher mass ($2.0$) = feels heavy with momentum inertia.

package/skills/train-sentence-transformers/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: train-sentence-transformers
+description: Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder, dense or static embedding model for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker, pair scoring for two-stage retrieval / pair classification), `SparseEncoder` (SPLADE, sparse embedding model for learned-sparse retrieval), and `MultiVectorEncoder` (ColBERT / late-interaction, per-token embeddings scored with MaxSim). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.
+---
+
+# Train a sentence-transformers Model
+
+**This SKILL.md is a router, not a manual.** It tells you which references and example scripts to load for your task. The actual content (recommended losses, evaluators, training-script structure, model selection, training-arg knobs, troubleshooting) lives in `references/` and `scripts/`.
+
+**Do not synthesize a training script from this file alone.** Open the per-type production template (`scripts/train_<type>_example.py`) and copy it as your starting point. The templates contain load-bearing scaffolding (autocast helper, model-card class, logger silencing list, `force=True`, `seed`, TF32, version-compatible imports, named-evaluator metric handling) that prior agent runs have repeatedly missed when rolling their own from a synthesized snippet.
+
+## 1. Identify the model type
+
+| Tag | Class | What it does | When to pick |
+|---|---|---|---|
+| **[SentenceTransformer]** | `SentenceTransformer` (bi-encoder) | Maps each input to a fixed-dim dense vector | Retrieval, similarity, clustering, classification, paraphrase mining, dedup |
+| **[CrossEncoder]** | `CrossEncoder` (reranker) | Scores `(query, passage)` pairs jointly | Two-stage retrieval (rerank top-100 from bi-encoder), pair classification |
+| **[SparseEncoder]** | `SparseEncoder` (SPLADE) | Sparse vectors over the vocabulary | Learned-sparse retrieval, inverted-index backends (Elasticsearch / OpenSearch / Lucene) |
+| **[MultiVectorEncoder]** | `MultiVectorEncoder` (ColBERT) | One embedding per token, scored with MaxSim | Late-interaction retrieval, recall gains over bi-encoders at higher storage cost, multimodal (ColPali / ColQwen2) |
+
+Tiebreakers when the request is ambiguous: "embedding model" / "vector search" / "similarity" → **[SentenceTransformer]**. "rerank" / "ranker" / "two-stage" → **[CrossEncoder]**. "SPLADE" / "sparse" / "inverted index" → **[SparseEncoder]**. "ColBERT" / "late interaction" / "multi-vector" / "MaxSim" / "ColPali" / "ColQwen" → **[MultiVectorEncoder]**. If still unclear, ask.
+
+## 2. Required reading
+
+**Read these in full before writing any code. Do not triage by perceived relevance.**
+
+### Per-type: always required
+
+**[SentenceTransformer]**
+- `references/losses_sentence_transformer.md`: loss-to-data-shape mapping, `BatchSamplers.NO_DUPLICATES` requirement for MNRL-family, `Cached*` ↔ `gradient_checkpointing` incompatibility.
+- `references/evaluators_sentence_transformer.md`: evaluator-to-task mapping, `metric_for_best_model` key construction (named vs unnamed), per-evaluator `primary_metric` values.
+- `references/model_architectures.md`: encoder vs decoder vs static vs Router pipelines, pooling rules (mean / cls / lasttoken), auto-mean-pooling behavior for fresh-start MLM bases.
+- `scripts/train_sentence_transformer_example.py`: production template. Copy this as your starting point.
+
+**[CrossEncoder]**
+- `references/losses_cross_encoder.md`: pointwise / pairwise / listwise / distillation, `pos_weight` derivation, `activation_fn=Identity()` mandatory for non-BCE losses (silent eval-rank collapse otherwise).
+- `references/evaluators_cross_encoder.md`: `CrossEncoderRerankingEvaluator` recipe, named-evaluator key format `eval_{name}_{primary_metric}`.
+- `scripts/train_cross_encoder_example.py`: production template. Copy this as your starting point.
+
+**[SparseEncoder]**
+- `references/losses_sparse_encoder.md`: `SpladeLoss` wrapper requirement, FLOPS regularizer weights, smoke-test active-dim ramp behavior.
+- `references/evaluators_sparse_encoder.md`: `SparseNanoBEIREvaluator` (English-only) and the in-domain alternative, `eval_{name}_{primary_metric}` key format.
+- `scripts/train_sparse_encoder_example.py`: production template. Copy this as your starting point.
+
+**[MultiVectorEncoder]**
+- `references/losses_multi_vector_encoder.md`: MaxSim scoring, scale choice per scoring mode (`scale=1.0` for MaxSim, roughly the average query length for MeanMaxSim), MNRL / CachedMNRL / MarginMSE / DistillKLDiv, XTR-vs-ColBERT scoring, CachedMNRL ↔ `gradient_checkpointing` incompatibility.
+- `references/evaluators_multi_vector_encoder.md`: `MultiVectorNanoBEIREvaluator` (English-only) and the in-domain alternative, `eval_NanoBEIR_mean_maxsim_ndcg@10` key format, distillation-eval spearman variant.
+- `scripts/train_multi_vector_encoder_example.py`: production template. Copy this as your starting point.
+
+### Cross-cutting: always required (regardless of task)
+
+- `references/training_args.md`: `TrainingArguments` knobs, precision rules (load fp32 + autocast bf16/fp16, never `torch_dtype=bfloat16`), `warmup_steps` (float) vs deprecated `warmup_ratio`, `save_steps` must be a multiple of `eval_steps` for `load_best_model_at_end`, schedulers, HPO, tracker, resume, hub-push variants.
+- `references/dataset_formats.md`: column-matching rules (label name auto-detection, column-order-not-name), reshaping recipes, hard-negative mining options.
+- `references/base_model_selection.md`: discovery commands, per-type model namespaces, ModernBERT-family `max_seq_length=8192` trap, `datasets >= 4` script-loader rejection, non-English starting-point shortcuts.
+- `references/troubleshooting.md`: symptom-indexed failure recipes. Skim the section headings on every run, even a healthy one. The "Metrics don't improve" and "Hub push fails" entries cover bugs that bite frequently and are cheaper to recognize before they fire than to debug after.
+
+### Cross-cutting: load when applicable
+
+- `references/hardware_guide.md`: VRAM sizing, multi-GPU, FSDP / DeepSpeed, HF Jobs flavors. Required for >24GB models, multi-GPU, or HF Jobs runs.
+- `references/hf_jobs_execution.md`: required when running on HF Jobs.
+- `references/prompts_and_instructions.md`: required when using prompt-tuned bases (E5, BGE, GTE, Qwen3-Embedding, Instructor, Nomic, etc.) or adding `query: ` / `passage: ` style prefixes.
+
+### Variant scripts (open when the task matches)
+- **[SentenceTransformer]** `scripts/train_sentence_transformer_<matryoshka|multi_dataset|with_lora|distillation|make_multilingual|static_embedding>_example.py`.
+- **[CrossEncoder]** `scripts/train_cross_encoder_<distillation|listwise>_example.py`.
+- **[SparseEncoder]** `scripts/train_sparse_encoder_distillation_example.py`.
+- Hard-negative mining CLI: `scripts/mine_hard_negatives.py`.
+
+## 3. Defaults
+
+Override only if the user specifies otherwise:
+- **Local execution.** Pitch HF Jobs only if local hardware can't fit the job.
+- **Single run.** After it completes, propose experimentation if the user would benefit (weak/marginal verdict, "see how high you can push it" framing, etc.). Iteration rules in `references/training_args.md` (Experimentation section).
+- **Public Hub push at end-of-run, wrapped in try-except.** On HF Jobs (ephemeral env) ALSO enable in-trainer push (`push_to_hub=True` + `hub_strategy="every_save"`). Details in `references/hf_jobs_execution.md`.
+
+## 4. Constraints the produced script must satisfy
+
+These are non-negotiable contracts. Implementation lives in the production templates and references. Do not reinvent.
+
+- Capture the pre-training evaluator score as `baseline_eval` **before** `trainer.train()`.
+- Emit a single end-of-run line: `VERDICT: WIN|MARGINAL|REGRESSION | score=... | baseline=... | delta=...`. A monitor scrapes for this.
+- Silence `httpx`, `httpcore`, `huggingface_hub`, `urllib3`, `filelock`, `fsspec` to WARNING (otherwise HF download URLs flood the agent's context).
+- Tee logs to `logs/{RUN_NAME}.log`.
+- End with `model.push_to_hub(...)` wrapped in `try/except`.
+- Smoke-test before any long run (`max_steps=1` + tiny dataset slice). The production templates show one common pattern (`SMOKE_TEST` env var).
+- **[CrossEncoder]** Include `EarlyStoppingCallback(patience>=3)`. CE rerankers often peak mid-training and regress.
+- **[SparseEncoder]** Log `query_active_dims` / `corpus_active_dims` on the verdict line. High nDCG with collapsed sparsity is not a win. The keys come back name-prefixed (e.g. `..._query_active_dims`). Use suffix matching to pluck them. See the SPARSE production template for the exact pattern.
+- **[MultiVectorEncoder]** Match `scale` to the scoring mode on any MNRL-family loss: near `1.0` for unnormalized MaxSim (do not copy `scale=20.0` from bi-encoder MNRL), roughly the average query length with length-normalized MeanMaxSim, since each score is divided by its query's token count. `XTRScores` is a train-only `similarity_fct`: the evaluators reject it, so evaluation always scores with MaxSim, including for XTR-trained models.
+
+## 5. Workflow
+
+1. Identify the model type (§1). Ask if ambiguous.
+2. Load the §2 required-reading files for that type.
+3. Open `scripts/train_<type>_example.py` and copy it as your starting point.
+4. Replace `MODEL_NAME`, `DATASET_NAME`, `RUN_NAME`, the loss, and the evaluator with the user's task. Cross-check loss/data-shape match against `references/losses_<type>.md`. Cross-check the `metric_for_best_model` key against `references/evaluators_<type>.md` (named evaluators format the key as `eval_{name}_{primary_metric}`).
+5. Smoke-test (`max_steps=1`).
+6. Run.
+7. After the run, append to `logs/experiments.md` and propose iteration if the verdict is weak/marginal.
+
+## Prerequisites
+
+```bash
+pip install "sentence-transformers[train]>=5.0"        # add [train,image] / [audio] / [video] for [SentenceTransformer] multimodal
+                                                       # [MultiVectorEncoder] requires >=6.0
+pip install trackio                                    # optional tracker (or wandb / tensorboard / mlflow)
+hf auth login                                          # or set HF_TOKEN with write scope (for Hub push)
+```
+
+GPU strongly recommended. CPU works only for demos and `[SentenceTransformer]` `StaticEmbedding`.

package/skills/train-sentence-transformers/references/base_model_selection.md

@@ -0,0 +1,79 @@
+# Base Model Selection
+
+Leaderboards rotate every few months. Don't trust any hardcoded "best" pick. Discover current options live. Run **both** sort orders since most-downloaded surfaces proven options and trending surfaces recent SOTA that may not have download volume yet.
+
+## Discovery commands
+
+**[BI]**:
+```bash
+hf models list --filter sentence-transformers --sort downloads --limit 20
+hf models list --filter sentence-transformers --sort trending  --limit 20
+```
+
+**[CE]**:
+```bash
+hf models list --filter sentence-transformers --filter text-ranking --sort downloads --limit 20
+hf models list --filter sentence-transformers --filter text-ranking --sort trending  --limit 20
+```
+
+**[SPARSE]**:
+```bash
+hf models list --filter sentence-transformers --filter sparse-encoder --sort downloads --limit 20
+hf models list --filter sentence-transformers --filter sparse-encoder --sort trending  --limit 20
+```
+
+Optional language narrowing (any type): add `--filter <language-code>`. Not all multilingual models tag each language, so missing matches doesn't mean the model can't handle that language. Re-run without the filter to compare.
+
+```bash
+hf models card <id> --text                        # confirm dimensions, max_seq_length, license, languages
+```
+
+Cross-check the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard) (pick the relevant tab) before committing to a multi-hour run.
+
+## [BI] Bi-Encoder
+
+Continue from an existing retriever beats fresh-start + 100k-500k pairs. Common namespaces as of 2026-Q2 (verify against discovery commands, since the field rotates):
+
+- **English encoder retrievers**: `sentence-transformers/all-*` (MiniLM-L6-v2, mpnet-base-v2 still the most-downloaded models on the Hub), `BAAI/bge-*-en-v1.5`, `nomic-ai/nomic-embed-text-v1.5`, `mixedbread-ai/mxbai-embed-large-v1`, `Alibaba-NLP/gte-*`, `Snowflake/snowflake-arctic-embed-*`, `jinaai/jina-embeddings-v5-text-small` / `-nano`, `microsoft/harrier-oss-v1-270m` / `-0.6b`.
+- **Multilingual encoder retrievers**: `sentence-transformers/paraphrase-multilingual-*`, `intfloat/multilingual-e5-*`, `ibm-granite/granite-embedding-*-multilingual-r2`, `google/embeddinggemma-300m`, `voyageai/voyage-4-nano`.
+- **Long documents (8k+)**: `nomic-ai/modernbert-embed-*`, `answerdotai/ModernBERT-large`.
+- **Decoder LLM retrievers** (multilingual, **need last-token pooling**): `Qwen/Qwen3-Embedding-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Embedding-*` (multimodal), `codefuse-ai/F2LLM-v2-*`.
+- **Fresh-start English** (≥500k pairs + domain-fit reason): `microsoft/mpnet-base`, `answerdotai/ModernBERT-base`, `google-bert/bert-base-uncased`, `jhu-clsp/ettin-encoder-*` (17m / 32m / 68m / 150m / 400m / 1b, paired ModernBERT encoder family).
+- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (MLM-only, needs contrastive training), `microsoft/mdeberta-v3-base`, `jhu-clsp/mmBERT-base` / `-small`.
+- **CPU / small footprint** (`StaticEmbedding`): `StaticEmbedding(tokenizer, embedding_dim=...)`. **Model size = `vocab_size × dim × 4 bytes`**. Pick a small-vocab tokenizer or you get a giant model: 30k-vocab `bert-base-uncased` × 128 dim ≈ 15 MB, **250k-vocab `paraphrase-multilingual-MiniLM-L12-v2` × 256 dim ≈ 256 MB**. Random init needs 1M+ pairs. Warm-start (`StaticEmbedding.from_distillation(...)`) helps under ~100k pairs.
+
+Architecture variants (encoder / decoder / static / Router), pooling rules, and decoder-vs-encoder setup paths: `model_architectures.md`.
+
+**ModernBERT-family bases default to `max_seq_length=8192`.** That allocates activation memory for 8192-token sequences regardless of your data length and silently drives Windows VRAM into "shared memory" spillover. After loading any ModernBERT / mmBERT / Ettin / gte-modernbert / nomic-modernbert base, **explicitly set `model.max_seq_length = 256` (or 512 for documents)** unless you actually need long context.
+
+## [CE] Cross-Encoder
+
+Continue from an existing reranker beats fresh-start + 100k-500k pairs in most domains. Default to this unless you have a strong reason otherwise. Common namespaces as of 2026-Q2:
+
+- **English encoder rerankers**: `cross-encoder/ms-marco-*`, `BAAI/bge-reranker-*`, `mixedbread-ai/mxbai-rerank-*-v1` / `-v2`, `Alibaba-NLP/gte-reranker-modernbert-*`, `ibm-granite/granite-embedding-reranker-english-*`.
+- **Multilingual encoder rerankers**: `cross-encoder/mmarco-*`, `BAAI/bge-reranker-v2-m3`, `Alibaba-NLP/gte-multilingual-reranker-*`, `ibm-granite/granite-embedding-reranker-multilingual-*`.
+- **Decoder LLM rerankers** (multilingual, `num_labels=1` last-token-style scoring): `Qwen/Qwen3-Reranker-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Reranker-*` (multimodal).
+- **Fresh-start**: `microsoft/MiniLM-L12-H384-uncased`, `answerdotai/ModernBERT-base` / `-large`, `jhu-clsp/ettin-encoder-*`, `FacebookAI/xlm-roberta-base` (multilingual), `microsoft/mdeberta-v3-base` (multilingual), `jhu-clsp/mmBERT-base` / `-small` (multilingual). Pass `num_labels >= 2` for classification cross-encoders.
+
+Encoder-only bases are still the latency-efficient default (bidirectional attention is well-suited to the reranking use case at small parameter counts), but decoder LLM rerankers are now competitive at the top of MTEB Reranking when latency / memory budget allows.
+
+**Minimum dataset:** 500k+ labeled `(query, passage, label)` tuples for production. 10k-100k labeled pairs for continue-training on domain data. Low-resource languages may have less than 10k labeled pairs. In that case, lean on a multilingual base's pretraining and accept a noisier signal.
+
+**"Small" multilingual is ~100M+ params**, not 17M-50M like the English small models. mMiniLMv2-L12-H384 (~117M) is roughly the small-end for usable multilingual rerankers.
+
+## [SPARSE] Sparse Encoder (SPLADE)
+
+SPLADE requires a fill-mask / `AutoModelForMaskedLM`-compatible checkpoint. Encoder-only MLM models work out of the box. **Decoder LLMs do not**.
+
+- **Continue from existing SPLADE (English)**: `naver/splade-*` (the canonical family), `opensearch-project/opensearch-neural-sparse-encoding-*` (incl. `-doc-v2-distill`, `-doc-v3-distill` / `-doc-v3-gte`), `prithivida/Splade_PP_en_v*`, `ibm-granite/granite-embedding-30m-sparse`.
+- **Continue from existing SPLADE (multilingual)**: `opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1`.
+- **Fresh-start English** (≥500k pairs): any encoder with an MLM head (`distilbert/distilbert-base-uncased`, `google-bert/bert-base-uncased`). Pure `AutoModel` checkpoints without MLM won't work. Discover MLM bases: `hf models list --filter fill-mask --sort downloads --limit 20`.
+- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (has MLM head). For other multilingual MLM bases: add `--filter <language-code>`.
+
+**Minimum dataset:** 500k+ triplets (with mined hard negatives) for a competitive SPLADE. 50k+ triplets for domain adaptation on existing SPLADE.
+
+## Cross-cutting tips
+
+- **Non-English retrieval starting points** (when language tag returns 0 results): check `intfloat/multilingual_e5_train_data` for parallel pair data, MIRACL via the `sentence-transformers/miracl` mirror for multilingual retrieval, mMARCO via `unicamp-dl/mmarco` (14 languages, parquet-backed).
+- **Avoid script-based dataset loaders.** `datasets >= 4` rejects them with `RuntimeError: Dataset scripts are no longer supported`. Look for parquet-backed mirrors (e.g. `sentence-transformers/miracl` instead of `miracl/miracl`).
+- **`hf datasets sql` requires DuckDB** (`pip install duckdb`). Without it, fall back to `python -c "from datasets import load_dataset; ds = load_dataset('<id>', ...); print(ds.column_names, ds[0])"`.

package/skills/train-sentence-transformers/references/dataset_formats.md

@@ -0,0 +1,128 @@
+# Dataset Formats
+
+This reference covers: how datasets map to losses, how to reshape data when it doesn't fit, and how to mine hard negatives.
+
+## The two rules
+
+From the sentence-transformers training overview:
+
+1. If the loss requires a label, the dataset must have a column named **`label`, `labels`, `score`, or `scores`**. Any column with one of these names is the label.
+2. All other columns are **inputs**. The loss defines how many input columns it expects. Column **names don't matter. Order does**.
+
+Example: `CoSENTLoss` expects 2 inputs + a float label. A dataset with columns `["premise", "hypothesis", "score"]` works. A dataset with `["score", "premise", "hypothesis"]` does not. Reorder first.
+
+## Per-loss data shapes
+
+The per-type loss references (`losses_sentence_transformer.md`, `losses_cross_encoder.md`, `losses_sparse_encoder.md`) are the canonical mappings from data shape to loss. Cross-cutting recipe bits those tables don't show:
+
+- **`CosineSimilarityLoss`** wants `score` normalized to `[0, 1]`. `CoSENTLoss` / `AnglELoss` are pairwise-ranking and ignore absolute scale, so on `stsb` (raw 0-5) divide by 5 only when using cosine-similarity.
+- **`BatchAllTripletLoss` / `BatchHardTripletLoss` / `BatchSemiHardTripletLoss`** need `batch_sampler=BatchSamplers.GROUP_BY_LABEL` so multiple samples per label appear in the same batch.
+- **`MSELoss` (distillation)** label is the teacher's full embedding vector (a list of floats), not a scalar score.
+- **`MarginMSELoss` (distillation)** label is `teacher_score(q, pos) - teacher_score(q, neg)`, precomputed per row.
+- **N-tuple shape** for MNRL `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed) is produced by `mine_hard_negatives(..., output_format="n-tuple")`. "labeled-list" output_format produces the CrossEncoder listwise shape.
+
+## Reshaping operations
+
+If your data doesn't fit the loss's expected shape:
+
+### Reorder columns
+
+```python
+# Columns are ["hypothesis", "premise", "score"] but CoSENTLoss expects premise first.
+dataset = dataset.select_columns(["premise", "hypothesis", "score"])
+```
+
+### Rename label column
+
+```python
+# Your label is called "relevance" but ST wants "label".
+dataset = dataset.rename_column("relevance", "label")
+```
+
+### Drop extra columns
+
+```python
+# ST will treat every non-label column as an input. Drop metadata.
+dataset = dataset.remove_columns(["source_id", "created_at", "language"])
+```
+
+### Convert dtypes
+
+```python
+# Label is str, need float for CoSENTLoss.
+dataset = dataset.map(lambda x: {"label": float(x["label"])})
+```
+
+## Hard-negative mining
+
+`mine_hard_negatives` (in `sentence_transformers.util`) produces a training dataset with mined negatives using a retriever. Hard negatives are the single highest-leverage lever for retrieval-model quality.
+
+### Basic usage
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.util import mine_hard_negatives
+
+retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+
+mined = mine_hard_negatives(
+    dataset=train_pairs,                # has (anchor, positive) or (q, a) columns
+    model=retriever,
+    num_negatives=5,
+    range_min=0, range_max=100,         # rank window to sample hard negatives from
+    sampling_strategy="top",            # "top" = rank-1 hardest; "random" = random in window
+    output_format="n-tuple",            # "triplet" | "n-tuple" | "labeled-pair" | "labeled-list"
+    use_faiss=True,
+)
+```
+
+### Output formats
+
+- `"triplet"`: `(anchor, positive, negative)` triplets. One row per `(query, negative)` pair.
+- `"n-tuple"`: `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed). One row per query.
+- `"labeled-pair"`: `(anchor, text, label)` with `label=1` for positives and `label=0` for negatives. Good for `BinaryCrossEntropyLoss`.
+- `"labeled-list"`: `(anchor, texts, labels)`. One row per query with a list of candidates. Good for listwise losses.
+
+### Filtering false negatives
+
+If the retriever returns "negatives" that are actually relevant, they become false negatives and hurt training. Filter them:
+
+```python
+mined = mine_hard_negatives(
+    dataset=train_pairs,
+    model=retriever,
+    cross_encoder=CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2"),   # score candidates
+    num_negatives=5,
+    max_score=0.9,                      # drop candidates scoring above 0.9
+    relative_margin=0.05,               # require neg_score <= pos_score - abs(pos_score) * 0.05
+    absolute_margin=0.2,                # require neg_score <= pos_score - 0.2
+    output_format="n-tuple",
+    use_faiss=True,
+)
+```
+
+Use **either** `relative_margin` or `absolute_margin`, usually not both. `max_score` is independently useful as a hard ceiling.
+
+### CLI
+
+`scripts/mine_hard_negatives.py` is a CLI wrapper. See it for a ready-to-run command.
+
+## Choosing the right `range_min` / `range_max`
+
+`range_max=None` is the default. Pass an integer to cap how far down the ranked list to sample from.
+
+- `range_min=0`, `range_max=100`: sample from the top-100 retrieved. Good default.
+- `range_min=10`, `range_max=100`: skip the top-10 (often contains true positives). Safer if you lack a cross-encoder.
+- `range_min=0`, `range_max=1000`: wider net, more diverse negatives, slower.
+- `sampling_strategy="top"`: always pick the rank-1 hardest. Maximum training signal per row.
+- `sampling_strategy="random"`: pick randomly within the range. More robust if your retriever is itself noisy.
+
+## Quick Hub-side dataset checks
+
+`hf datasets sql "SELECT * FROM 'hf://datasets/<id>/<split>' LIMIT 5"` streams rows via DuckDB without `load_dataset(...)`. This is the fastest way to confirm column names match your loss before a full validation run. `hf datasets info <id>` shows config / splits / size. `hf datasets card <id> --text` renders the README.
+
+## Gotchas
+
+- **`remove_unused_columns=True` (default)**: the trainer drops columns that aren't passed to the model's forward. Usually fine, but if you rely on a custom collator that uses metadata columns, set `remove_unused_columns=False`.
+- **Floats stored as strings after CSV load**: `load_dataset("csv", ...)` keeps columns as strings by default. Cast with `.map(lambda x: {"label": float(x["label"])})`.
+- **Mined hard negatives with `include_positives=True`** include the positive as a negative in the output list, only useful when you're building an evaluator or want to measure rank of the positive. For training, leave it `False`.