opencode-skills-collection 3.1.11 → 3.1.13

package/bundled-skills/testng-skill/SKILL.md

@@ -0,0 +1,163 @@
+---
+name: testng-skill
+description: 'Generates TestNG tests in Java with groups, data providers, parallel execution, XML suite configuration, and listeners. Use when user mentions "TestNG", "@DataProvider", "testng.xml", "groups". Triggers on: "TestNG", "@DataProvider", "testng.xml", "TestNG suite", "parallel tests Java".'
+risk: unknown
+source: https://github.com/LambdaTest/agent-skills/tree/main/testng-skill
+source_repo: LambdaTest/agent-skills
+source_type: community
+date_added: 2026-07-01
+license: MIT
+license_source: https://github.com/LambdaTest/agent-skills/blob/main/LICENSE
+---
+
+# TestNG Testing Skill
+## When to Use
+
+Use this skill when you need generates TestNG tests in Java with groups, data providers, parallel execution, XML suite configuration, and listeners. Use when user mentions "TestNG", "@DataProvider", "testng.xml", "groups". Triggers on: "TestNG", "@DataProvider", "testng.xml", "TestNG suite", "parallel tests Java".
+
+
+## Core Patterns
+
+### Basic Test with Groups
+
+```java
+import org.testng.annotations.*;
+import org.testng.Assert;
+
+public class LoginTest {
+    @BeforeMethod
+    public void setUp() { /* setup */ }
+
+    @Test(groups = "smoke")
+    public void testLoginSuccess() {
+        Assert.assertTrue(loginService.login("user@test.com", "password123"));
+    }
+
+    @Test(groups = "regression", dependsOnMethods = "testLoginSuccess")
+    public void testAccessDashboard() {
+        Assert.assertNotNull(dashboard.getContent());
+    }
+
+    @Test(expectedExceptions = AuthenticationException.class)
+    public void testLoginInvalidPassword() {
+        loginService.login("user@test.com", "wrong");
+    }
+
+    @AfterMethod
+    public void tearDown() { /* cleanup */ }
+}
+```
+
+### Data Providers
+
+```java
+@DataProvider(name = "loginData")
+public Object[][] loginData() {
+    return new Object[][] {
+        {"admin@test.com", "admin123", true},
+        {"user@test.com", "password", true},
+        {"invalid@test.com", "wrong", false},
+    };
+}
+
+@Test(dataProvider = "loginData")
+public void testLogin(String email, String password, boolean expected) {
+    Assert.assertEquals(loginService.login(email, password), expected);
+}
+```
+
+### TestNG XML Suite
+
+```xml
+<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
+<suite name="Regression" parallel="tests" thread-count="5">
+  <test name="Smoke">
+    <groups><run><include name="smoke"/></run></groups>
+    <classes><class name="tests.LoginTest"/></classes>
+  </test>
+  <test name="Full">
+    <groups><run><include name="regression"/><exclude name="flaky"/></run></groups>
+    <packages><package name="tests.*"/></packages>
+  </test>
+</suite>
+```
+
+### Parallel Execution
+
+```xml
+<suite parallel="methods" thread-count="5">   <!-- Method level -->
+<suite parallel="classes" thread-count="5">    <!-- Class level -->
+<suite parallel="tests" thread-count="5">      <!-- Test level -->
+```
+
+### Soft Assertions
+
+```java
+SoftAssert soft = new SoftAssert();
+soft.assertEquals(user.getName(), "Alice");
+soft.assertEquals(user.getAge(), 25);
+soft.assertTrue(user.isActive());
+soft.assertAll();  // Reports all failures at once
+```
+
+### Listeners
+
+```java
+public class TestListener implements ITestListener {
+    @Override public void onTestFailure(ITestResult result) {
+        System.out.println("Failed: " + result.getName());
+        // Take screenshot, log, etc.
+    }
+}
+
+@Listeners(TestListener.class)
+public class LoginTest { /* ... */ }
+```
+
+### Lifecycle Annotations
+
+```
+@BeforeSuite → @BeforeTest → @BeforeClass → @BeforeMethod → @Test → @AfterMethod → @AfterClass → @AfterTest → @AfterSuite
+```
+
+### Anti-Patterns
+
+| Bad | Good | Why |
+|-----|------|-----|
+| `dependsOnMethods` everywhere | Independent tests | Cascading failures |
+| No groups | `@Test(groups = "smoke")` | Can't run subsets |
+| Hard-coded test data | `@DataProvider` | Reusable |
+| Priority ordering | Independent tests | Fragile |
+
+## Quick Reference
+
+| Task | Command |
+|------|---------|
+| Run suite | `mvn test -DsuiteXmlFile=testng.xml` |
+| Run group | `mvn test -Dgroups=smoke` |
+| Run class | `mvn test -Dtest=LoginTest` |
+| Reports | `test-output/index.html` |
+
+## Deep Patterns → `reference/playbook.md`
+
+| § | Section | Lines |
+|---|---------|-------|
+| 1 | Project Setup & Configuration | Maven + Surefire config |
+| 2 | Suite XML Configuration | Multi-env, parallel, groups |
+| 3 | BaseTest & Thread-Safe Driver | ThreadLocal, ConfigReader |
+| 4 | Data Providers (Advanced) | Excel, JSON, CSV, parallel, cross-class |
+| 5 | Factory Pattern | Cross-browser matrix |
+| 6 | Listeners (Production Suite) | Retry, screenshot, timing |
+| 7 | Soft Assertions & Dependencies | Groups, method deps |
+| 8 | Page Object Integration | PageFactory, fluent POs |
+| 9 | Parallel Execution Strategies | Method/class/test/mixed |
+| 10 | Reporting Integration | Allure, ExtentReports |
+| 11 | CI/CD Integration | GitHub Actions, Jenkins |
+| 12 | Debugging Quick-Reference | 12 common problems |
+| 13 | Best Practices Checklist | 14 items |
+
+## Limitations
+
+- Use this skill only when the task clearly matches its upstream source and local project context.
+- Verify commands, generated code, dependencies, credentials, and external service behavior before applying changes.
+- Do not treat examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.

package/bundled-skills/train-sentence-transformers/SKILL.md

@@ -0,0 +1,118 @@
+---
+name: train-sentence-transformers
+description: Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval / pair...
+risk: unknown
+source: https://github.com/huggingface/skills/tree/main/skills/train-sentence-transformers
+source_repo: huggingface/skills
+source_type: official
+date_added: 2026-07-01
+license: Apache-2.0
+license_source: https://github.com/huggingface/skills/blob/main/LICENSE
+---
+
+# Train a sentence-transformers Model
+## When to Use
+
+Use this skill when you need train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval / pair...
+
+
+**This SKILL.md is a router, not a manual.** It tells you which references and example scripts to load for your task. The actual content — recommended losses, evaluators, training-script structure, model selection, training-arg knobs, troubleshooting — lives in `references/` and `scripts/`.
+
+**Do not synthesize a training script from this file alone.** Open the per-type production template (`scripts/train_<type>_example.py`) and copy it as your starting point. The templates contain load-bearing scaffolding (autocast helper, model-card class, logger silencing list, `force=True`, `seed`, TF32, version-compatible imports, named-evaluator metric handling) that prior agent runs have repeatedly missed when rolling their own from a synthesized snippet.
+
+## 1. Identify the model type
+
+| Tag | Class | What it does | When to pick |
+|---|---|---|---|
+| **[SentenceTransformer]** | `SentenceTransformer` (bi-encoder) | Maps each input to a fixed-dim dense vector | Retrieval, similarity, clustering, classification, paraphrase mining, dedup |
+| **[CrossEncoder]** | `CrossEncoder` (reranker) | Scores `(query, passage)` pairs jointly | Two-stage retrieval (rerank top-100 from bi-encoder), pair classification |
+| **[SparseEncoder]** | `SparseEncoder` (SPLADE) | Sparse vectors over the vocabulary | Learned-sparse retrieval, inverted-index backends (Elasticsearch / OpenSearch / Lucene) |
+
+Tiebreakers when the request is ambiguous: "embedding model" / "vector search" / "similarity" → **[SentenceTransformer]**. "rerank" / "ranker" / "two-stage" → **[CrossEncoder]**. "SPLADE" / "sparse" / "inverted index" → **[SparseEncoder]**. If still unclear, ask.
+
+## 2. Required reading
+
+**Read these in full before writing any code. Do not triage by perceived relevance.**
+
+### Per-type — always required
+
+**[SentenceTransformer]**
+- `references/losses_sentence_transformer.md` — loss-to-data-shape mapping; `BatchSamplers.NO_DUPLICATES` requirement for MNRL-family; `Cached*` ↔ `gradient_checkpointing` incompatibility.
+- `references/evaluators_sentence_transformer.md` — evaluator-to-task mapping; `metric_for_best_model` key construction (named vs unnamed); per-evaluator `primary_metric` values.
+- `references/model_architectures.md` — encoder vs decoder vs static vs Router pipelines; pooling rules (mean / cls / lasttoken); auto-mean-pooling behavior for fresh-start MLM bases.
+- `scripts/train_sentence_transformer_example.py` — production template; copy this as your starting point.
+
+**[CrossEncoder]**
+- `references/losses_cross_encoder.md` — pointwise / pairwise / listwise / distillation; `pos_weight` derivation; `activation_fn=Identity()` mandatory for non-BCE losses (silent eval-rank collapse otherwise).
+- `references/evaluators_cross_encoder.md` — `CrossEncoderRerankingEvaluator` recipe; named-evaluator key format `eval_{name}_{primary_metric}`.
+- `scripts/train_cross_encoder_example.py` — production template; copy this as your starting point.
+
+**[SparseEncoder]**
+- `references/losses_sparse_encoder.md` — `SpladeLoss` wrapper requirement; FLOPS regularizer weights; smoke-test active-dim ramp behavior.
+- `references/evaluators_sparse_encoder.md` — `SparseNanoBEIREvaluator` (English-only) and the in-domain alternative; `eval_{name}_{primary_metric}` key format.
+- `scripts/train_sparse_encoder_example.py` — production template; copy this as your starting point.
+
+### Cross-cutting — always required (regardless of task)
+
+- `references/training_args.md` — `TrainingArguments` knobs, precision rules (load fp32 + autocast bf16/fp16; never `torch_dtype=bfloat16`), `warmup_steps` (float) vs deprecated `warmup_ratio`, `save_steps` must be a multiple of `eval_steps` for `load_best_model_at_end`, schedulers, HPO, tracker, resume, hub-push variants.
+- `references/dataset_formats.md` — column-matching rules (label name auto-detection; column-order-not-name); reshaping recipes; hard-negative mining options.
+- `references/base_model_selection.md` — discovery commands; per-type model namespaces; ModernBERT-family `max_seq_length=8192` trap; `datasets >= 4` script-loader rejection; non-English starting-point shortcuts.
+- `references/troubleshooting.md` — symptom-indexed failure recipes. Skim the section headings on every run, even a healthy one; the "Metrics don't improve" and "Hub push fails" entries cover bugs that bite frequently and are cheaper to recognize before they fire than to debug after.
+
+### Cross-cutting — load when applicable
+
+- `references/hardware_guide.md` — VRAM sizing, multi-GPU, FSDP / DeepSpeed, HF Jobs flavors. Required for >24GB models, multi-GPU, or HF Jobs runs.
+- `references/hf_jobs_execution.md` — required when running on HF Jobs.
+- `references/prompts_and_instructions.md` — required when using prompt-tuned bases (E5, BGE, GTE, Qwen3-Embedding, Instructor, Nomic, etc.) or adding `query: ` / `passage: ` style prefixes.
+
+### Variant scripts (open when the task matches)
+- **[SentenceTransformer]** `scripts/train_sentence_transformer_<matryoshka|multi_dataset|with_lora|distillation|make_multilingual|static_embedding>_example.py`.
+- **[CrossEncoder]** `scripts/train_cross_encoder_<distillation|listwise>_example.py`.
+- **[SparseEncoder]** `scripts/train_sparse_encoder_distillation_example.py`.
+- Hard-negative mining CLI — `scripts/mine_hard_negatives.py`.
+
+## 3. Defaults
+
+Override only if the user specifies otherwise:
+- **Local execution.** Pitch HF Jobs only if local hardware can't fit the job.
+- **Single run.** After it completes, propose experimentation if the user would benefit (weak/marginal verdict, "see how high you can push it" framing, etc.). Iteration rules in `references/training_args.md` (Experimentation section).
+- **Public Hub push at end-of-run, wrapped in try-except.** On HF Jobs (ephemeral env) ALSO enable in-trainer push (`push_to_hub=True` + `hub_strategy="every_save"`); details in `references/hf_jobs_execution.md`.
+
+## 4. Constraints the produced script must satisfy
+
+These are non-negotiable contracts. Implementation lives in the production templates and references — do not reinvent.
+
+- Capture the pre-training evaluator score as `baseline_eval` **before** `trainer.train()`.
+- Emit a single end-of-run line: `VERDICT: WIN|MARGINAL|REGRESSION | score=... | baseline=... | delta=...`. A monitor scrapes for this.
+- Silence `httpx`, `httpcore`, `huggingface_hub`, `urllib3`, `filelock`, `fsspec` to WARNING (otherwise HF download URLs flood the agent's context).
+- Tee logs to `logs/{RUN_NAME}.log`.
+- End with `model.push_to_hub(...)` wrapped in `try/except`.
+- Smoke-test before any long run (`max_steps=1` + tiny dataset slice). The production templates show one common pattern (`SMOKE_TEST` env var).
+- **[CrossEncoder]** Include `EarlyStoppingCallback(patience>=3)` — CE rerankers often peak mid-training and regress.
+- **[SparseEncoder]** Log `query_active_dims` / `corpus_active_dims` on the verdict line; high nDCG with collapsed sparsity is not a win. The keys come back name-prefixed (e.g. `..._query_active_dims`); use suffix matching to pluck them — see the SPARSE production template for the exact pattern.
+
+## 5. Workflow
+
+1. Identify the model type (§1). Ask if ambiguous.
+2. Load the §2 required-reading files for that type.
+3. Open `scripts/train_<type>_example.py` and copy it as your starting point.
+4. Replace `MODEL_NAME`, `DATASET_NAME`, `RUN_NAME`, the loss, and the evaluator with the user's task. Cross-check loss/data-shape match against `references/losses_<type>.md`; cross-check the `metric_for_best_model` key against `references/evaluators_<type>.md` (named evaluators format the key as `eval_{name}_{primary_metric}`).
+5. Smoke-test (`max_steps=1`).
+6. Run.
+7. After the run, append to `logs/experiments.md` and propose iteration if the verdict is weak/marginal.
+
+## Prerequisites
+
+```bash
+pip install "sentence-transformers[train]>=5.0"        # add [train,image] / [audio] / [video] for [SentenceTransformer] multimodal
+pip install trackio                                    # optional tracker; or wandb / tensorboard / mlflow
+hf auth login                                          # or set HF_TOKEN with write scope (for Hub push)
+```
+
+GPU strongly recommended. CPU works only for demos and `[SentenceTransformer]` `StaticEmbedding`.
+
+## Limitations
+
+- Use this skill only when the task clearly matches its upstream product or API scope.
+- Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
+- Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.

package/bundled-skills/train-sentence-transformers/references/base_model_selection.md

@@ -0,0 +1,79 @@
+# Base Model Selection
+
+Leaderboards rotate every few months; don't trust any hardcoded "best" pick. Discover current options live — run **both** sort orders since most-downloaded surfaces proven options and trending surfaces recent SOTA that may not have download volume yet.
+
+## Discovery commands
+
+**[BI]**:
+```bash
+hf models list --filter sentence-transformers --sort downloads --limit 20
+hf models list --filter sentence-transformers --sort trending  --limit 20
+```
+
+**[CE]**:
+```bash
+hf models list --filter sentence-transformers --filter text-ranking --sort downloads --limit 20
+hf models list --filter sentence-transformers --filter text-ranking --sort trending  --limit 20
+```
+
+**[SPARSE]**:
+```bash
+hf models list --filter sentence-transformers --filter sparse-encoder --sort downloads --limit 20
+hf models list --filter sentence-transformers --filter sparse-encoder --sort trending  --limit 20
+```
+
+Optional language narrowing (any type): add `--filter <language-code>`. Not all multilingual models tag each language, so missing matches doesn't mean the model can't handle that language — re-run without the filter to compare.
+
+```bash
+hf models card <id> --text                        # confirm dimensions, max_seq_length, license, languages
+```
+
+Cross-check the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard) (pick the relevant tab) before committing to a multi-hour run.
+
+## [BI] Bi-Encoder
+
+Continue from an existing retriever beats fresh-start + 100k–500k pairs. Common namespaces as of 2026-Q2 (verify against discovery commands — the field rotates):
+
+- **English encoder retrievers**: `sentence-transformers/all-*` (MiniLM-L6-v2, mpnet-base-v2 still the most-downloaded models on the Hub), `BAAI/bge-*-en-v1.5`, `nomic-ai/nomic-embed-text-v1.5`, `mixedbread-ai/mxbai-embed-large-v1`, `Alibaba-NLP/gte-*`, `Snowflake/snowflake-arctic-embed-*`, `jinaai/jina-embeddings-v5-text-small` / `-nano`, `microsoft/harrier-oss-v1-270m` / `-0.6b`.
+- **Multilingual encoder retrievers**: `sentence-transformers/paraphrase-multilingual-*`, `intfloat/multilingual-e5-*`, `ibm-granite/granite-embedding-*-multilingual-r2`, `google/embeddinggemma-300m`, `voyageai/voyage-4-nano`.
+- **Long documents (8k+)**: `nomic-ai/modernbert-embed-*`, `answerdotai/ModernBERT-large`.
+- **Decoder LLM retrievers** (multilingual; **need last-token pooling**): `Qwen/Qwen3-Embedding-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Embedding-*` (multimodal), `codefuse-ai/F2LLM-v2-*`.
+- **Fresh-start English** (≥500k pairs + domain-fit reason): `microsoft/mpnet-base`, `answerdotai/ModernBERT-base`, `google-bert/bert-base-uncased`, `jhu-clsp/ettin-encoder-*` (17m / 32m / 68m / 150m / 400m / 1b — paired ModernBERT encoder family).
+- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (MLM-only, needs contrastive training), `microsoft/mdeberta-v3-base`, `jhu-clsp/mmBERT-base` / `-small`.
+- **CPU / small footprint** (`StaticEmbedding`): `StaticEmbedding(tokenizer, embedding_dim=...)`. **Model size = `vocab_size × dim × 4 bytes`** — pick a small-vocab tokenizer or you get a giant model: 30k-vocab `bert-base-uncased` × 128 dim ≈ 15 MB; **250k-vocab `paraphrase-multilingual-MiniLM-L12-v2` × 256 dim ≈ 256 MB**. Random init needs 1M+ pairs; warm-start (`StaticEmbedding.from_distillation(...)`) helps under ~100k pairs.
+
+Architecture variants (encoder / decoder / static / Router), pooling rules, and decoder-vs-encoder setup paths: `model_architectures.md`.
+
+**ModernBERT-family bases default to `max_seq_length=8192`.** That allocates activation memory for 8192-token sequences regardless of your data length and silently drives Windows VRAM into "shared memory" spillover. After loading any ModernBERT / mmBERT / Ettin / gte-modernbert / nomic-modernbert base, **explicitly set `model.max_seq_length = 256` (or 512 for documents)** unless you actually need long context.
+
+## [CE] Cross-Encoder
+
+Continue from an existing reranker beats fresh-start + 100k–500k pairs in most domains; default to this unless you have a strong reason otherwise. Common namespaces as of 2026-Q2:
+
+- **English encoder rerankers**: `cross-encoder/ms-marco-*`, `BAAI/bge-reranker-*`, `mixedbread-ai/mxbai-rerank-*-v1` / `-v2`, `Alibaba-NLP/gte-reranker-modernbert-*`, `ibm-granite/granite-embedding-reranker-english-*`.
+- **Multilingual encoder rerankers**: `cross-encoder/mmarco-*`, `BAAI/bge-reranker-v2-m3`, `Alibaba-NLP/gte-multilingual-reranker-*`, `ibm-granite/granite-embedding-reranker-multilingual-*`.
+- **Decoder LLM rerankers** (multilingual; `num_labels=1` last-token-style scoring): `Qwen/Qwen3-Reranker-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Reranker-*` (multimodal).
+- **Fresh-start**: `microsoft/MiniLM-L12-H384-uncased`, `answerdotai/ModernBERT-base` / `-large`, `jhu-clsp/ettin-encoder-*`, `FacebookAI/xlm-roberta-base` (multilingual), `microsoft/mdeberta-v3-base` (multilingual), `jhu-clsp/mmBERT-base` / `-small` (multilingual). Pass `num_labels >= 2` for classification cross-encoders.
+
+Encoder-only bases are still the latency-efficient default (bidirectional attention is well-suited to the reranking use case at small parameter counts), but decoder LLM rerankers are now competitive at the top of MTEB Reranking when latency / memory budget allows.
+
+**Minimum dataset:** 500k+ labeled `(query, passage, label)` tuples for production; 10k–100k labeled pairs for continue-training on domain data. Low-resource languages may have less than 10k labeled pairs; in that case lean on a multilingual base's pretraining and accept a noisier signal.
+
+**"Small" multilingual is ~100M+ params**, not 17M-50M like the English small models. mMiniLMv2-L12-H384 (~117M) is roughly the small-end for usable multilingual rerankers.
+
+## [SPARSE] Sparse Encoder (SPLADE)
+
+SPLADE requires a fill-mask / `AutoModelForMaskedLM`-compatible checkpoint. Encoder-only MLM models work out of the box; **decoder LLMs do not**.
+
+- **Continue from existing SPLADE — English**: `naver/splade-*` (the canonical family), `opensearch-project/opensearch-neural-sparse-encoding-*` (incl. `-doc-v2-distill`, `-doc-v3-distill` / `-doc-v3-gte`), `prithivida/Splade_PP_en_v*`, `ibm-granite/granite-embedding-30m-sparse`.
+- **Continue from existing SPLADE — multilingual**: `opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1`.
+- **Fresh-start English** (≥500k pairs): any encoder with an MLM head — `distilbert/distilbert-base-uncased`, `google-bert/bert-base-uncased`. Pure `AutoModel` checkpoints without MLM won't work. Discover MLM bases: `hf models list --filter fill-mask --sort downloads --limit 20`.
+- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (has MLM head). For other multilingual MLM bases: add `--filter <language-code>`.
+
+**Minimum dataset:** 500k+ triplets (with mined hard negatives) for a competitive SPLADE; 50k+ triplets for domain adaptation on existing SPLADE.
+
+## Cross-cutting tips
+
+- **Non-English retrieval starting points** (when language tag returns 0 results): check `intfloat/multilingual_e5_train_data` for parallel pair data; MIRACL via the `sentence-transformers/miracl` mirror for multilingual retrieval; mMARCO via `unicamp-dl/mmarco` (14 languages, parquet-backed).
+- **Avoid script-based dataset loaders.** `datasets >= 4` rejects them with `RuntimeError: Dataset scripts are no longer supported`. Look for parquet-backed mirrors (e.g. `sentence-transformers/miracl` instead of `miracl/miracl`).
+- **`hf datasets sql` requires DuckDB** (`pip install duckdb`). Without it, fall back to `python -c "from datasets import load_dataset; ds = load_dataset('<id>', ...); print(ds.column_names, ds[0])"`.

package/bundled-skills/train-sentence-transformers/references/dataset_formats.md

@@ -0,0 +1,128 @@
+# Dataset Formats
+
+This reference covers: how datasets map to losses, how to reshape data when it doesn't fit, and how to mine hard negatives.
+
+## The two rules
+
+From the sentence-transformers training overview:
+
+1. If the loss requires a label, the dataset must have a column named **`label`, `labels`, `score`, or `scores`**. Any column with one of these names is the label.
+2. All other columns are **inputs**. The loss defines how many input columns it expects. Column **names don't matter; order does**.
+
+Example: `CoSENTLoss` expects 2 inputs + a float label. A dataset with columns `["premise", "hypothesis", "score"]` works. A dataset with `["score", "premise", "hypothesis"]` does not — reorder first.
+
+## Per-loss data shapes
+
+The per-type loss references (`losses_sentence_transformer.md`, `losses_cross_encoder.md`, `losses_sparse_encoder.md`) are the canonical mappings from data shape to loss. Cross-cutting recipe bits those tables don't show:
+
+- **`CosineSimilarityLoss`** wants `score` normalized to `[0, 1]`; `CoSENTLoss` / `AnglELoss` are pairwise-ranking and ignore absolute scale, so on `stsb` (raw 0-5) divide by 5 only when using cosine-similarity.
+- **`BatchAllTripletLoss` / `BatchHardTripletLoss` / `BatchSemiHardTripletLoss`** need `batch_sampler=BatchSamplers.GROUP_BY_LABEL` so multiple samples per label appear in the same batch.
+- **`MSELoss` (distillation)** label is the teacher's full embedding vector (a list of floats), not a scalar score.
+- **`MarginMSELoss` (distillation)** label is `teacher_score(q, pos) - teacher_score(q, neg)`, precomputed per row.
+- **N-tuple shape** for MNRL `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed) is produced by `mine_hard_negatives(..., output_format="n-tuple")`; "labeled-list" output_format produces the CrossEncoder listwise shape.
+
+## Reshaping operations
+
+If your data doesn't fit the loss's expected shape:
+
+### Reorder columns
+
+```python
+# Columns are ["hypothesis", "premise", "score"] but CoSENTLoss expects premise first.
+dataset = dataset.select_columns(["premise", "hypothesis", "score"])
+```
+
+### Rename label column
+
+```python
+# Your label is called "relevance" but ST wants "label".
+dataset = dataset.rename_column("relevance", "label")
+```
+
+### Drop extra columns
+
+```python
+# ST will treat every non-label column as an input. Drop metadata.
+dataset = dataset.remove_columns(["source_id", "created_at", "language"])
+```
+
+### Convert dtypes
+
+```python
+# Label is str, need float for CoSENTLoss.
+dataset = dataset.map(lambda x: {"label": float(x["label"])})
+```
+
+## Hard-negative mining
+
+`mine_hard_negatives` (in `sentence_transformers.util`) produces a training dataset with mined negatives using a retriever. Hard negatives are the single highest-leverage lever for retrieval-model quality.
+
+### Basic usage
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.util import mine_hard_negatives
+
+retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+
+mined = mine_hard_negatives(
+    dataset=train_pairs,                # has (anchor, positive) or (q, a) columns
+    model=retriever,
+    num_negatives=5,
+    range_min=0, range_max=100,         # rank window to sample hard negatives from
+    sampling_strategy="top",            # "top" = rank-1 hardest; "random" = random in window
+    output_format="n-tuple",            # "triplet" | "n-tuple" | "labeled-pair" | "labeled-list"
+    use_faiss=True,
+)
+```
+
+### Output formats
+
+- `"triplet"` — `(anchor, positive, negative)` triplets. One row per `(query, negative)` pair.
+- `"n-tuple"` — `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed) — one row per query.
+- `"labeled-pair"` — `(anchor, text, label)` with `label=1` for positives and `label=0` for negatives. Good for `BinaryCrossEntropyLoss`.
+- `"labeled-list"` — `(anchor, texts, labels)` — one row per query with a list of candidates. Good for listwise losses.
+
+### Filtering false negatives
+
+If the retriever returns "negatives" that are actually relevant, they become false negatives and hurt training. Filter them:
+
+```python
+mined = mine_hard_negatives(
+    dataset=train_pairs,
+    model=retriever,
+    cross_encoder=CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2"),   # score candidates
+    num_negatives=5,
+    max_score=0.9,                      # drop candidates scoring above 0.9
+    relative_margin=0.05,               # require neg_score < pos_score * (1 - 0.05)
+    absolute_margin=0.2,                # require neg_score < pos_score - 0.2
+    output_format="n-tuple",
+    use_faiss=True,
+)
+```
+
+Use **either** `relative_margin` or `absolute_margin`, usually not both. `max_score` is independently useful as a hard ceiling.
+
+### CLI
+
+`scripts/mine_hard_negatives.py` is a CLI wrapper — see it for a ready-to-run command.
+
+## Choosing the right `range_min` / `range_max`
+
+`range_max=None` is the default; pass an integer to cap how far down the ranked list to sample from.
+
+- `range_min=0`, `range_max=100` — sample from the top-100 retrieved. Good default.
+- `range_min=10`, `range_max=100` — skip the top-10 (often contains true positives). Safer if you lack a cross-encoder.
+- `range_min=0`, `range_max=1000` — wider net, more diverse negatives, slower.
+- `sampling_strategy="top"` — always pick the rank-1 hardest. Maximum training signal per row.
+- `sampling_strategy="random"` — pick randomly within the range. More robust if your retriever is itself noisy.
+
+## Quick Hub-side dataset checks
+
+`hf datasets sql "SELECT * FROM 'hf://datasets/<id>/<split>' LIMIT 5"` streams rows via DuckDB without `load_dataset(...)` — the fastest way to confirm column names match your loss before a full validation run. `hf datasets info <id>` shows config / splits / size; `hf datasets card <id> --text` renders the README.
+
+## Gotchas
+
+- **`remove_unused_columns=True` (default)**: the trainer drops columns that aren't passed to the model's forward. Usually fine, but if you rely on a custom collator that uses metadata columns, set `remove_unused_columns=False`.
+- **Floats stored as strings after CSV load**: `load_dataset("csv", ...)` keeps columns as strings by default. Cast with `.map(lambda x: {"label": float(x["label"])})`.
+- **Mined hard negatives with `include_positives=True`** include the positive as a negative in the output list — only useful when you're building an evaluator or want to measure rank of the positive. For training, leave it `False`.

package/bundled-skills/train-sentence-transformers/references/evaluators_cross_encoder.md

@@ -0,0 +1,116 @@
+# Evaluators (Cross-Encoder)
+
+All cross-encoder evaluators live in `sentence_transformers.cross_encoder.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Rerank retrieval results (nDCG@k on BM25 top-N) — fast default | `CrossEncoderNanoBEIREvaluator` |
+| Rerank with custom candidates per query | `CrossEncoderRerankingEvaluator` |
+| Binary / multi-class pair classification | `CrossEncoderClassificationEvaluator` |
+| Continuous pair scoring (STS-style) | `CrossEncoderCorrelationEvaluator` |
+
+Wrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`) to track them together:
+
+```python
+from sentence_transformers.base.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([nano_beir_eval, custom_rerank_eval])
+```
+
+## The default: `CrossEncoderNanoBEIREvaluator`
+
+Analog of `NanoBEIREvaluator` for rerankers. Takes BM25 top-100 for each NanoBEIR query and measures how well the cross-encoder re-ranks them.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderNanoBEIREvaluator
+
+evaluator = CrossEncoderNanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: 11 of 13 NanoBEIR datasets (excludes "arguana", "touche2020")
+    batch_size=64,
+    rerank_k=100,                                   # rerank the BM25 top-K
+)
+```
+
+Output key for `metric_for_best_model`: **`eval_NanoBEIR_R100_mean_ndcg@10`**. The `R100` signals "rerank top-100"; if you change `rerank_k`, the prefix changes (e.g. `R50`).
+
+Each individual dataset contributes `eval_Nano{DatasetName}_R100_ndcg@10` (e.g. `eval_NanoMSMARCO_R100_ndcg@10`) too.
+
+## Custom reranking with your own candidates
+
+Use when you have query + positive + distractor candidates that aren't part of NanoBEIR:
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderRerankingEvaluator
+
+samples = [
+    {"query": "...", "positive": ["the gold answer"], "documents": ["...", "...", ...]}
+    for ...
+]
+
+evaluator = CrossEncoderRerankingEvaluator(
+    samples=samples,
+    batch_size=64,
+    name="my-rerank",
+    always_rerank_positives=False,   # default is True; override to False for realistic eval
+)
+```
+
+- `always_rerank_positives=True` (the library default) forces the positive into the candidate pool even when the retriever missed it. The reranker is graded only on candidates it can actually score, so the metric reflects pure reranker quality.
+- `always_rerank_positives=False`: the positive is only reranked if it's already in `documents`. If the retriever missed it, the rank counts as N+1. This reflects end-to-end retriever+reranker quality. A positive the retriever missed is lost, regardless of reranker skill.
+
+Output key: `eval_{name}_ndcg@10`, `eval_{name}_map`, `eval_{name}_mrr@10`.
+
+## Classification-style cross-encoders
+
+### `CrossEncoderClassificationEvaluator`
+
+Works for both binary (`num_labels=1`) and multi-class (`num_labels>=2`) cross-encoders. Branches internally:
+- `num_labels=1`: binary mode. Sweeps thresholds to report accuracy, F1, precision, recall, and **average_precision** (primary).
+- `num_labels>=2`: multi-class mode (e.g. NLI: entailment / neutral / contradiction). Reports **f1_macro** (primary), f1_micro, f1_weighted, and per-class precision / recall.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderClassificationEvaluator
+
+evaluator = CrossEncoderClassificationEvaluator(
+    sentence_pairs=[(premise, hypothesis), ...],
+    labels=[0, 1, 2, ...],
+    batch_size=64,
+    name="nli-dev",
+)
+```
+
+Output keys (binary, `num_labels=1`): `eval_{name}_accuracy`, `eval_{name}_f1`, `eval_{name}_average_precision` (primary).
+Output keys (multi-class, `num_labels>=2`): `eval_{name}_f1_macro` (primary), `eval_{name}_f1_micro`, `eval_{name}_f1_weighted`.
+
+### `CrossEncoderCorrelationEvaluator`
+
+For continuous-score cross-encoders (like an STS cross-encoder outputting a similarity score). Reports Pearson/Spearman vs. gold scores.
+
+```python
+from sentence_transformers.cross_encoder.evaluation import CrossEncoderCorrelationEvaluator
+
+evaluator = CrossEncoderCorrelationEvaluator(
+    sentence_pairs=[(a, b), ...],
+    scores=[0.4, 0.8, ...],
+    name="stsb-dev",
+)
+```
+
+Output keys: `eval_{name}_spearman`, `eval_{name}_pearson`.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_R100_mean_ndcg@10` — `CrossEncoderNanoBEIREvaluator` default
+- `eval_{name}_ndcg@10` — `CrossEncoderRerankingEvaluator`
+- `eval_{name}_average_precision` — `CrossEncoderClassificationEvaluator` (binary, `num_labels=1`)
+- `eval_{name}_f1_macro` — `CrossEncoderClassificationEvaluator` (multi-class, `num_labels>=2`)
+- `eval_{name}_spearman` — `CrossEncoderCorrelationEvaluator`
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training** — pre-training baseline. Tiny post-training delta means the loss/data/base is wrong.
+- `CrossEncoderClassificationEvaluator` accepts both `num_labels=1` (binary, primary `average_precision`) and `num_labels>=2` (multi-class, primary `f1_macro`); `CrossEncoderCorrelationEvaluator` requires `num_labels=1`.
+- The default `dataset_names=None` excludes `arguana` and `touche2020` (Argument-Retrieval task differs from the rest); pass `dataset_names=list(DATASET_NAME_TO_HUMAN_READABLE)` from `sentence_transformers.cross_encoder.evaluation.nano_beir` to actually run all 13.
+- Subset NanoBEIR datasets during training (3–4) to keep eval cheap; run the broader set post-training.