opencode-skills-collection 3.1.12 → 3.1.13

package/bundled-skills/train-sentence-transformers/references/evaluators_sentence_transformer.md

@@ -0,0 +1,151 @@
+# Evaluators (Bi-Encoder)
+
+All bi-encoder evaluators live in `sentence_transformers.sentence_transformer.evaluation`.
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Retrieval (nDCG, MRR, Recall) — fast default | `NanoBEIREvaluator` |
+| Retrieval on your own corpus / qrels | `InformationRetrievalEvaluator` |
+| STS / continuous similarity | `EmbeddingSimilarityEvaluator` |
+| Binary classification | `BinaryClassificationEvaluator` |
+| Triplet accuracy | `TripletEvaluator` |
+| Reranking (from retrieval candidates) | `RerankingEvaluator` |
+| MSE vs. teacher (distillation) | `MSEEvaluator`, `MSEEvaluatorFromDataFrame` |
+| Paraphrase mining | `ParaphraseMiningEvaluator` |
+| Translation (cross-lingual alignment) | `TranslationEvaluator` |
+| Label accuracy (classification during training) | `LabelAccuracyEvaluator` |
+
+Wrap multiple evaluators in `SequentialEvaluator` to track all of them together:
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([evaluator1, evaluator2, evaluator3])
+```
+
+## The big three
+
+### `NanoBEIREvaluator` (retrieval)
+
+Small, fast subset of BEIR. Typically runs in <1 minute on a mid-range GPU. Default choice for retrieval training.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import NanoBEIREvaluator
+
+evaluator = NanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
+    batch_size=128,
+    show_progress_bar=False,
+)
+```
+
+- Default dataset list covers 13 tasks; pick a subset for speed during training.
+- Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_cosine_ndcg@10`** (bi-encoder default = cosine similarity).
+
+### `EmbeddingSimilarityEvaluator` (STS-style)
+
+Computes Pearson/Spearman correlation between model cosine similarities and gold labels.
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import EmbeddingSimilarityEvaluator
+from sentence_transformers.util.similarity import SimilarityFunction
+
+evaluator = EmbeddingSimilarityEvaluator(
+    sentences1=stsb["sentence1"],
+    sentences2=stsb["sentence2"],
+    scores=stsb["score"],
+    main_similarity=SimilarityFunction.COSINE,
+    name="sts-dev",
+)
+```
+
+- `main_similarity` can be `COSINE`, `DOT_PRODUCT`, `EUCLIDEAN`, `MANHATTAN`.
+- `name` is used in the output key: `eval_sts-dev_spearman_cosine`, `eval_sts-dev_pearson_cosine`, etc.
+
+### `InformationRetrievalEvaluator` (full retrieval)
+
+Use when you have your **own** corpus + queries + qrels (not one of the NanoBEIR tasks).
+
+```python
+from sentence_transformers.sentence_transformer.evaluation import InformationRetrievalEvaluator
+
+evaluator = InformationRetrievalEvaluator(
+    queries={qid: query_text for qid, query_text in ...},
+    corpus={doc_id: doc_text for doc_id, doc_text in ...},
+    relevant_docs={qid: {doc_id, ...} for qid in ...},   # qid -> set of relevant doc_ids
+    name="my-retrieval",
+    mrr_at_k=[10],
+    ndcg_at_k=[10],
+    accuracy_at_k=[1, 5, 10],
+    precision_recall_at_k=[1, 5, 10],
+    map_at_k=[100],
+    show_progress_bar=False,
+    batch_size=64,
+)
+```
+
+Output keys: `eval_{name}_cosine_ndcg@10`, `eval_{name}_cosine_mrr@10`, etc.
+
+Heavy for large corpora — each eval encodes the full corpus. Don't run it every 100 steps. Use `NanoBEIREvaluator` for frequent evaluation during training and reserve full IR for milestones / post-training.
+
+## Other bi-encoder evaluators
+
+### `BinaryClassificationEvaluator`
+
+For labeled pair classification (e.g. duplicate detection, entailment as binary). Reports accuracy, F1, precision/recall, AP. Supports all distance metrics — finds the best threshold per metric.
+
+### `TripletEvaluator`
+
+For `(anchor, positive, negative)` triplets. Reports the fraction of triplets where the positive is closer to the anchor than the negative.
+
+### `RerankingEvaluator`
+
+For custom re-ranking datasets: you provide candidates per query, the evaluator computes MAP and MRR. Good for measuring retrieval-quality on a held-out set.
+
+### `MSEEvaluator` / `MSEEvaluatorFromDataFrame`
+
+For distillation setups. Compares student embeddings against teacher embeddings (or teacher scores), reports MSE.
+
+### `ParaphraseMiningEvaluator`
+
+For paraphrase-mining tasks. Given a corpus of labeled paraphrase pairs, computes mining quality (F1 at various thresholds).
+
+### `TranslationEvaluator`
+
+For cross-lingual / `make_multilingual`-style alignment checking. Measures whether the student aligns sentences across languages.
+
+### `LabelAccuracyEvaluator`
+
+For a `SoftmaxLoss`-trained classifier head. Reports accuracy on held-out data.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_mean_cosine_ndcg@10` — `NanoBEIREvaluator`
+- `eval_sts-dev_spearman_cosine` — `EmbeddingSimilarityEvaluator(name="sts-dev")`
+- `eval_{name}_cosine_ndcg@10` — `InformationRetrievalEvaluator(name=...)`
+
+## Multi-dimensional evaluation (Matryoshka)
+
+For Matryoshka-trained models, evaluate at each target dimension:
+
+```python
+per_dim_evaluators = [
+    EmbeddingSimilarityEvaluator(
+        sentences1=..., sentences2=..., scores=...,
+        main_similarity=SimilarityFunction.COSINE,
+        name=f"sts-dev-{dim}",
+        truncate_dim=dim,
+    ) for dim in [768, 512, 256, 128, 64]
+]
+evaluator = SequentialEvaluator(per_dim_evaluators, main_score_function=lambda scores: scores[0])
+```
+
+The first evaluator's score drives `load_best_model_at_end`.
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training** — pre-training baseline. If the post-training delta is tiny, the loss/data/base is wrong.
+- Don't run `InformationRetrievalEvaluator` with a large corpus (>100k docs) at frequent `eval_steps` — use `NanoBEIREvaluator` during training, reserve full IR for end-of-training.
+- `greater_is_better=True` is the default; right for nDCG / MRR / accuracy.

package/bundled-skills/train-sentence-transformers/references/evaluators_sparse_encoder.md

@@ -0,0 +1,121 @@
+# Evaluators (Sparse Encoder)
+
+All sparse-encoder evaluators live in `sentence_transformers.sparse_encoder.evaluation`. They mirror the bi-encoder versions with a `Sparse` prefix and default to **dot product** similarity (cosine on sparse vectors is less meaningful).
+
+## Choosing the right evaluator
+
+| Task | Evaluator |
+|---|---|
+| Retrieval (nDCG, MRR, Recall) — fast default | `SparseNanoBEIREvaluator` |
+| Retrieval on your own corpus / qrels | `SparseInformationRetrievalEvaluator` |
+| STS / continuous similarity | `SparseEmbeddingSimilarityEvaluator` |
+| Binary classification | `SparseBinaryClassificationEvaluator` |
+| Triplet accuracy | `SparseTripletEvaluator` |
+| Reranking (from retrieval candidates) | `SparseRerankingEvaluator` |
+| MSE vs. teacher (distillation) | `SparseMSEEvaluator` |
+| Translation (cross-lingual alignment) | `SparseTranslationEvaluator` |
+| Hybrid BM25 + sparse retrieval | `ReciprocalRankFusionEvaluator` |
+
+Wrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`):
+
+```python
+from sentence_transformers.base.evaluation import SequentialEvaluator
+evaluator = SequentialEvaluator([sparse_nano_beir, my_custom_ir])
+```
+
+## The default: `SparseNanoBEIREvaluator`
+
+Small, fast subset of BEIR adapted for sparse retrieval. Typical runtime <1 minute on a mid-range GPU.
+
+```python
+from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator
+
+evaluator = SparseNanoBEIREvaluator(
+    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
+    batch_size=32,
+    show_progress_bar=False,
+)
+```
+
+Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_dot_ndcg@10`** (sparse defaults to dot product).
+
+### Sparsity tracking
+
+Unlike the dense variant, the sparse evaluator also reports **active dimension counts** so you can monitor sparsity during training:
+
+- `query_active_dims` — non-zero entries per query vector
+- `document_active_dims` — non-zero entries per document vector
+
+A healthy SPLADE checkpoint typically shows ~30–50 active dims for queries and ~150–250 for documents. If these drift toward the vocab size (~30k), the FLOPS regularization isn't doing its job — raise `query_regularizer_weight` / `document_regularizer_weight` in `SpladeLoss`.
+
+## Retrieval on your own corpus
+
+### `SparseInformationRetrievalEvaluator`
+
+Same shape as the dense version but operates on sparse vectors internally:
+
+```python
+from sentence_transformers.sparse_encoder.evaluation import SparseInformationRetrievalEvaluator
+
+evaluator = SparseInformationRetrievalEvaluator(
+    queries={qid: text for qid, text in ...},
+    corpus={doc_id: text for doc_id, text in ...},
+    relevant_docs={qid: {doc_id, ...} for qid in ...},
+    name="my-sparse-ir",
+    ndcg_at_k=[10],
+    mrr_at_k=[10],
+    accuracy_at_k=[1, 5, 10],
+    map_at_k=[100],
+    batch_size=32,
+)
+```
+
+Output keys: `eval_{name}_dot_ndcg@10`, `eval_{name}_dot_mrr@10`, etc. Also reports active-dims.
+
+Heavy for large corpora. Use `SparseNanoBEIREvaluator` during training; reserve full IR for post-training.
+
+## Hybrid retrieval
+
+### `ReciprocalRankFusionEvaluator`
+
+Measures the performance of combining your sparse encoder with BM25 (or any other retriever) via reciprocal-rank fusion. Useful when shipping a hybrid system is the actual deployment target.
+
+## Other sparse evaluators
+
+### `SparseEmbeddingSimilarityEvaluator`
+
+STS-style. Computes Pearson/Spearman between sparse vector similarities and gold labels. Uses dot product by default.
+
+### `SparseBinaryClassificationEvaluator`
+
+For labeled pair classification with sparse embeddings.
+
+### `SparseTripletEvaluator`
+
+For `(anchor, positive, negative)` triplets — reports fraction where the positive is closer than the negative (by dot product).
+
+### `SparseRerankingEvaluator`
+
+For custom re-ranking with sparse embeddings. Same semantics as the dense `RerankingEvaluator`.
+
+### `SparseMSEEvaluator`
+
+For distillation setups. Compares sparse student embeddings against teacher outputs.
+
+### `SparseTranslationEvaluator`
+
+For cross-lingual / `make_multilingual`-style alignment checking with sparse embeddings.
+
+## Writing `metric_for_best_model`
+
+Pattern: `f"eval_{evaluator.primary_metric}"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:
+- `eval_NanoBEIR_mean_dot_ndcg@10` — `SparseNanoBEIREvaluator` default
+- `eval_{name}_dot_ndcg@10` — `SparseInformationRetrievalEvaluator`
+- `eval_{name}_spearman_dot` — `SparseEmbeddingSimilarityEvaluator`
+
+## Gotchas
+
+- **Always run `evaluator(model)` once before training** — confirms the pipeline works (a fill-mask base scores ~0 on retrieval until trained).
+- Sparse evaluators default to dot product; cosine on sparse vectors isn't meaningful.
+- Don't compare dense and sparse metrics directly — different scales (cosine ∈ [-1, 1] vs. dot ∈ [0, ∞)).
+- Always check `query_active_dims` / `document_active_dims` — thousands of active dims per doc means the FLOPS regularizer is mistuned, even if nDCG looks OK.

package/bundled-skills/train-sentence-transformers/references/hardware_guide.md

@@ -0,0 +1,105 @@
+# Hardware Guide
+
+Training embedding models is memory-bound more often than compute-bound.
+
+## If you hit OOM
+
+Try in this order:
+
+1. **Reduce `per_device_train_batch_size`**. Raise `gradient_accumulation_steps` to keep the effective batch size for regression losses. (For MNRL, effective batch via grad-accum is **not** equivalent — see point 3.)
+2. **Enable `gradient_checkpointing=True`**. ~30% slower, ~40% less activation memory. Incompatible with `Cached*` losses.
+3. **Switch to a `Cached*` loss**:
+   - `CachedMultipleNegativesRankingLoss(model, mini_batch_size=32)` — forwards in mini-batches, accumulates the contrastive loss over the full batch. Can simulate batch sizes of 1024+ on a 24GB GPU.
+   - `CachedSpladeLoss(model, loss=..., mini_batch_size=16)` — same trick for sparse.
+   - `CachedGISTEmbedLoss(model, guide_model, mini_batch_size=32)` — GIST variant.
+4. **Enable PEFT / LoRA** for decoder models >1B. `LoraConfig(r=64, lora_alpha=128, task_type="FEATURE_EXTRACTION")`. See `../scripts/train_sentence_transformer_with_lora_example.py` (docstring covers when to use, hyperparams, QLoRA, sharing).
+5. **Move to multi-GPU**. See below.
+6. **Shorten sequences**. If truncating to 128 is already sufficient for your task, set `max_seq_length` on the transformer module.
+
+## Multi-GPU
+
+`sentence-transformers` uses `accelerate` under the hood. Distributed training works without code changes.
+
+### Data parallel (DDP)
+
+Launch:
+
+```bash
+accelerate launch train.py
+
+# or explicitly:
+accelerate launch --multi_gpu --num_processes=4 train.py
+```
+
+`per_device_train_batch_size` stays per-GPU. Effective batch size scales linearly. MNRL's in-batch negatives remain **per-device**, not global, unless you pass `gather_across_devices=True` to losses that support it (`MultipleNegativesRankingLoss`, `CachedMultipleNegativesRankingLoss`, the symmetric variants, `GISTEmbedLoss`, `CachedGISTEmbedLoss`, `SparseMultipleNegativesRankingLoss`).
+
+### FSDP / DeepSpeed
+
+For models >3B, use `accelerate config` to enable FSDP or DeepSpeed ZeRO. Both are supported — `sentence-transformers` doesn't require any code changes, only the launch config.
+
+```bash
+accelerate config                   # interactive; choose FSDP or DeepSpeed
+accelerate launch train.py
+```
+
+With FSDP full-shard: a 7B model trains on 4×24GB GPUs that would OOM on any single one of them.
+
+**FSDP caveats** (from the [distributed training docs](https://sbert.net/docs/sentence_transformer/training/distributed.html)):
+- **Evaluators don't run under FSDP** as of writing — the eval hooks call `model.encode()` which FSDP-wrapped modules can't service mid-training. Plan to evaluate post-training on a single-GPU load of the final checkpoint instead, or train with DDP if you need mid-training evaluation.
+- **Layer wrapping must be specified**, e.g. `fsdp_config={"transformer_layer_cls_to_wrap": "BertLayer"}` (substitute the right layer class for your model: `BertLayer`, `LlamaDecoderLayer`, `Qwen2DecoderLayer`, etc.). Without this FSDP sharding can silently misbehave.
+- **Slower than DDP** for models that fit on a single GPU — only reach for FSDP when you actually need the memory savings.
+
+DeepSpeed ZeRO-2/3 is an alternative with its own config; works identically at the `accelerate config` level.
+
+## Effective batch size for contrastive losses
+
+For `MultipleNegativesRankingLoss` and its variants, **batch size is a quality knob**, not just a speed knob. Bigger batches = more in-batch negatives = richer gradients.
+
+The effective pool of in-batch negatives per anchor:
+
+| Setup | In-batch negatives per anchor |
+|---|---|
+| Single GPU, batch 64 | 63 |
+| 4× DDP, per-device batch 64 | 63 local only by default; 255 with `MultipleNegativesRankingLoss(model, gather_across_devices=True)` |
+| Single GPU, CachedMNRL, mini_batch 32, batch 256 | 255 |
+| 4× DDP, CachedMNRL, per-device 256 | 255 local; 1023 with `gather_across_devices=True` |
+
+For large corpora (retrieval), push toward 512+ effective negatives. For small, clean datasets (STS), 64 is plenty.
+
+## Precision choice by GPU
+
+| GPU generation | Recommended |
+|---|---|
+| T4, V100, GTX 1xxx, RTX 2xxx | `fp16=True` |
+| RTX 3xxx, A10G, A100, L4 | `bf16=True` |
+| RTX 4xxx, H100, B200 | `bf16=True` (or fp8 on H100 via specific kernels — not default) |
+| Apple M-series / ROCm | MPS/ROCm support is variable; `fp16` or `fp32` most reliable |
+
+bf16 is more numerically stable and almost always preferred when available.
+
+## Hugging Face Jobs flavor guide
+
+Hugging Face Jobs requires a Pro/Team/Enterprise plan. Pricing is approximate and subject to change — see the [Jobs pricing page](https://huggingface.co/docs/huggingface_hub/guides/jobs).
+
+| Flavor | Memory | Typical use | Est. $/hr |
+|---|---|---|---|
+| `cpu-basic` | ~2 GB | Dataset prep, validation, hard-neg mining (small) | <$0.10 |
+| `cpu-upgrade` | ~4 GB | Same, slightly bigger | $0.10 |
+| `t4-small` | 16 GB | Demos, MiniLM/DistilBERT with small batches | ~$0.75 |
+| `t4-medium` | 16 GB | MiniLM / DistilBERT with larger batch | ~$1.50 |
+| `l4x1` | 24 GB | BERT-base, MPNet, ModernBERT-base | ~$2.50 |
+| `a10g-small` | 24 GB | BERT-base to BERT-large | ~$3.50 |
+| `a10g-large` | 48 GB | ModernBERT-large, Qwen3-0.6B | ~$5.00 |
+| `a10g-largex2` | 96 GB (2× 48GB) | Mid-size multi-GPU | ~$10 |
+| `a100-large` | 80 GB | Large models or big contrastive batches | ~$10–12 |
+| `h100` | 80 GB | Biggest single-GPU | ~$12 |
+| `h100x8` | 640 GB | LLM-scale distributed | ~$96 |
+
+Defaults by base model:
+- MiniLM / DistilBERT -> `t4-small`
+- BERT-base / MPNet / ModernBERT-base -> `a10g-small` or `l4x1`
+- BERT-large / ModernBERT-large -> `a10g-large`
+- Qwen3-0.6B decoder base -> `a10g-large`
+- 1B+ decoder bases with LoRA -> `a10g-large` or `a100-large`
+
+Always start one flavor **smaller** than you think you need: OOM on Jobs is cheap ($0.50–$5 for a failed run). Underprovisioned is better than overprovisioned for the first attempt. When budgeting `timeout`, add **20–30% buffer** for model loading, checkpoint saving, and Hub push.

package/bundled-skills/train-sentence-transformers/references/hf_jobs_execution.md

@@ -0,0 +1,173 @@
+# Hugging Face Jobs Execution
+
+Run training on Hugging Face's managed GPUs without provisioning any local infrastructure. The same training script runs locally and on Jobs — this reference covers only the Jobs-specific concerns.
+
+## Prerequisites
+
+- Hugging Face account with a **Pro, Team, or Enterprise** plan. Jobs are paid.
+- `HF_TOKEN` with **write** permission. Log in once locally with `hf auth login` (the modern command from the `hf` CLI; the older `huggingface-cli login` still works but is deprecated).
+- Access to the `hf_jobs()` MCP tool, or the `hf` CLI. If installing the CLI from the hosted script, download it to a temporary file, inspect it, and then run it locally.
+
+## The three submission paths
+
+### 1. Inline script via MCP (recommended in Claude Code)
+
+Pass the full training script as `script`. Dependencies come from the PEP 723 header.
+
+```python
+hf_jobs("uv", {
+    "script": """
+# /// script
+# requires-python = ">=3.10"
+# dependencies = ["sentence-transformers[train]>=5.0", "trackio"]
+# ///
+
+# <full training script content>
+""",
+    "flavor": "a10g-large",
+    "timeout": "3h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
+})
+```
+
+### 2. Script-from-URL via MCP
+
+Upload the script to the Hub (as a model or dataset repo file) or a Gist, then reference by URL:
+
+```python
+hf_jobs("uv", {
+    "script": "https://huggingface.co/USERNAME/scripts/resolve/main/train_bi_encoder.py",
+    "flavor": "a10g-large",
+    "timeout": "3h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
+})
+```
+
+Local file paths (`./train.py`, `/path/to/train.py`) **do not work** — Jobs run in isolated containers without access to your filesystem.
+
+### 3. CLI
+
+```bash
+hf jobs uv run \
+    --flavor a10g-large \
+    --timeout 3h \
+    --secrets HF_TOKEN \
+    "https://huggingface.co/USERNAME/scripts/resolve/main/train.py"
+```
+
+Syntax gotchas:
+- Command order is `hf jobs uv run`, **not** `hf jobs run uv`.
+- Flags (`--flavor`, `--timeout`, `--secrets`) go **before** the script URL.
+- `--secrets` (plural), not `--secret`.
+
+## Required script modifications for Jobs
+
+Add these to your `TrainingArguments`:
+
+```python
+args = SentenceTransformerTrainingArguments(
+    ...,
+    push_to_hub=True,
+    hub_model_id="your-username/my-model",
+    hub_strategy="every_save",        # push each checkpoint; timeout-safe
+    save_strategy="steps",
+    save_steps=0.1,                   # 10 saves/pushes per epoch; scales with dataset size
+)
+```
+
+Why each matters:
+
+| Argument | Why |
+|---|---|
+| `push_to_hub=True` | The Jobs container is destroyed after the job finishes. Without Hub push, all weights are lost. |
+| `hub_model_id` | Required to identify the destination repo. |
+| `hub_strategy="every_save"` | Default, but worth being deliberate about on Jobs: each checkpoint is pushed as it's written, so a timeout leaves all completed checkpoints on the Hub. `"end"` only pushes once `trainer.train()` returns, so a timeout loses everything. |
+| `save_strategy="steps"` + `save_steps=0.1` | Checkpoints must actually be saved for `hub_strategy="every_save"` to push them. Fractional `0.1` = save every 10% of training, auto-scales with dataset size. |
+
+## Secrets
+
+Secrets are environment variables injected into the Jobs container. They never appear in logs and are not part of the script.
+
+| Secret | Required when |
+|---|---|
+| `HF_TOKEN` | Always, for Hub push. Also covers Trackio auth. |
+| `WANDB_API_KEY` | Using `report_to="wandb"`. |
+| `MLFLOW_TRACKING_URI`, `MLFLOW_TRACKING_TOKEN` | Using MLflow with a remote server. |
+
+The `$HF_TOKEN` syntax in the job config references the value from your local environment at submission time — the literal string `$HF_TOKEN` is replaced with your token's value. Never hardcode tokens in the script itself.
+
+Trackio (the default tracker in this skill) uses `HF_TOKEN` for auth, so no extra secrets are needed. Only switch to the W&B / MLflow rows above if you're using those trackers.
+
+## Timeout
+
+Default is **30 minutes**, which is too short for almost any real training. Set explicitly:
+
+```python
+"timeout": "2h"       # 2 hours
+"timeout": "90m"      # 90 minutes
+"timeout": "1.5h"     # 90 minutes
+"timeout": 7200       # seconds, as integer
+```
+
+Rule: **estimated training time × 1.3**. The extra buffer covers model loading, dataset caching, checkpoint saving, and Hub push.
+
+On timeout, the container is killed immediately. Only data on the Hub (`hub_strategy="every_save"` saves you here) or in persistent volumes survives.
+
+## Dataset caching
+
+Hugging Face datasets are cached at `~/.cache/huggingface/datasets` by default — **inside the container**, which is destroyed after the job. Each Jobs run re-downloads the dataset.
+
+For large datasets (>5 GB), this matters. Options:
+
+- **Persistent `/data` volume** (Jobs feature, check current documentation): set `HF_DATASETS_CACHE=/data/datasets` so caches persist across jobs.
+- **Pre-cache locally, push to Hub**: if the dataset is on Hub already, nothing to do. If it's local-only, `dataset.push_to_hub(...)` once so subsequent jobs load from Hub.
+
+## Monitoring a running job
+
+```bash
+hf jobs ps [--all]                        # running (or all) jobs
+hf jobs inspect <job-id>                  # full config + status
+hf jobs logs <job-id> [--follow|--tail N] # tail or stream
+hf jobs cancel <job-id>
+hf jobs hardware                          # list flavors + hourly rates
+```
+
+`hf jobs logs <id> --follow` under `Bash run_in_background` pairs nicely with a `Monitor` watching for the `VERDICT:` line emitted by your training script's verdict block.
+
+MCP equivalents (signatures may vary by server version — check the actual
+tool listing): `hf_jobs("ps")`, `hf_jobs("logs", {"job_id": ...})`,
+`hf_jobs("cancel", {"job_id": ...})`.
+
+For recurring runs, `hf jobs scheduled uv run "<cron>" <script> ...`
+schedules; `hf jobs scheduled ps/suspend/delete` manages.
+
+## Common failures
+
+### "Model not found on Hub" after a successful-looking run
+
+The run succeeded but `push_to_hub` was not enabled. The container is gone; the weights are gone.
+
+Fix: always set `push_to_hub=True` + `hub_model_id=...` + `secrets={"HF_TOKEN": "$HF_TOKEN"}`.
+
+### Tracker not connecting
+
+- **Trackio:** `HF_TOKEN` missing or lacks write permission. Add `"secrets": {"HF_TOKEN": "$HF_TOKEN"}` and make sure the token has write access.
+- **W&B:** `WANDB_API_KEY` missing. Add `"secrets": {"HF_TOKEN": "$HF_TOKEN", "WANDB_API_KEY": "$WANDB_API_KEY"}`.
+
+### OOM on first step
+
+Flavor too small. Move up one tier (see `hardware_guide.md`).
+
+### Training starts but eval hangs forever
+
+`eval_strategy="steps"` with no `eval_dataset`. Always provide an eval dataset, or set `eval_strategy="no"`.
+
+### Dataset download times out
+
+Large dataset or slow cold-cache. Increase `timeout` or pre-cache to a persistent volume.
+
+### `CachedMultipleNegativesRankingLoss` + `gradient_checkpointing=True` crash
+
+The cached losses are incompatible with gradient checkpointing. Disable `gradient_checkpointing`.
+
+After submission, the MCP returns a job ID. Monitor with `hf_jobs("logs", {"job_id": ...})` when you want an update — don't poll in a tight loop. End-to-end submission templates live in `scripts/train_sentence_transformer_example.py` / `scripts/train_cross_encoder_example.py` / `scripts/train_sparse_encoder_example.py`; wrap the script contents in the inline pattern from §1 above.