better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-saelens.md

@@ -0,0 +1,439 @@
+---
+title: "Saelens — Train sparse autoencoders to interpret model features"
+sidebar_label: "Saelens"
+description: "Train sparse autoencoders to interpret model features"
+---
+
+{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
+
+# Saelens
+
+Train sparse autoencoders to interpret model features.
+
+## Skill metadata
+
+| | |
+|---|---|
+| Source | Optional — install with `hermes skills install official/mlops/saelens` |
+| Path | `optional-skills/mlops\saelens` |
+| Version | `1.0.1` |
+| Author | Orchestra Research |
+| License | MIT |
+| Dependencies | `sae-lens>=6.0.0`, `transformer-lens>=2.0.0`, `torch>=2.0.0` |
+| Platforms | linux, macos, windows |
+| Tags | `Sparse Autoencoders`, `SAE`, `Mechanistic Interpretability`, `Feature Discovery`, `Superposition` |
+
+## Reference: full SKILL.md
+
+:::info
+The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
+:::
+
+# SAELens: Sparse Autoencoders for Mechanistic Interpretability
+
+SAELens is the primary library for training and analyzing Sparse Autoencoders (SAEs) - a technique for decomposing polysemantic neural network activations into sparse, interpretable features. Based on Anthropic's groundbreaking research on monosemanticity.
+
+**GitHub**: [jbloomAus/SAELens](https://github.com/jbloomAus/SAELens) (1,100+ stars)
+
+## The Problem: Polysemanticity & Superposition
+
+Individual neurons in neural networks are **polysemantic** - they activate in multiple, semantically distinct contexts. This happens because models use **superposition** to represent more features than they have neurons, making interpretability difficult.
+
+**SAEs solve this** by decomposing dense activations into sparse, monosemantic features - typically only a small number of features activate for any given input, and each feature corresponds to an interpretable concept.
+
+## When to Use SAELens
+
+**Use SAELens when you need to:**
+- Discover interpretable features in model activations
+- Understand what concepts a model has learned
+- Study superposition and feature geometry
+- Perform feature-based steering or ablation
+- Analyze safety-relevant features (deception, bias, harmful content)
+
+**Consider alternatives when:**
+- You need basic activation analysis → Use **TransformerLens** directly
+- You want causal intervention experiments → Use **pyvene** or **TransformerLens**
+- You need production steering → Consider direct activation engineering
+
+## Installation
+
+```bash
+pip install sae-lens
+```
+
+Requirements: Python 3.10+, transformer-lens>=2.0.0
+
+## Core Concepts
+
+### What SAEs Learn
+
+SAEs are trained to reconstruct model activations through a sparse bottleneck:
+
+```
+Input Activation → Encoder → Sparse Features → Decoder → Reconstructed Activation
+    (d_model)       ↓        (d_sae >> d_model)    ↓         (d_model)
+                 sparsity                      reconstruction
+                 penalty                          loss
+```
+
+**Loss Function**: `MSE(original, reconstructed) + L1_coefficient × L1(features)`
+
+### Key Validation (Anthropic Research)
+
+In "Towards Monosemanticity", human evaluators found **70% of SAE features genuinely interpretable**. Features discovered include:
+- DNA sequences, legal language, HTTP requests
+- Hebrew text, nutrition statements, code syntax
+- Sentiment, named entities, grammatical structures
+
+## Workflow 1: Loading and Analyzing Pre-trained SAEs
+
+### Step-by-Step
+
+```python
+from transformer_lens import HookedTransformer
+from sae_lens import SAE
+
+# 1. Load model and pre-trained SAE
+model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
+# In sae-lens v6, SAE.from_pretrained() returns JUST the SAE (not a tuple).
+sae = SAE.from_pretrained(
+    release="gpt2-small-res-jb",
+    sae_id="blocks.8.hook_resid_pre",
+    device="cuda"
+)
+# If you also need the cfg dict and feature sparsity, use:
+# sae, cfg_dict, sparsity = SAE.from_pretrained_with_cfg_and_sparsity(...)
+
+# 2. Get model activations
+tokens = model.to_tokens("The capital of France is Paris")
+_, cache = model.run_with_cache(tokens)
+activations = cache["resid_pre", 8]  # [batch, pos, d_model]
+
+# 3. Encode to SAE features
+sae_features = sae.encode(activations)  # [batch, pos, d_sae]
+print(f"Active features: {(sae_features > 0).sum()}")
+
+# 4. Find top features for each position
+for pos in range(tokens.shape[1]):
+    top_features = sae_features[0, pos].topk(5)
+    token = model.to_str_tokens(tokens[0, pos:pos+1])[0]
+    print(f"Token '{token}': features {top_features.indices.tolist()}")
+
+# 5. Reconstruct activations
+reconstructed = sae.decode(sae_features)
+reconstruction_error = (activations - reconstructed).norm()
+```
+
+### Available Pre-trained SAEs
+
+| Release | Model | Layers |
+|---------|-------|--------|
+| `gpt2-small-res-jb` | GPT-2 Small | Multiple residual streams |
+| `gemma-2b-res` | Gemma 2B | Residual streams |
+| Various on HuggingFace | Search tag `saelens` | Various |
+
+### Checklist
+- [ ] Load model with TransformerLens
+- [ ] Load matching SAE for target layer
+- [ ] Encode activations to sparse features
+- [ ] Identify top-activating features per token
+- [ ] Validate reconstruction quality
+
+## Workflow 2: Training a Custom SAE
+
+### Step-by-Step
+
+```python
+from sae_lens import (
+    LanguageModelSAETrainingRunner,
+    LanguageModelSAERunnerConfig,
+    StandardTrainingSAEConfig,
+    LoggingConfig,
+)
+
+# 1. Configure training (v6 uses a NESTED config: SAE-specific options live in a
+#    `sae=` sub-config, and logging options live in a `logger=` sub-config).
+#    Note: `architecture`, `d_sae`, `l1_coefficient` etc. are now on the SAE sub-config,
+#    and legacy flat options like `hook_layer`, `activation_fn`, `log_to_wandb` were removed.
+cfg = LanguageModelSAERunnerConfig(
+    # SAE architecture + sparsity (nested)
+    sae=StandardTrainingSAEConfig(
+        d_in=768,          # Model dimension
+        d_sae=768 * 8,     # Expansion factor of 8
+        l1_coefficient=8e-5,  # Sparsity penalty
+        apply_b_dec_to_input=True,
+        normalize_activations="expected_average_only_in",
+    ),
+
+    # Data-generating function (model + hook point)
+    model_name="gpt2-small",
+    hook_name="blocks.8.hook_resid_pre",  # layer is inferred from hook_name (no hook_layer)
+
+    # Training
+    lr=4e-4,
+    l1_warm_up_steps=1000,
+    train_batch_size_tokens=4096,
+    training_tokens=100_000_000,
+
+    # Data
+    dataset_path="monology/pile-uncopyrighted",
+    context_size=128,
+
+    # Logging (nested)
+    logger=LoggingConfig(
+        log_to_wandb=True,
+        wandb_project="sae-training",
+    ),
+
+    # Checkpointing
+    checkpoint_path="checkpoints",
+    n_checkpoints=5,
+)
+
+# 2. Train
+trainer = LanguageModelSAETrainingRunner(cfg)  # SAETrainingRunner still works as an alias
+sae = trainer.run()
+
+# 3. Evaluate
+print(f"L0 (avg active features): {trainer.metrics['l0']}")
+print(f"CE Loss Recovered: {trainer.metrics['ce_loss_score']}")
+```
+
+> **v6 migration note:** For other SAE types swap the `sae=` sub-config —
+> `GatedTrainingSAEConfig`, `TopKTrainingSAEConfig` (set `k` directly), or
+> `JumpReLUTrainingSAEConfig` (uses `l0_coefficient`). Legacy flat options
+> (`architecture`, `expansion_factor`, `hook_layer`, `activation_fn`/`activation_fn_kwargs`,
+> `use_ghost_grads`, ghost grads, b_dec/decoder init options) were removed in v6.
+
+### Key Hyperparameters
+
+| Parameter | Typical Value | Effect |
+|-----------|---------------|--------|
+| `d_sae` | 4-16× d_model | More features, higher capacity |
+| `l1_coefficient` | 5e-5 to 1e-4 | Higher = sparser, less accurate |
+| `lr` | 1e-4 to 1e-3 | Standard optimizer LR |
+| `l1_warm_up_steps` | 500-2000 | Prevents early feature death |
+
+### Evaluation Metrics
+
+| Metric | Target | Meaning |
+|--------|--------|---------|
+| **L0** | 50-200 | Average active features per token |
+| **CE Loss Score** | 80-95% | Cross-entropy recovered vs original |
+| **Dead Features** | <5% | Features that never activate |
+| **Explained Variance** | >90% | Reconstruction quality |
+
+### Checklist
+- [ ] Choose target layer and hook point
+- [ ] Set expansion factor (d_sae = 4-16× d_model)
+- [ ] Tune L1 coefficient for desired sparsity
+- [ ] Enable L1 warm-up to prevent dead features
+- [ ] Monitor metrics during training (W&B)
+- [ ] Validate L0 and CE loss recovery
+- [ ] Check dead feature ratio
+
+## Workflow 3: Feature Analysis and Steering
+
+### Analyzing Individual Features
+
+```python
+from transformer_lens import HookedTransformer
+from sae_lens import SAE
+import torch
+
+model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
+sae = SAE.from_pretrained(  # v6 returns just the SAE
+    release="gpt2-small-res-jb",
+    sae_id="blocks.8.hook_resid_pre",
+    device="cuda"
+)
+
+# Find what activates a specific feature
+feature_idx = 1234
+test_texts = [
+    "The scientist conducted an experiment",
+    "I love chocolate cake",
+    "The code compiles successfully",
+    "Paris is beautiful in spring",
+]
+
+for text in test_texts:
+    tokens = model.to_tokens(text)
+    _, cache = model.run_with_cache(tokens)
+    features = sae.encode(cache["resid_pre", 8])
+    activation = features[0, :, feature_idx].max().item()
+    print(f"{activation:.3f}: {text}")
+```
+
+### Feature Steering
+
+```python
+def steer_with_feature(model, sae, prompt, feature_idx, strength=5.0):
+    """Add SAE feature direction to residual stream."""
+    tokens = model.to_tokens(prompt)
+
+    # Get feature direction from decoder
+    feature_direction = sae.W_dec[feature_idx]  # [d_model]
+
+    def steering_hook(activation, hook):
+        # Add scaled feature direction at all positions
+        activation += strength * feature_direction
+        return activation
+
+    # Generate with steering
+    output = model.generate(
+        tokens,
+        max_new_tokens=50,
+        fwd_hooks=[("blocks.8.hook_resid_pre", steering_hook)]
+    )
+    return model.to_string(output[0])
+```
+
+### Feature Attribution
+
+```python
+# Which features most affect a specific output?
+tokens = model.to_tokens("The capital of France is")
+_, cache = model.run_with_cache(tokens)
+
+# Get features at final position
+features = sae.encode(cache["resid_pre", 8])[0, -1]  # [d_sae]
+
+# Get logit attribution per feature
+# Feature contribution = feature_activation × decoder_weight × unembedding
+W_dec = sae.W_dec  # [d_sae, d_model]
+W_U = model.W_U    # [d_model, vocab]
+
+# Contribution to "Paris" logit
+paris_token = model.to_single_token(" Paris")
+feature_contributions = features * (W_dec @ W_U[:, paris_token])
+
+top_features = feature_contributions.topk(10)
+print("Top features for 'Paris' prediction:")
+for idx, val in zip(top_features.indices, top_features.values):
+    print(f"  Feature {idx.item()}: {val.item():.3f}")
+```
+
+## Common Issues & Solutions
+
+> All examples below use the v6 nested config: SAE-specific options go in the `sae=`
+> sub-config (`StandardTrainingSAEConfig` / `TopKTrainingSAEConfig` / etc.), training
+> knobs stay on the top-level `LanguageModelSAERunnerConfig`.
+
+### Issue: High dead feature ratio
+```python
+from sae_lens import LanguageModelSAERunnerConfig, StandardTrainingSAEConfig
+
+# WRONG: no warm-up, features die early
+cfg = LanguageModelSAERunnerConfig(
+    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=1e-4),
+    l1_warm_up_steps=0,  # Bad!
+)
+
+# RIGHT: warm up the L1 penalty (v6 removed ghost grads; warm-up is the lever now)
+cfg = LanguageModelSAERunnerConfig(
+    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=8e-5),
+    l1_warm_up_steps=1000,  # Gradually increase
+)
+```
+
+### Issue: Poor reconstruction (low CE recovery)
+```python
+# Reduce sparsity penalty and/or add capacity (both on the SAE sub-config)
+cfg = LanguageModelSAERunnerConfig(
+    sae=StandardTrainingSAEConfig(
+        d_in=768,
+        d_sae=768 * 16,       # More capacity
+        l1_coefficient=5e-5,  # Lower = better reconstruction
+    ),
+)
+```
+
+### Issue: Features not interpretable
+```python
+from sae_lens import LanguageModelSAERunnerConfig, StandardTrainingSAEConfig, TopKTrainingSAEConfig
+
+# Increase sparsity (higher L1)
+cfg = LanguageModelSAERunnerConfig(
+    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=1e-4),
+)
+# Or use a TopK SAE (k is set directly in v6, not via activation_fn_kwargs)
+cfg = LanguageModelSAERunnerConfig(
+    sae=TopKTrainingSAEConfig(d_in=768, d_sae=768*8, k=50),  # Exactly 50 active features
+)
+```
+
+### Issue: Memory errors during training
+```python
+cfg = LanguageModelSAERunnerConfig(
+    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=8e-5),
+    train_batch_size_tokens=2048,  # Reduce batch size
+    store_batch_size_prompts=4,    # Fewer prompts in buffer
+    n_batches_in_buffer=8,         # Smaller activation buffer
+)
+```
+
+## Integration with Neuronpedia
+
+Browse pre-trained SAE features at [neuronpedia.org](https://neuronpedia.org):
+
+```python
+# Features are indexed by SAE ID
+# Example: gpt2-small layer 8 feature 1234
+# → neuronpedia.org/gpt2-small/8-res-jb/1234
+```
+
+## Key Classes Reference
+
+| Class | Purpose |
+|-------|---------|
+| `SAE` | Sparse Autoencoder model |
+| `LanguageModelSAERunnerConfig` | Top-level training configuration (nests `sae=` and `logger=`) |
+| `StandardTrainingSAEConfig` / `TopKTrainingSAEConfig` / `GatedTrainingSAEConfig` / `JumpReLUTrainingSAEConfig` | SAE-type-specific sub-configs (v6) |
+| `LoggingConfig` | Logging/W&B sub-config (v6) |
+| `LanguageModelSAETrainingRunner` | Training loop manager (alias: `SAETrainingRunner`) |
+| `ActivationsStore` | Activation collection and batching |
+| `HookedSAETransformer` | TransformerLens + SAE integration |
+
+## Reference Documentation
+
+For detailed API documentation, tutorials, and advanced usage, see the `references/` folder:
+
+| File | Contents |
+|------|----------|
+| [references/README.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\saelens/references/README.md) | Overview and quick start guide |
+| [references/api.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\saelens/references/api.md) | Complete API reference for SAE, TrainingSAE, configurations |
+| [references/tutorials.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\saelens/references/tutorials.md) | Step-by-step tutorials for training, analysis, steering |
+
+## External Resources
+
+### Tutorials
+- [Basic Loading & Analysis](https://github.com/jbloomAus/SAELens/blob/main/tutorials/basic_loading_and_analysing.ipynb)
+- [Training a Sparse Autoencoder](https://github.com/jbloomAus/SAELens/blob/main/tutorials/training_a_sparse_autoencoder.ipynb)
+- [ARENA SAE Curriculum](https://www.lesswrong.com/posts/LnHowHgmrMbWtpkxx/intro-to-superposition-and-sparse-autoencoders-colab)
+
+### Papers
+- [Towards Monosemanticity](https://transformer-circuits.pub/2023/monosemantic-features) - Anthropic (2023)
+- [Scaling Monosemanticity](https://transformer-circuits.pub/2024/scaling-monosemanticity/) - Anthropic (2024)
+- [Sparse Autoencoders Find Highly Interpretable Features](https://arxiv.org/abs/2309.08600) - Cunningham et al. (ICLR 2024)
+
+### Official Documentation
+- [SAELens Docs](https://jbloomaus.github.io/SAELens/)
+- [Neuronpedia](https://neuronpedia.org) - Feature browser
+
+## SAE Architectures
+
+| Architecture | Description | Use Case |
+|--------------|-------------|----------|
+| **Standard** | ReLU + L1 penalty | General purpose |
+| **Gated** | Learned gating mechanism | Better sparsity control |
+| **TopK** | Exactly K active features | Consistent sparsity |
+
+```python
+from sae_lens import LanguageModelSAERunnerConfig, TopKTrainingSAEConfig
+
+# TopK SAE (exactly 50 features active) — `k` is set on the SAE sub-config in v6
+cfg = LanguageModelSAERunnerConfig(
+    sae=TopKTrainingSAEConfig(d_in=768, d_sae=768*8, k=50),
+)
+```

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-simpo.md

@@ -0,0 +1,237 @@
+---
+title: "Simpo — Reference-free preference alignment, simpler than DPO"
+sidebar_label: "Simpo"
+description: "Reference-free preference alignment, simpler than DPO"
+---
+
+{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
+
+# Simpo
+
+Reference-free preference alignment, simpler than DPO.
+
+## Skill metadata
+
+| | |
+|---|---|
+| Source | Optional — install with `hermes skills install official/mlops/simpo` |
+| Path | `optional-skills/mlops\simpo` |
+| Version | `1.0.0` |
+| Author | Orchestra Research |
+| License | MIT |
+| Dependencies | `torch`, `transformers`, `datasets`, `trl`, `accelerate` |
+| Platforms | linux, macos, windows |
+| Tags | `Post-Training`, `SimPO`, `Preference Optimization`, `Alignment`, `DPO Alternative`, `Reference-Free`, `LLM Alignment`, `Efficient Training` |
+
+## Reference: full SKILL.md
+
+:::info
+The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
+:::
+
+# SimPO - Simple Preference Optimization
+
+## Quick start
+
+SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.
+
+**Installation**:
+```bash
+# Create environment
+conda create -n simpo python=3.10 && conda activate simpo
+
+# Install PyTorch 2.2.2
+# Visit: https://pytorch.org/get-started/locally/
+
+# Install alignment-handbook
+git clone https://github.com/huggingface/alignment-handbook.git
+cd alignment-handbook
+python -m pip install .
+
+# Install Flash Attention 2
+python -m pip install flash-attn --no-build-isolation
+```
+
+**Training** (Mistral 7B):
+```bash
+ACCELERATE_LOG_LEVEL=info accelerate launch \
+  --config_file accelerate_configs/deepspeed_zero3.yaml \
+  scripts/run_simpo.py \
+  training_configs/mistral-7b-base-simpo.yaml
+```
+
+## Common workflows
+
+### Workflow 1: Train from base model (Mistral 7B)
+
+**Config** (`mistral-7b-base-simpo.yaml`):
+```yaml
+# Model
+model_name_or_path: mistralai/Mistral-7B-v0.1
+torch_dtype: bfloat16
+
+# Dataset
+dataset_mixer:
+  HuggingFaceH4/ultrafeedback_binarized: 1.0
+dataset_splits:
+  - train_prefs
+  - test_prefs
+
+# SimPO hyperparameters
+beta: 2.0                  # Reward scaling (2.0-10.0)
+gamma_beta_ratio: 0.5       # Target margin (0-1)
+loss_type: sigmoid          # sigmoid or hinge
+sft_weight: 0.0             # Optional SFT regularization
+
+# Training
+learning_rate: 5e-7         # Critical: 3e-7 to 1e-6
+num_train_epochs: 1
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 8
+
+# Output
+output_dir: ./outputs/mistral-7b-simpo
+```
+
+**Launch training**:
+```bash
+accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
+  scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml
+```
+
+### Workflow 2: Fine-tune instruct model (Llama 3 8B)
+
+**Config** (`llama3-8b-instruct-simpo.yaml`):
+```yaml
+model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
+
+dataset_mixer:
+  argilla/ultrafeedback-binarized-preferences-cleaned: 1.0
+
+beta: 2.5
+gamma_beta_ratio: 0.5
+learning_rate: 5e-7
+sft_weight: 0.1             # Add SFT loss to preserve capabilities
+
+num_train_epochs: 1
+per_device_train_batch_size: 2
+gradient_accumulation_steps: 4
+output_dir: ./outputs/llama3-8b-simpo
+```
+
+**Launch**:
+```bash
+accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
+  scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml
+```
+
+### Workflow 3: Reasoning-intensive tasks (lower LR)
+
+**For math/code tasks**:
+```yaml
+model_name_or_path: deepseek-ai/deepseek-math-7b-base
+
+dataset_mixer:
+  argilla/distilabel-math-preference-dpo: 1.0
+
+beta: 5.0                   # Higher for stronger signal
+gamma_beta_ratio: 0.7       # Larger margin
+learning_rate: 3e-7         # Lower LR for reasoning
+sft_weight: 0.0
+
+num_train_epochs: 1
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 16
+```
+
+## When to use vs alternatives
+
+**Use SimPO when**:
+- Want simpler training than DPO (no reference model)
+- Have preference data (chosen/rejected pairs)
+- Need better performance than DPO
+- Limited compute resources
+- Single-node training sufficient
+
+**Algorithm selection**:
+- **SimPO**: Simplest, best performance, no reference model
+- **DPO**: Need reference model baseline, more conservative
+- **PPO**: Maximum control, need reward model, complex setup
+- **GRPO**: Memory-efficient RL, no critic
+
+**Use alternatives instead**:
+- **OpenRLHF**: Multi-node distributed training, PPO/GRPO
+- **TRL**: Need multiple methods in one framework
+- **DPO**: Established baseline comparison
+
+## Common issues
+
+**Issue: Loss divergence**
+
+Reduce learning rate:
+```yaml
+learning_rate: 3e-7  # Reduce from 5e-7
+```
+
+Reduce beta:
+```yaml
+beta: 1.0  # Reduce from 2.0
+```
+
+**Issue: Model forgets capabilities**
+
+Add SFT regularization:
+```yaml
+sft_weight: 0.1  # Add SFT loss component
+```
+
+**Issue: Poor preference separation**
+
+Increase beta and margin:
+```yaml
+beta: 5.0            # Increase from 2.0
+gamma_beta_ratio: 0.8  # Increase from 0.5
+```
+
+**Issue: OOM during training**
+
+Reduce batch size:
+```yaml
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 16  # Maintain effective batch
+```
+
+Enable gradient checkpointing:
+```yaml
+gradient_checkpointing: true
+```
+
+## Advanced topics
+
+**Loss functions**: See [references/loss-functions.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\simpo/references/loss-functions.md) for sigmoid vs hinge loss, mathematical formulations, and when to use each.
+
+**Hyperparameter tuning**: See [references/hyperparameters.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\simpo/references/hyperparameters.md) for beta, gamma, learning rate selection guide, and model-size-specific recommendations.
+
+**Dataset preparation**: See [references/datasets.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\simpo/references/datasets.md) for preference data formats, quality filtering, and custom dataset creation.
+
+## Hardware requirements
+
+- **GPU**: NVIDIA A100/H100 recommended
+- **VRAM**:
+  - 7B model: 1× A100 40GB (DeepSpeed ZeRO-3)
+  - 8B model: 2× A100 40GB
+  - 70B model: 8× A100 80GB
+- **Single-node**: DeepSpeed ZeRO-3 sufficient
+- **Mixed precision**: BF16 recommended
+
+**Memory optimization**:
+- DeepSpeed ZeRO-3 (default config)
+- Gradient checkpointing
+- Flash Attention 2
+
+## Resources
+
+- Paper: https://arxiv.org/abs/2405.14734 (NeurIPS 2024)
+- GitHub: https://github.com/princeton-nlp/SimPO
+- Models: https://huggingface.co/princeton-nlp
+- Alignment Handbook: https://github.com/huggingface/alignment-handbook