better-dsh 0.2.5-c → 0.2.5-d

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-saelens.md

@@ -1,439 +0,0 @@
----
-title: "Saelens — Train sparse autoencoders to interpret model features"
-sidebar_label: "Saelens"
-description: "Train sparse autoencoders to interpret model features"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Saelens
-
-Train sparse autoencoders to interpret model features.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/saelens` |
-| Path | `optional-skills/mlops\saelens` |
-| Version | `1.0.1` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `sae-lens>=6.0.0`, `transformer-lens>=2.0.0`, `torch>=2.0.0` |
-| Platforms | linux, macos, windows |
-| Tags | `Sparse Autoencoders`, `SAE`, `Mechanistic Interpretability`, `Feature Discovery`, `Superposition` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# SAELens: Sparse Autoencoders for Mechanistic Interpretability
-
-SAELens is the primary library for training and analyzing Sparse Autoencoders (SAEs) - a technique for decomposing polysemantic neural network activations into sparse, interpretable features. Based on Anthropic's groundbreaking research on monosemanticity.
-
-**GitHub**: [jbloomAus/SAELens](https://github.com/jbloomAus/SAELens) (1,100+ stars)
-
-## The Problem: Polysemanticity & Superposition
-
-Individual neurons in neural networks are **polysemantic** - they activate in multiple, semantically distinct contexts. This happens because models use **superposition** to represent more features than they have neurons, making interpretability difficult.
-
-**SAEs solve this** by decomposing dense activations into sparse, monosemantic features - typically only a small number of features activate for any given input, and each feature corresponds to an interpretable concept.
-
-## When to Use SAELens
-
-**Use SAELens when you need to:**
-- Discover interpretable features in model activations
-- Understand what concepts a model has learned
-- Study superposition and feature geometry
-- Perform feature-based steering or ablation
-- Analyze safety-relevant features (deception, bias, harmful content)
-
-**Consider alternatives when:**
-- You need basic activation analysis → Use **TransformerLens** directly
-- You want causal intervention experiments → Use **pyvene** or **TransformerLens**
-- You need production steering → Consider direct activation engineering
-
-## Installation
-
-```bash
-pip install sae-lens
-```
-
-Requirements: Python 3.10+, transformer-lens>=2.0.0
-
-## Core Concepts
-
-### What SAEs Learn
-
-SAEs are trained to reconstruct model activations through a sparse bottleneck:
-
-```
-Input Activation → Encoder → Sparse Features → Decoder → Reconstructed Activation
-    (d_model)       ↓        (d_sae >> d_model)    ↓         (d_model)
-                 sparsity                      reconstruction
-                 penalty                          loss
-```
-
-**Loss Function**: `MSE(original, reconstructed) + L1_coefficient × L1(features)`
-
-### Key Validation (Anthropic Research)
-
-In "Towards Monosemanticity", human evaluators found **70% of SAE features genuinely interpretable**. Features discovered include:
-- DNA sequences, legal language, HTTP requests
-- Hebrew text, nutrition statements, code syntax
-- Sentiment, named entities, grammatical structures
-
-## Workflow 1: Loading and Analyzing Pre-trained SAEs
-
-### Step-by-Step
-
-```python
-from transformer_lens import HookedTransformer
-from sae_lens import SAE
-
-# 1. Load model and pre-trained SAE
-model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
-# In sae-lens v6, SAE.from_pretrained() returns JUST the SAE (not a tuple).
-sae = SAE.from_pretrained(
-    release="gpt2-small-res-jb",
-    sae_id="blocks.8.hook_resid_pre",
-    device="cuda"
-)
-# If you also need the cfg dict and feature sparsity, use:
-# sae, cfg_dict, sparsity = SAE.from_pretrained_with_cfg_and_sparsity(...)
-
-# 2. Get model activations
-tokens = model.to_tokens("The capital of France is Paris")
-_, cache = model.run_with_cache(tokens)
-activations = cache["resid_pre", 8]  # [batch, pos, d_model]
-
-# 3. Encode to SAE features
-sae_features = sae.encode(activations)  # [batch, pos, d_sae]
-print(f"Active features: {(sae_features > 0).sum()}")
-
-# 4. Find top features for each position
-for pos in range(tokens.shape[1]):
-    top_features = sae_features[0, pos].topk(5)
-    token = model.to_str_tokens(tokens[0, pos:pos+1])[0]
-    print(f"Token '{token}': features {top_features.indices.tolist()}")
-
-# 5. Reconstruct activations
-reconstructed = sae.decode(sae_features)
-reconstruction_error = (activations - reconstructed).norm()
-```
-
-### Available Pre-trained SAEs
-
-| Release | Model | Layers |
-|---------|-------|--------|
-| `gpt2-small-res-jb` | GPT-2 Small | Multiple residual streams |
-| `gemma-2b-res` | Gemma 2B | Residual streams |
-| Various on HuggingFace | Search tag `saelens` | Various |
-
-### Checklist
-- [ ] Load model with TransformerLens
-- [ ] Load matching SAE for target layer
-- [ ] Encode activations to sparse features
-- [ ] Identify top-activating features per token
-- [ ] Validate reconstruction quality
-
-## Workflow 2: Training a Custom SAE
-
-### Step-by-Step
-
-```python
-from sae_lens import (
-    LanguageModelSAETrainingRunner,
-    LanguageModelSAERunnerConfig,
-    StandardTrainingSAEConfig,
-    LoggingConfig,
-)
-
-# 1. Configure training (v6 uses a NESTED config: SAE-specific options live in a
-#    `sae=` sub-config, and logging options live in a `logger=` sub-config).
-#    Note: `architecture`, `d_sae`, `l1_coefficient` etc. are now on the SAE sub-config,
-#    and legacy flat options like `hook_layer`, `activation_fn`, `log_to_wandb` were removed.
-cfg = LanguageModelSAERunnerConfig(
-    # SAE architecture + sparsity (nested)
-    sae=StandardTrainingSAEConfig(
-        d_in=768,          # Model dimension
-        d_sae=768 * 8,     # Expansion factor of 8
-        l1_coefficient=8e-5,  # Sparsity penalty
-        apply_b_dec_to_input=True,
-        normalize_activations="expected_average_only_in",
-    ),
-
-    # Data-generating function (model + hook point)
-    model_name="gpt2-small",
-    hook_name="blocks.8.hook_resid_pre",  # layer is inferred from hook_name (no hook_layer)
-
-    # Training
-    lr=4e-4,
-    l1_warm_up_steps=1000,
-    train_batch_size_tokens=4096,
-    training_tokens=100_000_000,
-
-    # Data
-    dataset_path="monology/pile-uncopyrighted",
-    context_size=128,
-
-    # Logging (nested)
-    logger=LoggingConfig(
-        log_to_wandb=True,
-        wandb_project="sae-training",
-    ),
-
-    # Checkpointing
-    checkpoint_path="checkpoints",
-    n_checkpoints=5,
-)
-
-# 2. Train
-trainer = LanguageModelSAETrainingRunner(cfg)  # SAETrainingRunner still works as an alias
-sae = trainer.run()
-
-# 3. Evaluate
-print(f"L0 (avg active features): {trainer.metrics['l0']}")
-print(f"CE Loss Recovered: {trainer.metrics['ce_loss_score']}")
-```
-
-> **v6 migration note:** For other SAE types swap the `sae=` sub-config —
-> `GatedTrainingSAEConfig`, `TopKTrainingSAEConfig` (set `k` directly), or
-> `JumpReLUTrainingSAEConfig` (uses `l0_coefficient`). Legacy flat options
-> (`architecture`, `expansion_factor`, `hook_layer`, `activation_fn`/`activation_fn_kwargs`,
-> `use_ghost_grads`, ghost grads, b_dec/decoder init options) were removed in v6.
-
-### Key Hyperparameters
-
-| Parameter | Typical Value | Effect |
-|-----------|---------------|--------|
-| `d_sae` | 4-16× d_model | More features, higher capacity |
-| `l1_coefficient` | 5e-5 to 1e-4 | Higher = sparser, less accurate |
-| `lr` | 1e-4 to 1e-3 | Standard optimizer LR |
-| `l1_warm_up_steps` | 500-2000 | Prevents early feature death |
-
-### Evaluation Metrics
-
-| Metric | Target | Meaning |
-|--------|--------|---------|
-| **L0** | 50-200 | Average active features per token |
-| **CE Loss Score** | 80-95% | Cross-entropy recovered vs original |
-| **Dead Features** | <5% | Features that never activate |
-| **Explained Variance** | >90% | Reconstruction quality |
-
-### Checklist
-- [ ] Choose target layer and hook point
-- [ ] Set expansion factor (d_sae = 4-16× d_model)
-- [ ] Tune L1 coefficient for desired sparsity
-- [ ] Enable L1 warm-up to prevent dead features
-- [ ] Monitor metrics during training (W&B)
-- [ ] Validate L0 and CE loss recovery
-- [ ] Check dead feature ratio
-
-## Workflow 3: Feature Analysis and Steering
-
-### Analyzing Individual Features
-
-```python
-from transformer_lens import HookedTransformer
-from sae_lens import SAE
-import torch
-
-model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
-sae = SAE.from_pretrained(  # v6 returns just the SAE
-    release="gpt2-small-res-jb",
-    sae_id="blocks.8.hook_resid_pre",
-    device="cuda"
-)
-
-# Find what activates a specific feature
-feature_idx = 1234
-test_texts = [
-    "The scientist conducted an experiment",
-    "I love chocolate cake",
-    "The code compiles successfully",
-    "Paris is beautiful in spring",
-]
-
-for text in test_texts:
-    tokens = model.to_tokens(text)
-    _, cache = model.run_with_cache(tokens)
-    features = sae.encode(cache["resid_pre", 8])
-    activation = features[0, :, feature_idx].max().item()
-    print(f"{activation:.3f}: {text}")
-```
-
-### Feature Steering
-
-```python
-def steer_with_feature(model, sae, prompt, feature_idx, strength=5.0):
-    """Add SAE feature direction to residual stream."""
-    tokens = model.to_tokens(prompt)
-
-    # Get feature direction from decoder
-    feature_direction = sae.W_dec[feature_idx]  # [d_model]
-
-    def steering_hook(activation, hook):
-        # Add scaled feature direction at all positions
-        activation += strength * feature_direction
-        return activation
-
-    # Generate with steering
-    output = model.generate(
-        tokens,
-        max_new_tokens=50,
-        fwd_hooks=[("blocks.8.hook_resid_pre", steering_hook)]
-    )
-    return model.to_string(output[0])
-```
-
-### Feature Attribution
-
-```python
-# Which features most affect a specific output?
-tokens = model.to_tokens("The capital of France is")
-_, cache = model.run_with_cache(tokens)
-
-# Get features at final position
-features = sae.encode(cache["resid_pre", 8])[0, -1]  # [d_sae]
-
-# Get logit attribution per feature
-# Feature contribution = feature_activation × decoder_weight × unembedding
-W_dec = sae.W_dec  # [d_sae, d_model]
-W_U = model.W_U    # [d_model, vocab]
-
-# Contribution to "Paris" logit
-paris_token = model.to_single_token(" Paris")
-feature_contributions = features * (W_dec @ W_U[:, paris_token])
-
-top_features = feature_contributions.topk(10)
-print("Top features for 'Paris' prediction:")
-for idx, val in zip(top_features.indices, top_features.values):
-    print(f"  Feature {idx.item()}: {val.item():.3f}")
-```
-
-## Common Issues & Solutions
-
-> All examples below use the v6 nested config: SAE-specific options go in the `sae=`
-> sub-config (`StandardTrainingSAEConfig` / `TopKTrainingSAEConfig` / etc.), training
-> knobs stay on the top-level `LanguageModelSAERunnerConfig`.
-
-### Issue: High dead feature ratio
-```python
-from sae_lens import LanguageModelSAERunnerConfig, StandardTrainingSAEConfig
-
-# WRONG: no warm-up, features die early
-cfg = LanguageModelSAERunnerConfig(
-    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=1e-4),
-    l1_warm_up_steps=0,  # Bad!
-)
-
-# RIGHT: warm up the L1 penalty (v6 removed ghost grads; warm-up is the lever now)
-cfg = LanguageModelSAERunnerConfig(
-    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=8e-5),
-    l1_warm_up_steps=1000,  # Gradually increase
-)
-```
-
-### Issue: Poor reconstruction (low CE recovery)
-```python
-# Reduce sparsity penalty and/or add capacity (both on the SAE sub-config)
-cfg = LanguageModelSAERunnerConfig(
-    sae=StandardTrainingSAEConfig(
-        d_in=768,
-        d_sae=768 * 16,       # More capacity
-        l1_coefficient=5e-5,  # Lower = better reconstruction
-    ),
-)
-```
-
-### Issue: Features not interpretable
-```python
-from sae_lens import LanguageModelSAERunnerConfig, StandardTrainingSAEConfig, TopKTrainingSAEConfig
-
-# Increase sparsity (higher L1)
-cfg = LanguageModelSAERunnerConfig(
-    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=1e-4),
-)
-# Or use a TopK SAE (k is set directly in v6, not via activation_fn_kwargs)
-cfg = LanguageModelSAERunnerConfig(
-    sae=TopKTrainingSAEConfig(d_in=768, d_sae=768*8, k=50),  # Exactly 50 active features
-)
-```
-
-### Issue: Memory errors during training
-```python
-cfg = LanguageModelSAERunnerConfig(
-    sae=StandardTrainingSAEConfig(d_in=768, d_sae=768*8, l1_coefficient=8e-5),
-    train_batch_size_tokens=2048,  # Reduce batch size
-    store_batch_size_prompts=4,    # Fewer prompts in buffer
-    n_batches_in_buffer=8,         # Smaller activation buffer
-)
-```
-
-## Integration with Neuronpedia
-
-Browse pre-trained SAE features at [neuronpedia.org](https://neuronpedia.org):
-
-```python
-# Features are indexed by SAE ID
-# Example: gpt2-small layer 8 feature 1234
-# → neuronpedia.org/gpt2-small/8-res-jb/1234
-```
-
-## Key Classes Reference
-
-| Class | Purpose |
-|-------|---------|
-| `SAE` | Sparse Autoencoder model |
-| `LanguageModelSAERunnerConfig` | Top-level training configuration (nests `sae=` and `logger=`) |
-| `StandardTrainingSAEConfig` / `TopKTrainingSAEConfig` / `GatedTrainingSAEConfig` / `JumpReLUTrainingSAEConfig` | SAE-type-specific sub-configs (v6) |
-| `LoggingConfig` | Logging/W&B sub-config (v6) |
-| `LanguageModelSAETrainingRunner` | Training loop manager (alias: `SAETrainingRunner`) |
-| `ActivationsStore` | Activation collection and batching |
-| `HookedSAETransformer` | TransformerLens + SAE integration |
-
-## Reference Documentation
-
-For detailed API documentation, tutorials, and advanced usage, see the `references/` folder:
-
-| File | Contents |
-|------|----------|
-| [references/README.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\saelens/references/README.md) | Overview and quick start guide |
-| [references/api.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\saelens/references/api.md) | Complete API reference for SAE, TrainingSAE, configurations |
-| [references/tutorials.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\saelens/references/tutorials.md) | Step-by-step tutorials for training, analysis, steering |
-
-## External Resources
-
-### Tutorials
-- [Basic Loading & Analysis](https://github.com/jbloomAus/SAELens/blob/main/tutorials/basic_loading_and_analysing.ipynb)
-- [Training a Sparse Autoencoder](https://github.com/jbloomAus/SAELens/blob/main/tutorials/training_a_sparse_autoencoder.ipynb)
-- [ARENA SAE Curriculum](https://www.lesswrong.com/posts/LnHowHgmrMbWtpkxx/intro-to-superposition-and-sparse-autoencoders-colab)
-
-### Papers
-- [Towards Monosemanticity](https://transformer-circuits.pub/2023/monosemantic-features) - Anthropic (2023)
-- [Scaling Monosemanticity](https://transformer-circuits.pub/2024/scaling-monosemanticity/) - Anthropic (2024)
-- [Sparse Autoencoders Find Highly Interpretable Features](https://arxiv.org/abs/2309.08600) - Cunningham et al. (ICLR 2024)
-
-### Official Documentation
-- [SAELens Docs](https://jbloomaus.github.io/SAELens/)
-- [Neuronpedia](https://neuronpedia.org) - Feature browser
-
-## SAE Architectures
-
-| Architecture | Description | Use Case |
-|--------------|-------------|----------|
-| **Standard** | ReLU + L1 penalty | General purpose |
-| **Gated** | Learned gating mechanism | Better sparsity control |
-| **TopK** | Exactly K active features | Consistent sparsity |
-
-```python
-from sae_lens import LanguageModelSAERunnerConfig, TopKTrainingSAEConfig
-
-# TopK SAE (exactly 50 features active) — `k` is set on the SAE sub-config in v6
-cfg = LanguageModelSAERunnerConfig(
-    sae=TopKTrainingSAEConfig(d_in=768, d_sae=768*8, k=50),
-)
-```

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-simpo.md

@@ -1,237 +0,0 @@
----
-title: "Simpo — Reference-free preference alignment, simpler than DPO"
-sidebar_label: "Simpo"
-description: "Reference-free preference alignment, simpler than DPO"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Simpo
-
-Reference-free preference alignment, simpler than DPO.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/simpo` |
-| Path | `optional-skills/mlops\simpo` |
-| Version | `1.0.0` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `torch`, `transformers`, `datasets`, `trl`, `accelerate` |
-| Platforms | linux, macos, windows |
-| Tags | `Post-Training`, `SimPO`, `Preference Optimization`, `Alignment`, `DPO Alternative`, `Reference-Free`, `LLM Alignment`, `Efficient Training` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# SimPO - Simple Preference Optimization
-
-## Quick start
-
-SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.
-
-**Installation**:
-```bash
-# Create environment
-conda create -n simpo python=3.10 && conda activate simpo
-
-# Install PyTorch 2.2.2
-# Visit: https://pytorch.org/get-started/locally/
-
-# Install alignment-handbook
-git clone https://github.com/huggingface/alignment-handbook.git
-cd alignment-handbook
-python -m pip install .
-
-# Install Flash Attention 2
-python -m pip install flash-attn --no-build-isolation
-```
-
-**Training** (Mistral 7B):
-```bash
-ACCELERATE_LOG_LEVEL=info accelerate launch \
-  --config_file accelerate_configs/deepspeed_zero3.yaml \
-  scripts/run_simpo.py \
-  training_configs/mistral-7b-base-simpo.yaml
-```
-
-## Common workflows
-
-### Workflow 1: Train from base model (Mistral 7B)
-
-**Config** (`mistral-7b-base-simpo.yaml`):
-```yaml
-# Model
-model_name_or_path: mistralai/Mistral-7B-v0.1
-torch_dtype: bfloat16
-
-# Dataset
-dataset_mixer:
-  HuggingFaceH4/ultrafeedback_binarized: 1.0
-dataset_splits:
-  - train_prefs
-  - test_prefs
-
-# SimPO hyperparameters
-beta: 2.0                  # Reward scaling (2.0-10.0)
-gamma_beta_ratio: 0.5       # Target margin (0-1)
-loss_type: sigmoid          # sigmoid or hinge
-sft_weight: 0.0             # Optional SFT regularization
-
-# Training
-learning_rate: 5e-7         # Critical: 3e-7 to 1e-6
-num_train_epochs: 1
-per_device_train_batch_size: 1
-gradient_accumulation_steps: 8
-
-# Output
-output_dir: ./outputs/mistral-7b-simpo
-```
-
-**Launch training**:
-```bash
-accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
-  scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml
-```
-
-### Workflow 2: Fine-tune instruct model (Llama 3 8B)
-
-**Config** (`llama3-8b-instruct-simpo.yaml`):
-```yaml
-model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
-
-dataset_mixer:
-  argilla/ultrafeedback-binarized-preferences-cleaned: 1.0
-
-beta: 2.5
-gamma_beta_ratio: 0.5
-learning_rate: 5e-7
-sft_weight: 0.1             # Add SFT loss to preserve capabilities
-
-num_train_epochs: 1
-per_device_train_batch_size: 2
-gradient_accumulation_steps: 4
-output_dir: ./outputs/llama3-8b-simpo
-```
-
-**Launch**:
-```bash
-accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
-  scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml
-```
-
-### Workflow 3: Reasoning-intensive tasks (lower LR)
-
-**For math/code tasks**:
-```yaml
-model_name_or_path: deepseek-ai/deepseek-math-7b-base
-
-dataset_mixer:
-  argilla/distilabel-math-preference-dpo: 1.0
-
-beta: 5.0                   # Higher for stronger signal
-gamma_beta_ratio: 0.7       # Larger margin
-learning_rate: 3e-7         # Lower LR for reasoning
-sft_weight: 0.0
-
-num_train_epochs: 1
-per_device_train_batch_size: 1
-gradient_accumulation_steps: 16
-```
-
-## When to use vs alternatives
-
-**Use SimPO when**:
-- Want simpler training than DPO (no reference model)
-- Have preference data (chosen/rejected pairs)
-- Need better performance than DPO
-- Limited compute resources
-- Single-node training sufficient
-
-**Algorithm selection**:
-- **SimPO**: Simplest, best performance, no reference model
-- **DPO**: Need reference model baseline, more conservative
-- **PPO**: Maximum control, need reward model, complex setup
-- **GRPO**: Memory-efficient RL, no critic
-
-**Use alternatives instead**:
-- **OpenRLHF**: Multi-node distributed training, PPO/GRPO
-- **TRL**: Need multiple methods in one framework
-- **DPO**: Established baseline comparison
-
-## Common issues
-
-**Issue: Loss divergence**
-
-Reduce learning rate:
-```yaml
-learning_rate: 3e-7  # Reduce from 5e-7
-```
-
-Reduce beta:
-```yaml
-beta: 1.0  # Reduce from 2.0
-```
-
-**Issue: Model forgets capabilities**
-
-Add SFT regularization:
-```yaml
-sft_weight: 0.1  # Add SFT loss component
-```
-
-**Issue: Poor preference separation**
-
-Increase beta and margin:
-```yaml
-beta: 5.0            # Increase from 2.0
-gamma_beta_ratio: 0.8  # Increase from 0.5
-```
-
-**Issue: OOM during training**
-
-Reduce batch size:
-```yaml
-per_device_train_batch_size: 1
-gradient_accumulation_steps: 16  # Maintain effective batch
-```
-
-Enable gradient checkpointing:
-```yaml
-gradient_checkpointing: true
-```
-
-## Advanced topics
-
-**Loss functions**: See [references/loss-functions.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\simpo/references/loss-functions.md) for sigmoid vs hinge loss, mathematical formulations, and when to use each.
-
-**Hyperparameter tuning**: See [references/hyperparameters.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\simpo/references/hyperparameters.md) for beta, gamma, learning rate selection guide, and model-size-specific recommendations.
-
-**Dataset preparation**: See [references/datasets.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\simpo/references/datasets.md) for preference data formats, quality filtering, and custom dataset creation.
-
-## Hardware requirements
-
-- **GPU**: NVIDIA A100/H100 recommended
-- **VRAM**:
-  - 7B model: 1× A100 40GB (DeepSpeed ZeRO-3)
-  - 8B model: 2× A100 40GB
-  - 70B model: 8× A100 80GB
-- **Single-node**: DeepSpeed ZeRO-3 sufficient
-- **Mixed precision**: BF16 recommended
-
-**Memory optimization**:
-- DeepSpeed ZeRO-3 (default config)
-- Gradient checkpointing
-- Flash Attention 2
-
-## Resources
-
-- Paper: https://arxiv.org/abs/2405.14734 (NeurIPS 2024)
-- GitHub: https://github.com/princeton-nlp/SimPO
-- Models: https://huggingface.co/princeton-nlp
-- Alignment Handbook: https://github.com/huggingface/alignment-handbook