better-dsh 0.2.5 → 0.3.0-c

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-tensorrt-llm.md

@@ -1,208 +0,0 @@
----
-title: "Tensorrt Llm — High-throughput LLM inference on NVIDIA GPUs"
-sidebar_label: "Tensorrt Llm"
-description: "High-throughput LLM inference on NVIDIA GPUs"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Tensorrt Llm
-
-High-throughput LLM inference on NVIDIA GPUs.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/tensorrt-llm` |
-| Path | `optional-skills/mlops\tensorrt-llm` |
-| Version | `1.0.1` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `tensorrt-llm`, `torch` |
-| Platforms | linux, macos |
-| Tags | `Inference Serving`, `TensorRT-LLM`, `NVIDIA`, `Inference Optimization`, `High Throughput`, `Low Latency`, `Production`, `FP8`, `INT4`, `In-Flight Batching`, `Multi-GPU` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# TensorRT-LLM
-
-NVIDIA's open-source library for optimizing LLM inference with high performance on NVIDIA GPUs.
-
-## When to use TensorRT-LLM
-
-**Use TensorRT-LLM when:**
-- Deploying on NVIDIA GPUs (A100, H100, GB200)
-- Need maximum throughput (24,000+ tokens/sec on Llama 3)
-- Require low latency for real-time applications
-- Working with quantized models (FP8, INT4, FP4)
-- Scaling across multiple GPUs or nodes
-
-**Use vLLM instead when:**
-- Need simpler setup and Python-first API
-- Want PagedAttention without TensorRT compilation
-- Working with AMD GPUs or non-NVIDIA hardware
-
-**Use llama.cpp instead when:**
-- Deploying on CPU or Apple Silicon
-- Need edge deployment without NVIDIA GPUs
-- Want simpler GGUF quantization format
-
-## Quick start
-
-### Installation
-
-```bash
-# Docker (recommended) — images are on NGC (nvcr.io), not Docker Hub.
-# Replace x.y.z with the desired version (e.g. 1.2.1). Browse tags on NGC:
-# https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags
-docker pull nvcr.io/nvidia/tensorrt-llm/release:x.y.z
-
-# pip install (current stable GA)
-pip install tensorrt_llm
-
-# Requires CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.12
-```
-
-### Basic inference
-
-```python
-from tensorrt_llm import LLM, SamplingParams
-
-# Initialize model
-llm = LLM(model="meta-llama/Meta-Llama-3-8B")
-
-# Configure sampling
-sampling_params = SamplingParams(
-    max_tokens=100,
-    temperature=0.7,
-    top_p=0.9
-)
-
-# Generate
-prompts = ["Explain quantum computing"]
-outputs = llm.generate(prompts, sampling_params)
-
-for output in outputs:
-    print(output.text)
-```
-
-### Serving with trtllm-serve
-
-```bash
-# Start server (automatic model download and compilation)
-trtllm-serve meta-llama/Meta-Llama-3-8B \
-    --tp_size 4 \              # Tensor parallelism (4 GPUs)
-    --max_batch_size 256 \
-    --max_num_tokens 4096
-
-# Client request
-curl -X POST http://localhost:8000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -d '{
-    "model": "meta-llama/Meta-Llama-3-8B",
-    "messages": [{"role": "user", "content": "Hello!"}],
-    "temperature": 0.7,
-    "max_tokens": 100
-  }'
-```
-
-## Key features
-
-### Performance optimizations
-- **In-flight batching**: Dynamic batching during generation
-- **Paged KV cache**: Efficient memory management
-- **Flash Attention**: Optimized attention kernels
-- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference
-- **CUDA graphs**: Reduced kernel launch overhead
-
-### Parallelism
-- **Tensor parallelism (TP)**: Split model across GPUs
-- **Pipeline parallelism (PP)**: Layer-wise distribution
-- **Expert parallelism**: For Mixture-of-Experts models
-- **Multi-node**: Scale beyond single machine
-
-### Advanced features
-- **Speculative decoding**: Faster generation with draft models
-- **LoRA serving**: Efficient multi-adapter deployment
-- **Disaggregated serving**: Separate prefill and generation
-
-## Common patterns
-
-### Quantized model (FP8)
-
-```python
-from tensorrt_llm import LLM
-
-# Load FP8 quantized model (2× faster, 50% memory)
-llm = LLM(
-    model="meta-llama/Meta-Llama-3-70B",
-    dtype="fp8",
-    max_num_tokens=8192
-)
-
-# Inference same as before
-outputs = llm.generate(["Summarize this article..."])
-```
-
-### Multi-GPU deployment
-
-```python
-# Tensor parallelism across 8 GPUs
-llm = LLM(
-    model="meta-llama/Meta-Llama-3-405B",
-    tensor_parallel_size=8,
-    dtype="fp8"
-)
-```
-
-### Batch inference
-
-```python
-# Process 100 prompts efficiently
-prompts = [f"Question {i}: ..." for i in range(100)]
-
-outputs = llm.generate(
-    prompts,
-    sampling_params=SamplingParams(max_tokens=200)
-)
-
-# Automatic in-flight batching for maximum throughput
-```
-
-## Performance benchmarks
-
-**Meta Llama 3-8B** (H100 GPU):
-- Throughput: 24,000 tokens/sec
-- Latency: ~10ms per token
-- vs PyTorch: **100× faster**
-
-**Llama 3-70B** (8× A100 80GB):
-- FP8 quantization: 2× faster than FP16
-- Memory: 50% reduction with FP8
-
-## Supported models
-
-- **LLaMA family**: Llama 2, Llama 3, CodeLlama
-- **GPT family**: GPT-2, GPT-J, GPT-NeoX
-- **Qwen**: Qwen, Qwen2, QwQ
-- **DeepSeek**: DeepSeek-V2, DeepSeek-V3
-- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B
-- **Vision**: LLaVA, Phi-3-vision
-- **100+ models** on HuggingFace
-
-## References
-
-- **[Optimization Guide](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\tensorrt-llm/references/optimization.md)** - Quantization, batching, KV cache tuning
-- **[Multi-GPU Setup](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\tensorrt-llm/references/multi-gpu.md)** - Tensor/pipeline parallelism, multi-node
-- **[Serving Guide](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\tensorrt-llm/references/serving.md)** - Production deployment, monitoring, autoscaling
-
-## Resources
-
-- **Docs**: https://nvidia.github.io/TensorRT-LLM/
-- **GitHub**: https://github.com/NVIDIA/TensorRT-LLM
-- **Models**: https://huggingface.co/models?library=tensorrt_llm

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-torchtitan.md

@@ -1,404 +0,0 @@
----
-title: "Torchtitan — Pretrain LLMs at scale with PyTorch 4D parallelism"
-sidebar_label: "Torchtitan"
-description: "Pretrain LLMs at scale with PyTorch 4D parallelism"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Torchtitan
-
-Pretrain LLMs at scale with PyTorch 4D parallelism.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/torchtitan` |
-| Path | `optional-skills/mlops\torchtitan` |
-| Version | `1.0.1` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `torch>=2.6.0`, `torchtitan>=0.2.0`, `torchao>=0.5.0` |
-| Platforms | linux, macos |
-| Tags | `Model Architecture`, `Distributed Training`, `TorchTitan`, `FSDP2`, `Tensor Parallel`, `Pipeline Parallel`, `Context Parallel`, `Float8`, `Llama`, `Pretraining` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# TorchTitan - PyTorch Native Distributed LLM Pretraining
-
-## Quick start
-
-TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.
-
-**Installation**:
-```bash
-# From PyPI (stable)
-pip install torchtitan
-
-# From source (latest features, requires PyTorch nightly)
-git clone https://github.com/pytorch/torchtitan
-cd torchtitan
-pip install -r requirements.txt
-```
-
-**Download tokenizer**:
-```bash
-# Get HF token from https://huggingface.co/settings/tokens
-python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
-```
-
-**Start training on 8 GPUs**:
-```bash
-# Configs are selected by name from the Python config registry
-# (torchtitan/models/llama3/config_registry.py), not by TOML path
-MODULE=llama3 CONFIG=llama3_8b ./run_train.sh
-```
-
-## Common workflows
-
-### Workflow 1: Pretrain Llama 3.1 8B on single node
-
-Copy this checklist:
-
-```
-Single Node Pretraining:
-- [ ] Step 1: Download tokenizer
-- [ ] Step 2: Configure training
-- [ ] Step 3: Launch training
-- [ ] Step 4: Monitor and checkpoint
-```
-
-**Step 1: Download tokenizer**
-
-```bash
-python scripts/download_hf_assets.py \
-  --repo_id meta-llama/Llama-3.1-8B \
-  --assets tokenizer \
-  --hf_token=YOUR_HF_TOKEN
-```
-
-**Step 2: Configure training**
-
-In torchtitan's current layout, run configs are defined in a Python **config registry**
-(`torchtitan/models/llama3/config_registry.py`) and selected by name via `CONFIG=<name>`
-(or `--config <name>`). To customize, register your own config in the registry, or override
-individual fields on the command line (e.g. `--optimizer.lr 3e-4 --training.steps 1000`).
-
-The equivalent settings for an 8B run look like this (shown as fields; set them in the
-registry entry or as `--section.key value` overrides):
-
-```toml
-# fields for a llama3 8B run (register in config_registry.py or pass as --overrides)
-[job]
-dump_folder = "./outputs"
-description = "Llama 3.1 8B training"
-
-[model]
-name = "llama3"
-flavor = "8B"
-hf_assets_path = "./assets/hf/Llama-3.1-8B"
-
-[optimizer]
-name = "AdamW"
-lr = 3e-4
-
-[lr_scheduler]
-warmup_steps = 200
-
-[training]
-local_batch_size = 2
-seq_len = 8192
-max_norm = 1.0
-steps = 1000
-dataset = "c4"
-
-[parallelism]
-data_parallel_shard_degree = -1  # Use all GPUs for FSDP
-
-[activation_checkpoint]
-mode = "selective"
-selective_ac_option = "op"
-
-[checkpoint]
-enable = true
-folder = "checkpoint"
-interval = 500
-```
-
-**Step 3: Launch training**
-
-```bash
-# 8 GPUs on single node (config selected by name from the registry)
-MODULE=llama3 CONFIG=llama3_8b ./run_train.sh
-
-# Override individual fields on the command line
-MODULE=llama3 CONFIG=llama3_8b ./run_train.sh --optimizer.lr 3e-4 --training.steps 1000
-
-# Or explicitly with torchrun (run_train.sh wraps this)
-torchrun --nproc_per_node=8 \
-  -m torchtitan.train \
-  --module llama3 --config llama3_8b
-```
-
-**Step 4: Monitor and checkpoint**
-
-TensorBoard logs are saved to `./outputs/tb/`:
-```bash
-tensorboard --logdir ./outputs/tb
-```
-
-### Workflow 2: Multi-node training with SLURM
-
-```
-Multi-Node Training:
-- [ ] Step 1: Configure parallelism for scale
-- [ ] Step 2: Set up SLURM script
-- [ ] Step 3: Submit job
-- [ ] Step 4: Resume from checkpoint
-```
-
-**Step 1: Configure parallelism for scale**
-
-For 70B model on 256 GPUs (32 nodes):
-```toml
-[parallelism]
-data_parallel_shard_degree = 32  # FSDP across 32 ranks
-tensor_parallel_degree = 8        # TP within node
-pipeline_parallel_degree = 1      # No PP for 70B
-context_parallel_degree = 1       # Increase for long sequences
-```
-
-**Step 2: Set up SLURM script**
-
-```bash
-#!/bin/bash
-#SBATCH --job-name=llama70b
-#SBATCH --nodes=32
-#SBATCH --ntasks-per-node=8
-#SBATCH --gpus-per-node=8
-
-srun torchrun \
-  --nnodes=32 \
-  --nproc_per_node=8 \
-  --rdzv_backend=c10d \
-  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
-  -m torchtitan.train \
-  --module llama3 --config llama3_70b
-```
-
-**Step 3: Submit job**
-
-```bash
-sbatch multinode_trainer.slurm
-```
-
-**Step 4: Resume from checkpoint**
-
-Training auto-resumes if checkpoint exists in configured folder.
-
-### Workflow 3: Enable Float8 training for H100s
-
-Float8 provides 30-50% speedup on H100 GPUs.
-
-```
-Float8 Training:
-- [ ] Step 1: Install torchao
-- [ ] Step 2: Configure Float8
-- [ ] Step 3: Launch with compile
-```
-
-**Step 1: Install torchao**
-
-```bash
-USE_CPP=0 pip install git+https://github.com/pytorch/ao.git
-```
-
-**Step 2: Configure Float8**
-
-In the current torchtitan, Float8 is applied at config time via the `quantization`
-parameter in your `model_registry()` call inside the config registry (not via a
-`[quantize.linear.float8]` TOML section). Add a `Float8LinearConverter.Config`:
-
-```python
-# in torchtitan/models/llama3/config_registry.py (your model_registry(...) call)
-from torchtitan.components.quantization import Float8LinearConverter
-
-model_spec = model_registry(
-    "8B",
-    quantization=[
-        Float8LinearConverter.Config(
-            recipe_name="rowwise",          # or "rowwise_with_gw_hp"
-            filter_fqns=["output"],          # skip layers too small to benefit
-            model_compile_enabled=True,      # requires torch.compile for competitive perf
-        ),
-    ],
-)
-```
-
-Enable `torch.compile` in your run config too:
-```toml
-[compile]
-enable = true
-components = ["model", "loss"]
-```
-
-**Step 3: Launch with compile**
-
-```bash
-# Float8 config is baked into the registered config; just select it and enable compile
-MODULE=llama3 CONFIG=llama3_8b ./run_train.sh --compile.enable
-```
-
-### Workflow 4: 4D parallelism for 405B models
-
-```
-4D Parallelism (FSDP + TP + PP + CP):
-- [ ] Step 1: Create seed checkpoint
-- [ ] Step 2: Configure 4D parallelism
-- [ ] Step 3: Launch on 512 GPUs
-```
-
-**Step 1: Create seed checkpoint**
-
-Required for consistent initialization across PP stages:
-```bash
-NGPU=1 MODULE=llama3 CONFIG=llama3_405b ./run_train.sh \
-  --checkpoint.enable \
-  --checkpoint.create_seed_checkpoint \
-  --parallelism.data_parallel_shard_degree 1 \
-  --parallelism.tensor_parallel_degree 1 \
-  --parallelism.pipeline_parallel_degree 1
-```
-
-**Step 2: Configure 4D parallelism**
-
-```toml
-[parallelism]
-data_parallel_shard_degree = 8   # FSDP
-tensor_parallel_degree = 8       # TP within node
-pipeline_parallel_degree = 8     # PP across nodes
-context_parallel_degree = 1      # CP for long sequences
-
-[training]
-local_batch_size = 32
-seq_len = 8192
-```
-
-**Step 3: Launch on 512 GPUs**
-
-```bash
-# 64 nodes x 8 GPUs = 512 GPUs
-srun torchrun --nnodes=64 --nproc_per_node=8 \
-  -m torchtitan.train \
-  --module llama3 --config llama3_405b
-```
-
-## When to use vs alternatives
-
-**Use TorchTitan when:**
-- Pretraining LLMs from scratch (8B to 405B+)
-- Need PyTorch-native solution without third-party dependencies
-- Require composable 4D parallelism (FSDP2, TP, PP, CP)
-- Training on H100s with Float8 support
-- Want interoperable checkpoints with torchtune/HuggingFace
-
-**Use alternatives instead:**
-- **Megatron-LM**: Maximum performance for NVIDIA-only deployments
-- **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support
-- **Axolotl/TRL**: Fine-tuning rather than pretraining
-- **LitGPT**: Educational, smaller-scale training
-
-## Common issues
-
-**Issue: Out of memory on large models**
-
-Enable activation checkpointing and reduce batch size:
-```toml
-[activation_checkpoint]
-mode = "full"  # Instead of "selective"
-
-[training]
-local_batch_size = 1
-```
-
-Or use gradient accumulation:
-```toml
-[training]
-local_batch_size = 1
-global_batch_size = 32  # Accumulates gradients
-```
-
-**Issue: TP causes high memory with async collectives**
-
-Set environment variable:
-```bash
-export TORCH_NCCL_AVOID_RECORD_STREAMS=1
-```
-
-**Issue: Float8 training not faster**
-
-Float8 only benefits large GEMMs. Filter small layers via the converter's `filter_fqns`:
-```python
-from torchtitan.components.quantization import Float8LinearConverter
-
-Float8LinearConverter.Config(
-    # add "auto_filter_small_kn" to auto-skip layers too small to benefit
-    filter_fqns=["attention.wk", "attention.wv", "output", "auto_filter_small_kn"],
-    model_compile_enabled=True,
-)
-```
-
-**Issue: Checkpoint loading fails after parallelism change**
-
-Use DCP's resharding capability:
-```bash
-# Convert sharded checkpoint to single file
-python -m torch.distributed.checkpoint.format_utils \
-  dcp_to_torch checkpoint/step-1000 checkpoint.pt
-```
-
-**Issue: Pipeline parallelism initialization**
-
-Create seed checkpoint first (see Workflow 4, Step 1).
-
-## Supported models
-
-| Model | Sizes | Status |
-|-------|-------|--------|
-| Llama 3.1 | 8B, 70B, 405B | Production |
-| Llama 4 | Various | Experimental |
-| DeepSeek V3 | 16B, 236B, 671B (MoE) | Experimental |
-| GPT-OSS | 20B, 120B (MoE) | Experimental |
-| Qwen 3 | Various | Experimental |
-| Flux | Diffusion | Experimental |
-
-## Performance benchmarks (H100)
-
-| Model | GPUs | Parallelism | TPS/GPU | Techniques |
-|-------|------|-------------|---------|------------|
-| Llama 8B | 8 | FSDP | 5,762 | Baseline |
-| Llama 8B | 8 | FSDP+compile+FP8 | 8,532 | +48% |
-| Llama 70B | 256 | FSDP+TP+AsyncTP | 876 | 2D parallel |
-| Llama 405B | 512 | FSDP+TP+PP | 128 | 3D parallel |
-
-## Advanced topics
-
-**FSDP2 configuration**: See [references/fsdp.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\torchtitan/references/fsdp.md) for detailed FSDP2 vs FSDP1 comparison and ZeRO equivalents.
-
-**Float8 training**: See [references/float8.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\torchtitan/references/float8.md) for tensorwise vs rowwise scaling recipes.
-
-**Checkpointing**: See [references/checkpoint.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\torchtitan/references/checkpoint.md) for HuggingFace conversion and async checkpointing.
-
-**Adding custom models**: See [references/custom-models.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\torchtitan/references/custom-models.md) for TrainSpec protocol.
-
-## Resources
-
-- GitHub: https://github.com/pytorch/torchtitan
-- Paper: https://arxiv.org/abs/2410.06511
-- ICLR 2025: https://iclr.cc/virtual/2025/poster/29620
-- PyTorch Forum: https://discuss.pytorch.org/c/distributed/torchtitan/44