cli-jaw 0.1.6 → 0.1.8

package/skills_ref/hugging-face-model-trainer/references/gguf_conversion.md

@@ -1,296 +0,0 @@
-# GGUF Conversion Guide
-
-After training models with TRL on Hugging Face Jobs, convert them to **GGUF format** for use with llama.cpp, Ollama, LM Studio, and other local inference tools.
-
-**This guide provides production-ready, tested code based on successful conversions.** All critical dependencies and build steps are included.
-
-## What is GGUF?
-
-**GGUF** (GPT-Generated Unified Format):
-- Optimized format for CPU/GPU inference with llama.cpp
-- Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size
-- Compatible with: Ollama, LM Studio, Jan, GPT4All, llama.cpp
-- Typically 2-8GB for 7B models (vs 14GB unquantized)
-
-## When to Convert to GGUF
-
-**Convert when:**
-- Running models locally with Ollama or LM Studio
-- Using CPU-optimized inference
-- Reducing model size with quantization
-- Deploying to edge devices
-- Sharing models for local-first use
-
-## Critical Success Factors
-
-Based on production testing, these are **essential** for reliable conversion:
-
-### 1. ✅ Install Build Tools FIRST
-**Before cloning llama.cpp**, install build dependencies:
-```python
-subprocess.run(["apt-get", "update", "-qq"], check=True, capture_output=True)
-subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True, capture_output=True)
-```
-
-**Why:** The quantization tool requires gcc and cmake. Installing after cloning doesn't help.
-
-### 2. ✅ Use CMake (Not Make)
-**Build the quantize tool with CMake:**
-```python
-# Create build directory
-os.makedirs("/tmp/llama.cpp/build", exist_ok=True)
-
-# Configure
-subprocess.run([
-    "cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
-    "-DGGML_CUDA=OFF"  # Faster build, CUDA not needed for quantization
-], check=True, capture_output=True, text=True)
-
-# Build
-subprocess.run([
-    "cmake", "--build", "/tmp/llama.cpp/build",
-    "--target", "llama-quantize", "-j", "4"
-], check=True, capture_output=True, text=True)
-
-# Binary path
-quantize_bin = "/tmp/llama.cpp/build/bin/llama-quantize"
-```
-
-**Why:** CMake is more reliable than `make` and produces consistent binary paths.
-
-### 3. ✅ Include All Dependencies
-**PEP 723 header must include:**
-```python
-# /// script
-# dependencies = [
-#     "transformers>=4.36.0",
-#     "peft>=0.7.0",
-#     "torch>=2.0.0",
-#     "accelerate>=0.24.0",
-#     "huggingface_hub>=0.20.0",
-#     "sentencepiece>=0.1.99",  # Required for tokenizer
-#     "protobuf>=3.20.0",        # Required for tokenizer
-#     "numpy",
-#     "gguf",
-# ]
-# ///
-```
-
-**Why:** `sentencepiece` and `protobuf` are critical for tokenizer conversion. Missing them causes silent failures.
-
-### 4. ✅ Verify Names Before Use
-**Always verify repos exist:**
-```python
-# Before submitting job, verify:
-hub_repo_details([ADAPTER_MODEL], repo_type="model")
-hub_repo_details([BASE_MODEL], repo_type="model")
-```
-
-**Why:** Non-existent dataset/model names cause job failures that could be caught in seconds.
-
-## Complete Conversion Script
-
-See `scripts/convert_to_gguf.py` for the complete, production-ready script.
-
-**Key features:**
-- ✅ All dependencies in PEP 723 header
-- ✅ Build tools installed automatically
-- ✅ CMake build process (reliable)
-- ✅ Comprehensive error handling
-- ✅ Environment variable configuration
-- ✅ Automatic README generation
-
-## Quick Conversion Job
-
-```python
-# Before submitting: VERIFY MODELS EXIST
-hub_repo_details(["username/my-finetuned-model"], repo_type="model")
-hub_repo_details(["Qwen/Qwen2.5-0.5B"], repo_type="model")
-
-# Submit conversion job
-hf_jobs("uv", {
-    "script": open("trl/scripts/convert_to_gguf.py").read(),  # Or inline the script
-    "flavor": "a10g-large",
-    "timeout": "45m",
-    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
-    "env": {
-        "ADAPTER_MODEL": "username/my-finetuned-model",
-        "BASE_MODEL": "Qwen/Qwen2.5-0.5B",
-        "OUTPUT_REPO": "username/my-model-gguf",
-        "HF_USERNAME": "username"  # Optional, for README
-    }
-})
-```
-
-## Conversion Process
-
-The script performs these steps:
-
-1. **Load and Merge** - Load base model and LoRA adapter, merge them
-2. **Install Build Tools** - Install gcc, cmake (CRITICAL: before cloning llama.cpp)
-3. **Setup llama.cpp** - Clone repo, install Python dependencies
-4. **Convert to GGUF** - Create FP16 GGUF using llama.cpp converter
-5. **Build Quantize Tool** - Use CMake to build `llama-quantize`
-6. **Quantize** - Create Q4_K_M, Q5_K_M, Q8_0 versions
-7. **Upload** - Upload all versions + README to Hub
-
-## Quantization Options
-
-Common quantization formats (from smallest to largest):
-
-| Format | Size | Quality | Use Case |
-|--------|------|---------|----------|
-| **Q4_K_M** | ~300MB | Good | **Recommended** - best balance of size/quality |
-| **Q5_K_M** | ~350MB | Better | Higher quality, slightly larger |
-| **Q8_0** | ~500MB | Very High | Near-original quality |
-| **F16** | ~1GB | Original | Full precision, largest file |
-
-**Recommendation:** Create Q4_K_M, Q5_K_M, and Q8_0 versions to give users options.
-
-## Hardware Requirements
-
-**For conversion:**
-- Small models (<1B): CPU-basic works, but slow
-- Medium models (1-7B): a10g-large recommended
-- Large models (7B+): a10g-large or a100-large
-
-**Time estimates:**
-- 0.5B model: ~15-25 minutes on A10G
-- 3B model: ~30-45 minutes on A10G
-- 7B model: ~45-60 minutes on A10G
-
-## Using GGUF Models
-
-**GGUF models work on both CPU and GPU.** They're optimized for CPU inference but can also leverage GPU acceleration when available.
-
-### With Ollama (auto-detects GPU)
-```bash
-# Download GGUF
-huggingface-cli download username/my-model-gguf model-q4_k_m.gguf
-
-# Create Modelfile
-echo "FROM ./model-q4_k_m.gguf" > Modelfile
-
-# Create and run (uses GPU automatically if available)
-ollama create my-model -f Modelfile
-ollama run my-model
-```
-
-### With llama.cpp
-```bash
-# CPU only
-./llama-cli -m model-q4_k_m.gguf -p "Your prompt"
-
-# With GPU acceleration (offload 32 layers to GPU)
-./llama-cli -m model-q4_k_m.gguf -ngl 32 -p "Your prompt"
-```
-
-### With LM Studio
-1. Download the `.gguf` file
-2. Import into LM Studio
-3. Start chatting
-
-## Best Practices
-
-### ✅ DO:
-1. **Verify repos exist** before submitting jobs (use `hub_repo_details`)
-2. **Install build tools FIRST** before cloning llama.cpp
-3. **Use CMake** for building quantize tool (not make)
-4. **Include all dependencies** in PEP 723 header (especially sentencepiece, protobuf)
-5. **Create multiple quantizations** - Give users choice
-6. **Test on known models** before production use
-7. **Use A10G GPU** for faster conversion
-
-### ❌ DON'T:
-1. **Assume repos exist** - Always verify with hub tools
-2. **Use make** instead of CMake - Less reliable
-3. **Remove dependencies** to "simplify" - They're all needed
-4. **Skip build tools** - Quantization will fail silently
-5. **Use default paths** - CMake puts binaries in build/bin/
-
-## Common Issues
-
-### Out of memory during merge
-**Fix:**
-- Use larger GPU (a10g-large or a100-large)
-- Ensure `device_map="auto"` for automatic placement
-- Use `dtype=torch.float16` or `torch.bfloat16`
-
-### Conversion fails with architecture error
-**Fix:**
-- Ensure llama.cpp supports the model architecture
-- Check for standard architecture (Qwen, Llama, Mistral, etc.)
-- Update llama.cpp to latest: `git clone --depth 1 https://github.com/ggerganov/llama.cpp.git`
-- Check llama.cpp documentation for model support
-
-### Quantization fails
-**Fix:**
-- Verify build tools installed: `apt-get install build-essential cmake`
-- Use CMake (not make) to build quantize tool
-- Check binary path: `/tmp/llama.cpp/build/bin/llama-quantize`
-- Verify FP16 GGUF exists before quantizing
-
-### Missing sentencepiece error
-**Fix:**
-- Add to PEP 723 header: `"sentencepiece>=0.1.99", "protobuf>=3.20.0"`
-- Don't remove dependencies to "simplify" - all are required
-
-### Upload fails or times out
-**Fix:**
-- Large models (>2GB) need longer timeout: `"timeout": "1h"`
-- Upload quantized versions separately if needed
-- Check network/Hub status
-
-## Lessons Learned
-
-These are from production testing and real failures:
-
-### 1. Always Verify Before Use
-**Lesson:** Don't assume repos/datasets exist. Check first.
-```python
-# BEFORE submitting job
-hub_repo_details(["trl-lib/argilla-dpo-mix-7k"], repo_type="dataset")  # Would catch error
-```
-**Prevented failures:** Non-existent dataset names, typos in model names
-
-### 2. Prioritize Reliability Over Performance
-**Lesson:** Default to what's most likely to succeed.
-- Use CMake (not make) - more reliable
-- Disable CUDA in build - faster, not needed
-- Include all dependencies - don't "simplify"
-
-**Prevented failures:** Build failures, missing binaries
-
-### 3. Create Atomic, Self-Contained Scripts
-**Lesson:** Don't remove dependencies or steps. Scripts should work as a unit.
-- All dependencies in PEP 723 header
-- All build steps included
-- Clear error messages
-
-**Prevented failures:** Missing tokenizer libraries, build tool failures
-
-## References
-
-**In this skill:**
-- `scripts/convert_to_gguf.py` - Complete, production-ready script
-
-**External:**
-- [llama.cpp Repository](https://github.com/ggerganov/llama.cpp)
-- [GGUF Specification](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md)
-- [Ollama Documentation](https://ollama.ai)
-- [LM Studio](https://lmstudio.ai)
-
-## Summary
-
-**Critical checklist for GGUF conversion:**
-- [ ] Verify adapter and base models exist on Hub
-- [ ] Use production script from `scripts/convert_to_gguf.py`
-- [ ] All dependencies in PEP 723 header (including sentencepiece, protobuf)
-- [ ] Build tools installed before cloning llama.cpp
-- [ ] CMake used for building quantize tool (not make)
-- [ ] Correct binary path: `/tmp/llama.cpp/build/bin/llama-quantize`
-- [ ] A10G GPU selected for reasonable conversion time
-- [ ] Timeout set to 45m minimum
-- [ ] HF_TOKEN in secrets for Hub upload
-
-**The script in `scripts/convert_to_gguf.py` incorporates all these lessons and has been tested successfully in production.**

package/skills_ref/hugging-face-model-trainer/references/hardware_guide.md

@@ -1,283 +0,0 @@
-# Hardware Selection Guide
-
-Choosing the right hardware (flavor) is critical for cost-effective training.
-
-## Available Hardware
-
-### CPU
-- `cpu-basic` - Basic CPU, testing only
-- `cpu-upgrade` - Enhanced CPU
-
-**Use cases:** Dataset validation, preprocessing, testing scripts
-**Not recommended for training:** Too slow for any meaningful training
-
-### GPU Options
-
-| Flavor | GPU | Memory | Use Case | Cost/hour |
-|--------|-----|--------|----------|-----------|
-| `t4-small` | NVIDIA T4 | 16GB | <1B models, demos | ~$0.50-1 |
-| `t4-medium` | NVIDIA T4 | 16GB | 1-3B models, development | ~$1-2 |
-| `l4x1` | NVIDIA L4 | 24GB | 3-7B models, efficient training | ~$2-3 |
-| `l4x4` | 4x NVIDIA L4 | 96GB | Multi-GPU training | ~$8-12 |
-| `a10g-small` | NVIDIA A10G | 24GB | 3-7B models, production | ~$3-4 |
-| `a10g-large` | NVIDIA A10G | 24GB | 7-13B models | ~$4-6 |
-| `a10g-largex2` | 2x NVIDIA A10G | 48GB | Multi-GPU, large models | ~$8-12 |
-| `a10g-largex4` | 4x NVIDIA A10G | 96GB | Multi-GPU, very large models | ~$16-24 |
-| `a100-large` | NVIDIA A100 | 40GB | 13B+ models, fast training | ~$8-12 |
-
-### TPU Options
-
-| Flavor | Type | Use Case |
-|--------|------|----------|
-| `v5e-1x1` | TPU v5e | Small TPU workloads |
-| `v5e-2x2` | 4x TPU v5e | Medium TPU workloads |
-| `v5e-2x4` | 8x TPU v5e | Large TPU workloads |
-
-**Note:** TPUs require TPU-optimized code. Most TRL training uses GPUs.
-
-## Selection Guidelines
-
-### By Model Size
-
-**Tiny Models (<1B parameters)**
-- **Recommended:** `t4-small`
-- **Example:** Qwen2.5-0.5B, TinyLlama
-- **Batch size:** 4-8
-- **Training time:** 1-2 hours for 1K examples
-
-**Small Models (1-3B parameters)**
-- **Recommended:** `t4-medium` or `a10g-small`
-- **Example:** Qwen2.5-1.5B, Phi-2
-- **Batch size:** 2-4
-- **Training time:** 2-4 hours for 10K examples
-
-**Medium Models (3-7B parameters)**
-- **Recommended:** `a10g-small` or `a10g-large`
-- **Example:** Qwen2.5-7B, Mistral-7B
-- **Batch size:** 1-2 (or LoRA with 4-8)
-- **Training time:** 4-8 hours for 10K examples
-
-**Large Models (7-13B parameters)**
-- **Recommended:** `a10g-large` or `a100-large`
-- **Example:** Llama-3-8B, Mixtral-8x7B (with LoRA)
-- **Batch size:** 1 (full fine-tuning) or 2-4 (LoRA)
-- **Training time:** 6-12 hours for 10K examples
-- **Note:** Always use LoRA/PEFT
-
-**Very Large Models (13B+ parameters)**
-- **Recommended:** `a100-large` with LoRA
-- **Example:** Llama-3-13B, Llama-3-70B (LoRA only)
-- **Batch size:** 1-2 with LoRA
-- **Training time:** 8-24 hours for 10K examples
-- **Note:** Full fine-tuning not feasible, use LoRA/PEFT
-
-### By Budget
-
-**Minimal Budget (<$5 total)**
-- Use `t4-small`
-- Train on subset of data (100-500 examples)
-- Limit to 1-2 epochs
-- Use small model (<1B)
-
-**Small Budget ($5-20)**
-- Use `t4-medium` or `a10g-small`
-- Train on 1K-5K examples
-- 2-3 epochs
-- Model up to 3B parameters
-
-**Medium Budget ($20-50)**
-- Use `a10g-small` or `a10g-large`
-- Train on 5K-20K examples
-- 3-5 epochs
-- Model up to 7B parameters
-
-**Large Budget ($50-200)**
-- Use `a10g-large` or `a100-large`
-- Full dataset training
-- Multiple epochs
-- Model up to 13B parameters with LoRA
-
-### By Training Type
-
-**Quick Demo/Experiment**
-- `t4-small`
-- 50-100 examples
-- 5-10 steps
-- ~10-15 minutes
-
-**Development/Iteration**
-- `t4-medium` or `a10g-small`
-- 1K examples
-- 1 epoch
-- ~30-60 minutes
-
-**Production Training**
-- `a10g-large` or `a100-large`
-- Full dataset
-- 3-5 epochs
-- 4-12 hours
-
-**Research/Experimentation**
-- `a100-large`
-- Multiple runs
-- Various hyperparameters
-- Budget for 20-50 hours
-
-## Memory Considerations
-
-### Estimating Memory Requirements
-
-**Full fine-tuning:**
-```
-Memory (GB) ≈ (Model params in billions) × 20
-```
-
-**LoRA fine-tuning:**
-```
-Memory (GB) ≈ (Model params in billions) × 4
-```
-
-**Examples:**
-- Qwen2.5-0.5B full: ~10GB ✅ fits t4-small
-- Qwen2.5-1.5B full: ~30GB ❌ exceeds most GPUs
-- Qwen2.5-1.5B LoRA: ~6GB ✅ fits t4-small
-- Qwen2.5-7B full: ~140GB ❌ not feasible
-- Qwen2.5-7B LoRA: ~28GB ✅ fits a10g-large
-
-### Memory Optimization
-
-If hitting memory limits:
-
-1. **Use LoRA/PEFT**
-   ```python
-   peft_config=LoraConfig(r=16, lora_alpha=32)
-   ```
-
-2. **Reduce batch size**
-   ```python
-   per_device_train_batch_size=1
-   ```
-
-3. **Increase gradient accumulation**
-   ```python
-   gradient_accumulation_steps=8  # Effective batch size = 1×8
-   ```
-
-4. **Enable gradient checkpointing**
-   ```python
-   gradient_checkpointing=True
-   ```
-
-5. **Use mixed precision**
-   ```python
-   bf16=True  # or fp16=True
-   ```
-
-6. **Upgrade to larger GPU**
-   - t4 → a10g → a100
-
-## Cost Estimation
-
-### Formula
-
-```
-Total Cost = (Hours of training) × (Cost per hour)
-```
-
-### Example Calculations
-
-**Quick demo:**
-- Hardware: t4-small ($0.75/hour)
-- Time: 15 minutes (0.25 hours)
-- Cost: $0.19
-
-**Development training:**
-- Hardware: a10g-small ($3.50/hour)
-- Time: 2 hours
-- Cost: $7.00
-
-**Production training:**
-- Hardware: a10g-large ($5/hour)
-- Time: 6 hours
-- Cost: $30.00
-
-**Large model with LoRA:**
-- Hardware: a100-large ($10/hour)
-- Time: 8 hours
-- Cost: $80.00
-
-### Cost Optimization Tips
-
-1. **Start small:** Test on t4-small with subset
-2. **Use LoRA:** 4-5x cheaper than full fine-tuning
-3. **Optimize hyperparameters:** Fewer epochs if possible
-4. **Set appropriate timeout:** Don't waste compute on stalled jobs
-5. **Use checkpointing:** Resume if job fails
-6. **Monitor costs:** Check running jobs regularly
-
-## Multi-GPU Training
-
-TRL automatically handles multi-GPU training with Accelerate when using multi-GPU flavors.
-
-**Multi-GPU flavors:**
-- `l4x4` - 4x L4 GPUs
-- `a10g-largex2` - 2x A10G GPUs
-- `a10g-largex4` - 4x A10G GPUs
-
-**When to use:**
-- Models >13B parameters
-- Need faster training (linear speedup)
-- Large datasets (>50K examples)
-
-**Example:**
-```python
-hf_jobs("uv", {
-    "script": "train.py",
-    "flavor": "a10g-largex2",  # 2 GPUs
-    "timeout": "4h",
-    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
-})
-```
-
-No code changes needed—TRL/Accelerate handles distribution automatically.
-
-## Choosing Between Options
-
-### a10g vs a100
-
-**Choose a10g when:**
-- Model <13B parameters
-- Budget conscious
-- Training time not critical
-
-**Choose a100 when:**
-- Model 13B+ parameters
-- Need fastest training
-- Memory requirements high
-- Budget allows
-
-### Single vs Multi-GPU
-
-**Choose single GPU when:**
-- Model <7B parameters
-- Budget constrained
-- Simpler debugging
-
-**Choose multi-GPU when:**
-- Model >13B parameters
-- Need faster training
-- Large batch sizes required
-- Cost-effective for large jobs
-
-## Quick Reference
-
-```python
-# Model size → Hardware selection
-HARDWARE_MAP = {
-    "<1B":     "t4-small",
-    "1-3B":    "a10g-small",
-    "3-7B":    "a10g-large",
-    "7-13B":   "a10g-large (LoRA) or a100-large",
-    ">13B":    "a100-large (LoRA required)"
-}
-```