@synsci/cli-darwin-x64 1.1.49

package/bin/skills/gptq/SKILL.md

@@ -0,0 +1,450 @@
+---
+name: gptq
+description: Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Optimization, GPTQ, Quantization, 4-Bit, Post-Training, Memory Optimization, Consumer GPUs, Fast Inference, QLoRA, Group-Wise Quantization]
+dependencies: [auto-gptq, transformers, optimum, peft]
+---
+
+# GPTQ (Generative Pre-trained Transformer Quantization)
+
+Post-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.
+
+## When to use GPTQ
+
+**Use GPTQ when:**
+- Need to fit large models (70B+) on limited GPU memory
+- Want 4× memory reduction with <2% accuracy loss
+- Deploying on consumer GPUs (RTX 4090, 3090)
+- Need faster inference (3-4× speedup vs FP16)
+
+**Use AWQ instead when:**
+- Need slightly better accuracy (<1% loss)
+- Have newer GPUs (Ampere, Ada)
+- Want Marlin kernel support (2× faster on some GPUs)
+
+**Use bitsandbytes instead when:**
+- Need simple integration with transformers
+- Want 8-bit quantization (less compression, better quality)
+- Don't need pre-quantized model files
+
+## Quick start
+
+### Installation
+
+```bash
+# Install AutoGPTQ
+pip install auto-gptq
+
+# With Triton (Linux only, faster)
+pip install auto-gptq[triton]
+
+# With CUDA extensions (faster)
+pip install auto-gptq --no-build-isolation
+
+# Full installation
+pip install auto-gptq transformers accelerate
+```
+
+### Load pre-quantized model
+
+```python
+from transformers import AutoTokenizer
+from auto_gptq import AutoGPTQForCausalLM
+
+# Load quantized model from HuggingFace
+model_name = "TheBloke/Llama-2-7B-Chat-GPTQ"
+
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    device="cuda:0",
+    use_triton=False  # Set True on Linux for speed
+)
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# Generate
+prompt = "Explain quantum computing"
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
+outputs = model.generate(**inputs, max_new_tokens=200)
+print(tokenizer.decode(outputs[0]))
+```
+
+### Quantize your own model
+
+```python
+from transformers import AutoTokenizer
+from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
+from datasets import load_dataset
+
+# Load model
+model_name = "meta-llama/Llama-2-7b-chat-hf"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# Quantization config
+quantize_config = BaseQuantizeConfig(
+    bits=4,              # 4-bit quantization
+    group_size=128,      # Group size (recommended: 128)
+    desc_act=False,      # Activation order (False for CUDA kernel)
+    damp_percent=0.01    # Dampening factor
+)
+
+# Load model for quantization
+model = AutoGPTQForCausalLM.from_pretrained(
+    model_name,
+    quantize_config=quantize_config
+)
+
+# Prepare calibration data
+dataset = load_dataset("c4", split="train", streaming=True)
+calibration_data = [
+    tokenizer(example["text"])["input_ids"][:512]
+    for example in dataset.take(128)
+]
+
+# Quantize
+model.quantize(calibration_data)
+
+# Save quantized model
+model.save_quantized("llama-2-7b-gptq")
+tokenizer.save_pretrained("llama-2-7b-gptq")
+
+# Push to HuggingFace
+model.push_to_hub("username/llama-2-7b-gptq")
+```
+
+## Group-wise quantization
+
+**How GPTQ works**:
+1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)
+2. **Quantize per-group**: Each group has its own scale/zero-point
+3. **Minimize error**: Uses Hessian information to minimize quantization error
+4. **Result**: 4-bit weights with near-FP16 accuracy
+
+**Group size trade-off**:
+
+| Group Size | Model Size | Accuracy | Speed | Recommendation |
+|------------|------------|----------|-------|----------------|
+| -1 (per-column) | Smallest | Best | Slowest | Research only |
+| 32 | Smaller | Better | Slower | High accuracy needed |
+| **128** | Medium | Good | **Fast** | **Recommended default** |
+| 256 | Larger | Lower | Faster | Speed critical |
+| 1024 | Largest | Lowest | Fastest | Not recommended |
+
+**Example**:
+```
+Weight matrix: [1024, 4096] = 4.2M elements
+
+Group size = 128:
+- Groups: 4.2M / 128 = 32,768 groups
+- Each group: own 4-bit scale + zero-point
+- Result: Better granularity → better accuracy
+```
+
+## Quantization configurations
+
+### Standard 4-bit (recommended)
+
+```python
+from auto_gptq import BaseQuantizeConfig
+
+config = BaseQuantizeConfig(
+    bits=4,              # 4-bit quantization
+    group_size=128,      # Standard group size
+    desc_act=False,      # Faster CUDA kernel
+    damp_percent=0.01    # Dampening factor
+)
+```
+
+**Performance**:
+- Memory: 4× reduction (70B model: 140GB → 35GB)
+- Accuracy: ~1.5% perplexity increase
+- Speed: 3-4× faster than FP16
+
+### High accuracy (3-bit with larger groups)
+
+```python
+config = BaseQuantizeConfig(
+    bits=3,              # 3-bit (more compression)
+    group_size=128,      # Keep standard group size
+    desc_act=True,       # Better accuracy (slower)
+    damp_percent=0.01
+)
+```
+
+**Trade-off**:
+- Memory: 5× reduction
+- Accuracy: ~3% perplexity increase
+- Speed: 5× faster (but less accurate)
+
+### Maximum accuracy (4-bit with small groups)
+
+```python
+config = BaseQuantizeConfig(
+    bits=4,
+    group_size=32,       # Smaller groups (better accuracy)
+    desc_act=True,       # Activation reordering
+    damp_percent=0.005   # Lower dampening
+)
+```
+
+**Trade-off**:
+- Memory: 3.5× reduction (slightly larger)
+- Accuracy: ~0.8% perplexity increase (best)
+- Speed: 2-3× faster (kernel overhead)
+
+## Kernel backends
+
+### ExLlamaV2 (default, fastest)
+
+```python
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    device="cuda:0",
+    use_exllama=True,      # Use ExLlamaV2
+    exllama_config={"version": 2}
+)
+```
+
+**Performance**: 1.5-2× faster than Triton
+
+### Marlin (Ampere+ GPUs)
+
+```python
+# Quantize with Marlin format
+config = BaseQuantizeConfig(
+    bits=4,
+    group_size=128,
+    desc_act=False  # Required for Marlin
+)
+
+model.quantize(calibration_data, use_marlin=True)
+
+# Load with Marlin
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    device="cuda:0",
+    use_marlin=True  # 2× faster on A100/H100
+)
+```
+
+**Requirements**:
+- NVIDIA Ampere or newer (A100, H100, RTX 40xx)
+- Compute capability ≥ 8.0
+
+### Triton (Linux only)
+
+```python
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    device="cuda:0",
+    use_triton=True  # Linux only
+)
+```
+
+**Performance**: 1.2-1.5× faster than CUDA backend
+
+## Integration with transformers
+
+### Direct transformers usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Load quantized model (transformers auto-detects GPTQ)
+model = AutoModelForCausalLM.from_pretrained(
+    "TheBloke/Llama-2-13B-Chat-GPTQ",
+    device_map="auto",
+    trust_remote_code=False
+)
+
+tokenizer = AutoTokenizer.from_pretrained("TheBloke/Llama-2-13B-Chat-GPTQ")
+
+# Use like any transformers model
+inputs = tokenizer("Hello", return_tensors="pt").to("cuda")
+outputs = model.generate(**inputs, max_new_tokens=100)
+```
+
+### QLoRA fine-tuning (GPTQ + LoRA)
+
+```python
+from transformers import AutoModelForCausalLM
+from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model
+
+# Load GPTQ model
+model = AutoModelForCausalLM.from_pretrained(
+    "TheBloke/Llama-2-7B-GPTQ",
+    device_map="auto"
+)
+
+# Prepare for LoRA training
+model = prepare_model_for_kbit_training(model)
+
+# LoRA config
+lora_config = LoraConfig(
+    r=16,
+    lora_alpha=32,
+    target_modules=["q_proj", "v_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM"
+)
+
+# Add LoRA adapters
+model = get_peft_model(model, lora_config)
+
+# Fine-tune (memory efficient!)
+# 70B model trainable on single A100 80GB
+```
+
+## Performance benchmarks
+
+### Memory reduction
+
+| Model | FP16 | GPTQ 4-bit | Reduction |
+|-------|------|------------|-----------|
+| Llama 2-7B | 14 GB | 3.5 GB | 4× |
+| Llama 2-13B | 26 GB | 6.5 GB | 4× |
+| Llama 2-70B | 140 GB | 35 GB | 4× |
+| Llama 3-405B | 810 GB | 203 GB | 4× |
+
+**Enables**:
+- 70B on single A100 80GB (vs 2× A100 needed for FP16)
+- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)
+- 13B on RTX 4090 24GB (vs OOM with FP16)
+
+### Inference speed (Llama 2-7B, A100)
+
+| Precision | Tokens/sec | vs FP16 |
+|-----------|------------|---------|
+| FP16 | 25 tok/s | 1× |
+| GPTQ 4-bit (CUDA) | 85 tok/s | 3.4× |
+| GPTQ 4-bit (ExLlama) | 105 tok/s | 4.2× |
+| GPTQ 4-bit (Marlin) | 120 tok/s | 4.8× |
+
+### Accuracy (perplexity on WikiText-2)
+
+| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |
+|-------|------|---------------------|-------------|
+| Llama 2-7B | 5.47 | 5.55 | +1.5% |
+| Llama 2-13B | 4.88 | 4.95 | +1.4% |
+| Llama 2-70B | 3.32 | 3.38 | +1.8% |
+
+**Excellent quality preservation** - less than 2% degradation!
+
+## Common patterns
+
+### Multi-GPU deployment
+
+```python
+# Automatic device mapping
+model = AutoGPTQForCausalLM.from_quantized(
+    "TheBloke/Llama-2-70B-GPTQ",
+    device_map="auto",  # Automatically split across GPUs
+    max_memory={0: "40GB", 1: "40GB"}  # Limit per GPU
+)
+
+# Manual device mapping
+device_map = {
+    "model.embed_tokens": 0,
+    "model.layers.0-39": 0,  # First 40 layers on GPU 0
+    "model.layers.40-79": 1,  # Last 40 layers on GPU 1
+    "model.norm": 1,
+    "lm_head": 1
+}
+
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    device_map=device_map
+)
+```
+
+### CPU offloading
+
+```python
+# Offload some layers to CPU (for very large models)
+model = AutoGPTQForCausalLM.from_quantized(
+    "TheBloke/Llama-2-405B-GPTQ",
+    device_map="auto",
+    max_memory={
+        0: "80GB",  # GPU 0
+        1: "80GB",  # GPU 1
+        2: "80GB",  # GPU 2
+        "cpu": "200GB"  # Offload overflow to CPU
+    }
+)
+```
+
+### Batch inference
+
+```python
+# Process multiple prompts efficiently
+prompts = [
+    "Explain AI",
+    "Explain ML",
+    "Explain DL"
+]
+
+inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=100,
+    pad_token_id=tokenizer.eos_token_id
+)
+
+for i, output in enumerate(outputs):
+    print(f"Prompt {i}: {tokenizer.decode(output)}")
+```
+
+## Finding pre-quantized models
+
+**TheBloke on HuggingFace**:
+- https://huggingface.co/TheBloke
+- 1000+ models in GPTQ format
+- Multiple group sizes (32, 128)
+- Both CUDA and Marlin formats
+
+**Search**:
+```bash
+# Find GPTQ models on HuggingFace
+https://huggingface.co/models?library=gptq
+```
+
+**Download**:
+```python
+from auto_gptq import AutoGPTQForCausalLM
+
+# Automatically downloads from HuggingFace
+model = AutoGPTQForCausalLM.from_quantized(
+    "TheBloke/Llama-2-70B-Chat-GPTQ",
+    device="cuda:0"
+)
+```
+
+## Supported models
+
+- **LLaMA family**: Llama 2, Llama 3, Code Llama
+- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B
+- **Qwen**: Qwen, Qwen2, QwQ
+- **DeepSeek**: V2, V3
+- **Phi**: Phi-2, Phi-3
+- **Yi, Falcon, BLOOM, OPT**
+- **100+ models** on HuggingFace
+
+## References
+
+- **[Calibration Guide](references/calibration.md)** - Dataset selection, quantization process, quality optimization
+- **[Integration Guide](references/integration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM
+- **[Troubleshooting](references/troubleshooting.md)** - Common issues, performance optimization
+
+## Resources
+
+- **GitHub**: https://github.com/AutoGPTQ/AutoGPTQ
+- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)
+- **Models**: https://huggingface.co/models?library=gptq
+- **Discord**: https://discord.gg/autogptq
+
+