@r1z4x/tezgah 1.0.0

package/skills/ai-research/10-optimization/hqq/references/troubleshooting.md

@@ -0,0 +1,504 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 10-optimization/hqq/references/troubleshooting.md; MIT (c) Orchestra Research -->
+# HQQ Troubleshooting Guide
+
+## Installation Issues
+
+### Package Not Found
+
+**Error**: `ModuleNotFoundError: No module named 'hqq'`
+
+**Fix**:
+```bash
+pip install hqq
+
+# Verify installation
+python -c "import hqq; print(hqq.__version__)"
+```
+
+### Backend Dependencies Missing
+
+**Error**: `ImportError: Cannot import marlin backend`
+
+**Fix**:
+```bash
+# Install specific backend
+pip install hqq[marlin]
+
+# Or all backends
+pip install hqq[all]
+
+# For BitBlas
+pip install bitblas
+
+# For TorchAO
+pip install torchao
+```
+
+### CUDA Version Mismatch
+
+**Error**: `RuntimeError: CUDA error: no kernel image is available`
+
+**Fix**:
+```bash
+# Check CUDA version
+nvcc --version
+python -c "import torch; print(torch.version.cuda)"
+
+# Reinstall PyTorch with matching CUDA
+pip install torch --index-url https://download.pytorch.org/whl/cu121
+
+# Then reinstall hqq
+pip install hqq --force-reinstall
+```
+
+## Quantization Errors
+
+### Out of Memory During Quantization
+
+**Error**: `torch.cuda.OutOfMemoryError`
+
+**Solutions**:
+
+1. **Use CPU offloading**:
+```python
+from transformers import AutoModelForCausalLM, HqqConfig
+
+model = AutoModelForCausalLM.from_pretrained(
+    "meta-llama/Llama-3.1-8B",
+    quantization_config=HqqConfig(nbits=4, group_size=64),
+    device_map="auto",
+    offload_folder="./offload"
+)
+```
+
+2. **Quantize layer by layer**:
+```python
+from hqq.models.hf.base import AutoHQQHFModel
+
+model = AutoHQQHFModel.from_pretrained(
+    "meta-llama/Llama-3.1-8B",
+    quantization_config=config,
+    device_map="sequential"
+)
+```
+
+3. **Reduce group size**:
+```python
+config = HqqConfig(
+    nbits=4,
+    group_size=32  # Smaller groups use less memory during quantization
+)
+```
+
+### NaN Values After Quantization
+
+**Error**: `RuntimeWarning: invalid value encountered` or NaN outputs
+
+**Solutions**:
+
+1. **Check for outliers**:
+```python
+import torch
+
+def check_weight_stats(model):
+    for name, param in model.named_parameters():
+        if param.numel() > 0:
+            has_nan = torch.isnan(param).any().item()
+            has_inf = torch.isinf(param).any().item()
+            if has_nan or has_inf:
+                print(f"{name}: NaN={has_nan}, Inf={has_inf}")
+                print(f"  min={param.min():.4f}, max={param.max():.4f}")
+
+check_weight_stats(model)
+```
+
+2. **Use higher precision for problematic layers**:
+```python
+layer_configs = {
+    "problematic_layer": BaseQuantizeConfig(nbits=8, group_size=128),
+    "default": BaseQuantizeConfig(nbits=4, group_size=64)
+}
+```
+
+3. **Skip embedding/lm_head**:
+```python
+config = HqqConfig(
+    nbits=4,
+    group_size=64,
+    skip_modules=["embed_tokens", "lm_head"]
+)
+```
+
+### Wrong Output Shape
+
+**Error**: `RuntimeError: shape mismatch`
+
+**Fix**:
+```python
+# Ensure axis is correct for your model
+config = BaseQuantizeConfig(
+    nbits=4,
+    group_size=64,
+    axis=1  # Usually 1 for most models, try 0 if issues
+)
+```
+
+## Backend Issues
+
+### Marlin Backend Not Working
+
+**Error**: `RuntimeError: Marlin kernel not available`
+
+**Requirements**:
+- Ampere (A100) or newer GPU (compute capability >= 8.0)
+- 4-bit quantization only
+- Group size must be 128
+
+**Fix**:
+```python
+# Check GPU compatibility
+import torch
+device = torch.cuda.get_device_properties(0)
+print(f"Compute capability: {device.major}.{device.minor}")
+
+# Marlin requires >= 8.0
+if device.major >= 8:
+    HQQLinear.set_backend("marlin")
+else:
+    HQQLinear.set_backend("aten")  # Fallback
+```
+
+### TorchAO Backend Errors
+
+**Error**: `ImportError: torchao not found`
+
+**Fix**:
+```bash
+pip install torchao
+
+# Verify
+python -c "import torchao; print('TorchAO installed')"
+```
+
+**Error**: `RuntimeError: torchao int4 requires specific shapes`
+
+**Fix**:
+```python
+# TorchAO int4 has shape requirements
+# Ensure dimensions are divisible by 32
+config = BaseQuantizeConfig(
+    nbits=4,
+    group_size=64  # Must be power of 2
+)
+```
+
+### Fallback to PyTorch Backend
+
+```python
+from hqq.core.quantize import HQQLinear
+
+def safe_set_backend(preferred_backend):
+    """Set backend with fallback."""
+    try:
+        HQQLinear.set_backend(preferred_backend)
+        print(f"Using {preferred_backend} backend")
+    except Exception as e:
+        print(f"Failed to set {preferred_backend}: {e}")
+        print("Falling back to pytorch backend")
+        HQQLinear.set_backend("pytorch")
+
+safe_set_backend("marlin")
+```
+
+## Performance Issues
+
+### Slow Inference
+
+**Problem**: Inference slower than expected
+
+**Solutions**:
+
+1. **Use optimized backend**:
+```python
+from hqq.core.quantize import HQQLinear
+
+# Try backends in order of speed
+for backend in ["marlin", "torchao_int4", "aten", "pytorch_compile"]:
+    try:
+        HQQLinear.set_backend(backend)
+        print(f"Using {backend}")
+        break
+    except:
+        continue
+```
+
+2. **Enable torch.compile**:
+```python
+import torch
+model = torch.compile(model, mode="reduce-overhead")
+```
+
+3. **Use CUDA graphs** (for fixed input shapes):
+```python
+# Warmup
+for _ in range(3):
+    model.generate(**inputs, max_new_tokens=100)
+
+# Enable CUDA graphs
+torch.cuda.synchronize()
+```
+
+### High Memory Usage During Inference
+
+**Problem**: Memory usage higher than expected for quantized model
+
+**Solutions**:
+
+1. **Clear KV cache**:
+```python
+# Use past_key_values management
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=100,
+    use_cache=True,
+    return_dict_in_generate=True
+)
+# Clear after use
+del outputs.past_key_values
+torch.cuda.empty_cache()
+```
+
+2. **Reduce batch size**:
+```python
+# Process in smaller batches
+batch_size = 4  # Reduce if OOM
+for i in range(0, len(prompts), batch_size):
+    batch = prompts[i:i+batch_size]
+    outputs = model.generate(...)
+    torch.cuda.empty_cache()
+```
+
+3. **Use gradient checkpointing** (for training):
+```python
+model.gradient_checkpointing_enable()
+```
+
+## Quality Issues
+
+### Poor Generation Quality
+
+**Problem**: Quantized model produces gibberish or low-quality output
+
+**Solutions**:
+
+1. **Increase precision**:
+```python
+# Try higher bit-width
+config = HqqConfig(nbits=8, group_size=128)  # Start high
+# Then gradually reduce: 8 -> 4 -> 3 -> 2
+```
+
+2. **Use smaller group size**:
+```python
+config = HqqConfig(
+    nbits=4,
+    group_size=32  # Smaller = more accurate, more memory
+)
+```
+
+3. **Skip sensitive layers**:
+```python
+config = HqqConfig(
+    nbits=4,
+    group_size=64,
+    skip_modules=["embed_tokens", "lm_head", "model.layers.0"]
+)
+```
+
+4. **Compare outputs**:
+```python
+def compare_outputs(original_model, quantized_model, prompt):
+    """Compare outputs between original and quantized."""
+    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+
+    with torch.no_grad():
+        orig_out = original_model.generate(**inputs, max_new_tokens=50)
+        quant_out = quantized_model.generate(**inputs, max_new_tokens=50)
+
+    print("Original:", tokenizer.decode(orig_out[0]))
+    print("Quantized:", tokenizer.decode(quant_out[0]))
+```
+
+### Perplexity Degradation
+
+**Problem**: Significant perplexity increase after quantization
+
+**Diagnosis**:
+```python
+import torch
+from datasets import load_dataset
+
+def measure_perplexity(model, tokenizer, dataset_name="wikitext", split="test"):
+    """Measure model perplexity."""
+    dataset = load_dataset(dataset_name, "wikitext-2-raw-v1", split=split)
+    text = "\n\n".join(dataset["text"])
+
+    encodings = tokenizer(text, return_tensors="pt")
+    max_length = 2048
+    stride = 512
+
+    nlls = []
+    for i in range(0, encodings.input_ids.size(1), stride):
+        begin = max(i + stride - max_length, 0)
+        end = min(i + stride, encodings.input_ids.size(1))
+
+        input_ids = encodings.input_ids[:, begin:end].to(model.device)
+        target_ids = input_ids.clone()
+        target_ids[:, :-stride] = -100
+
+        with torch.no_grad():
+            outputs = model(input_ids, labels=target_ids)
+            nlls.append(outputs.loss)
+
+    ppl = torch.exp(torch.stack(nlls).mean())
+    return ppl.item()
+
+# Compare
+orig_ppl = measure_perplexity(original_model, tokenizer)
+quant_ppl = measure_perplexity(quantized_model, tokenizer)
+print(f"Original PPL: {orig_ppl:.2f}")
+print(f"Quantized PPL: {quant_ppl:.2f}")
+print(f"Degradation: {((quant_ppl - orig_ppl) / orig_ppl * 100):.1f}%")
+```
+
+## Integration Issues
+
+### HuggingFace Integration Errors
+
+**Error**: `ValueError: Unknown quantization method: hqq`
+
+**Fix**:
+```bash
+# Update transformers
+pip install -U transformers>=4.36.0
+```
+
+**Error**: `AttributeError: 'HqqConfig' object has no attribute`
+
+**Fix**:
+```python
+from transformers import HqqConfig
+
+# Use correct parameter names
+config = HqqConfig(
+    nbits=4,           # Not 'bits'
+    group_size=64,     # Not 'groupsize'
+    axis=1             # Not 'quant_axis'
+)
+```
+
+### vLLM Integration Issues
+
+**Error**: `ValueError: HQQ quantization not supported`
+
+**Fix**:
+```bash
+# Update vLLM
+pip install -U vllm>=0.3.0
+```
+
+**Usage**:
+```python
+from vllm import LLM
+
+# Load pre-quantized model
+llm = LLM(
+    model="mobiuslabsgmbh/Llama-3.1-8B-HQQ-4bit",
+    quantization="hqq"
+)
+```
+
+### PEFT Integration Issues
+
+**Error**: `RuntimeError: Cannot apply LoRA to quantized layer`
+
+**Fix**:
+```python
+from peft import prepare_model_for_kbit_training
+
+# Prepare model for training
+model = prepare_model_for_kbit_training(model)
+
+# Then apply LoRA
+model = get_peft_model(model, lora_config)
+```
+
+## Debugging Tips
+
+### Enable Verbose Logging
+
+```python
+import logging
+
+logging.basicConfig(level=logging.DEBUG)
+logging.getLogger("hqq").setLevel(logging.DEBUG)
+```
+
+### Verify Quantization Applied
+
+```python
+def verify_quantization(model):
+    """Check if model is properly quantized."""
+    from hqq.core.quantize import HQQLinear
+
+    total_linear = 0
+    quantized_linear = 0
+
+    for name, module in model.named_modules():
+        if isinstance(module, torch.nn.Linear):
+            total_linear += 1
+        elif isinstance(module, HQQLinear):
+            quantized_linear += 1
+            print(f"Quantized: {name} ({module.W_q.dtype}, {module.W_q.shape})")
+
+    print(f"\nTotal Linear: {total_linear}")
+    print(f"Quantized: {quantized_linear}")
+    print(f"Ratio: {quantized_linear / max(total_linear + quantized_linear, 1) * 100:.1f}%")
+
+verify_quantization(model)
+```
+
+### Memory Profiling
+
+```python
+import torch
+
+def profile_memory():
+    """Profile GPU memory usage."""
+    print(f"Allocated: {torch.cuda.memory_allocated() / 1024**3:.2f} GB")
+    print(f"Reserved: {torch.cuda.memory_reserved() / 1024**3:.2f} GB")
+    print(f"Max Allocated: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB")
+
+# Before quantization
+profile_memory()
+
+# After quantization
+model = load_quantized_model(...)
+profile_memory()
+```
+
+## Getting Help
+
+1. **GitHub Issues**: https://github.com/mobiusml/hqq/issues
+2. **HuggingFace Forums**: https://discuss.huggingface.co
+3. **Discord**: Check HQQ community channels
+
+### Reporting Issues
+
+Include:
+- HQQ version: `pip show hqq`
+- PyTorch version: `python -c "import torch; print(torch.__version__)"`
+- CUDA version: `nvcc --version`
+- GPU model: `nvidia-smi --query-gpu=name --format=csv`
+- Full error traceback
+- Minimal reproducible code