@r1z4x/tezgah 1.0.0

package/skills/ai-research/10-optimization/awq/references/troubleshooting.md

@@ -0,0 +1,345 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 10-optimization/awq/references/troubleshooting.md; MIT (c) Orchestra Research -->
+# AWQ Troubleshooting Guide
+
+## Installation Issues
+
+### CUDA Version Mismatch
+
+**Error**: `RuntimeError: CUDA error: no kernel image is available for execution`
+
+**Fix**: Install matching CUDA version:
+```bash
+# Check your CUDA version
+nvcc --version
+
+# Install matching autoawq
+pip install autoawq --extra-index-url https://download.pytorch.org/whl/cu118  # For CUDA 11.8
+pip install autoawq --extra-index-url https://download.pytorch.org/whl/cu121  # For CUDA 12.1
+```
+
+### Compute Capability Too Low
+
+**Error**: `AssertionError: Compute capability must be >= 7.5`
+
+**Fix**: AWQ requires NVIDIA GPUs with compute capability 7.5+ (Turing or newer):
+- RTX 20xx series: 7.5 (supported)
+- RTX 30xx series: 8.6 (supported)
+- RTX 40xx series: 8.9 (supported)
+- A100/H100: 8.0/9.0 (supported)
+
+Older GPUs (GTX 10xx, V100) are not supported.
+
+### Transformers Version Conflict
+
+**Error**: `ImportError: cannot import name 'AwqConfig'`
+
+**Fix**: AutoAWQ may downgrade transformers. Reinstall correct version:
+```bash
+pip install autoawq
+pip install transformers>=4.45.0 --upgrade
+```
+
+### Triton Not Found (Linux)
+
+**Error**: `ModuleNotFoundError: No module named 'triton'`
+
+**Fix**:
+```bash
+pip install triton
+# Or install with kernels
+pip install autoawq[kernels]
+```
+
+## Quantization Issues
+
+### CUDA Out of Memory During Quantization
+
+**Error**: `torch.cuda.OutOfMemoryError: CUDA out of memory`
+
+**Solutions**:
+
+1. **Reduce calibration samples**:
+```python
+model.quantize(
+    tokenizer,
+    quant_config=quant_config,
+    max_calib_samples=64  # Reduce from 128
+)
+```
+
+2. **Use CPU offloading**:
+```python
+model = AutoAWQForCausalLM.from_pretrained(
+    model_path,
+    low_cpu_mem_usage=True
+)
+```
+
+3. **Multi-GPU quantization**:
+```python
+model = AutoAWQForCausalLM.from_pretrained(
+    model_path,
+    device_map="auto"
+)
+```
+
+### NaN in Weights After Quantization
+
+**Error**: `AssertionError: NaN detected in weights`
+
+**Cause**: Calibration data issues or numerical instability.
+
+**Fix**:
+```python
+# Use more calibration samples
+model.quantize(
+    tokenizer,
+    quant_config=quant_config,
+    max_calib_samples=256,
+    max_calib_seq_len=1024
+)
+```
+
+### Empty Calibration Samples
+
+**Error**: `ValueError: Calibration samples are empty`
+
+**Fix**: Ensure tokenizer produces valid output:
+```python
+# Check tokenizer
+test = tokenizer("test", return_tensors="pt")
+print(f"Token count: {test.input_ids.shape[1]}")
+
+# Use explicit calibration data
+calib_data = ["Your sample text here..."] * 128
+model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)
+```
+
+### Unsupported Model Architecture
+
+**Error**: `TypeError: 'model_type' is not supported`
+
+**Cause**: Model architecture not in AWQ registry.
+
+**Check supported models**:
+```python
+from awq.models import AWQ_CAUSAL_LM_MODEL_MAP
+print(list(AWQ_CAUSAL_LM_MODEL_MAP.keys()))
+```
+
+**Supported**: llama, mistral, qwen2, falcon, mpt, phi, gemma, etc.
+
+## Inference Issues
+
+### Slow Inference Speed
+
+**Problem**: Inference slower than expected.
+
+**Solutions**:
+
+1. **Enable layer fusion**:
+```python
+model = AutoAWQForCausalLM.from_quantized(
+    model_name,
+    fuse_layers=True
+)
+```
+
+2. **Use correct kernel for batch size**:
+```python
+# For batch_size=1
+quant_config = {"version": "GEMV"}
+
+# For batch_size>1
+quant_config = {"version": "GEMM"}
+```
+
+3. **Use Marlin on Ampere+ GPUs**:
+```python
+from transformers import AwqConfig
+config = AwqConfig(bits=4, version="marlin")
+```
+
+### Wrong Output / Garbage Text
+
+**Problem**: Model produces nonsensical output after quantization.
+
+**Causes and fixes**:
+
+1. **Poor calibration data**: Use domain-relevant data
+```python
+calib_data = [
+    "Relevant examples from your use case...",
+]
+model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)
+```
+
+2. **Tokenizer mismatch**: Ensure same tokenizer
+```python
+tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
+```
+
+3. **Check generation config**:
+```python
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=200,
+    do_sample=True,
+    temperature=0.7,
+    pad_token_id=tokenizer.eos_token_id
+)
+```
+
+### FlashAttention2 Incompatibility
+
+**Error**: `ValueError: Cannot use FlashAttention2 with fused modules`
+
+**Fix**: Disable one or the other:
+```python
+# Option 1: Use fused modules (recommended for AWQ)
+model = AutoAWQForCausalLM.from_quantized(model_name, fuse_layers=True)
+
+# Option 2: Use FlashAttention2 without fusion
+from transformers import AutoModelForCausalLM
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    attn_implementation="flash_attention_2",
+    device_map="auto"
+)
+```
+
+### AMD GPU Issues
+
+**Error**: `RuntimeError: ROCm/HIP not found`
+
+**Fix**: Use ExLlama backend for AMD:
+```python
+from transformers import AwqConfig
+
+config = AwqConfig(bits=4, version="exllama")
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    quantization_config=config
+)
+```
+
+## Loading Issues
+
+### Model Not Found
+
+**Error**: `OSError: model_name is not a valid model identifier`
+
+**Fix**: Check HuggingFace model exists:
+```bash
+# Search AWQ models
+https://huggingface.co/models?library=awq
+
+# Common AWQ model providers
+TheBloke, teknium, Qwen, NousResearch
+```
+
+### Safetensors Error
+
+**Error**: `safetensors_rust.SafetensorError: Error while deserializing`
+
+**Fix**: Try loading without safetensors:
+```python
+model = AutoAWQForCausalLM.from_quantized(
+    model_name,
+    safetensors=False
+)
+```
+
+### Device Map Conflicts
+
+**Error**: `ValueError: You cannot use device_map with max_memory`
+
+**Fix**: Use one or the other:
+```python
+# Auto device map
+model = AutoAWQForCausalLM.from_quantized(model_name, device_map="auto")
+
+# OR manual memory limits
+model = AutoAWQForCausalLM.from_quantized(
+    model_name,
+    max_memory={0: "20GB", 1: "20GB"}
+)
+```
+
+## vLLM Integration Issues
+
+### Quantization Not Detected
+
+**Error**: vLLM loads model in FP16 instead of quantized.
+
+**Fix**: Explicitly specify quantization:
+```python
+from vllm import LLM
+
+llm = LLM(
+    model="TheBloke/Llama-2-7B-AWQ",
+    quantization="awq",  # Explicitly set
+    dtype="half"
+)
+```
+
+### Marlin Kernel Error in vLLM
+
+**Error**: `RuntimeError: Marlin kernel not supported`
+
+**Fix**: Check GPU compatibility:
+```python
+import torch
+print(torch.cuda.get_device_capability())  # Must be >= (8, 0)
+
+# If not supported, use GEMM
+llm = LLM(model="...", quantization="awq")  # Uses GEMM by default
+```
+
+## Performance Debugging
+
+### Memory Usage Check
+
+```python
+import torch
+
+def print_gpu_memory():
+    for i in range(torch.cuda.device_count()):
+        allocated = torch.cuda.memory_allocated(i) / 1e9
+        reserved = torch.cuda.memory_reserved(i) / 1e9
+        print(f"GPU {i}: {allocated:.2f}GB allocated, {reserved:.2f}GB reserved")
+
+print_gpu_memory()
+```
+
+### Profiling Inference
+
+```python
+import time
+
+def benchmark_model(model, tokenizer, prompt, n_runs=5):
+    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+
+    # Warmup
+    model.generate(**inputs, max_new_tokens=10)
+    torch.cuda.synchronize()
+
+    # Benchmark
+    times = []
+    for _ in range(n_runs):
+        start = time.perf_counter()
+        outputs = model.generate(**inputs, max_new_tokens=100)
+        torch.cuda.synchronize()
+        times.append(time.perf_counter() - start)
+
+    tokens = outputs.shape[1] - inputs.input_ids.shape[1]
+    avg_time = sum(times) / len(times)
+    print(f"Average: {tokens/avg_time:.2f} tokens/sec")
+```
+
+## Getting Help
+
+1. **Check deprecation notice**: AutoAWQ is deprecated, use llm-compressor for new projects
+2. **GitHub Issues**: https://github.com/casper-hansen/AutoAWQ/issues
+3. **HuggingFace Forums**: https://discuss.huggingface.co/
+4. **vLLM Discord**: For vLLM integration issues