@r1z4x/tezgah 1.0.0

package/skills/ai-research/18-multimodal/blip-2/references/troubleshooting.md

@@ -0,0 +1,527 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 18-multimodal/blip-2/references/troubleshooting.md; MIT (c) Orchestra Research -->
+# BLIP-2 Troubleshooting Guide
+
+## Installation Issues
+
+### Import errors
+
+**Error**: `ModuleNotFoundError: No module named 'transformers'`
+
+**Solutions**:
+```bash
+# Install transformers with vision support
+pip install transformers[vision] accelerate
+
+# Or install all optional dependencies
+pip install transformers accelerate torch Pillow scipy
+
+# Verify installation
+python -c "from transformers import Blip2ForConditionalGeneration; print('OK')"
+```
+
+### LAVIS installation fails
+
+**Error**: Errors installing salesforce-lavis
+
+**Solutions**:
+```bash
+# Install from source
+git clone https://github.com/salesforce/LAVIS.git
+cd LAVIS
+pip install -e .
+
+# Or specific version
+pip install salesforce-lavis==1.0.2
+
+# Install dependencies separately if issues persist
+pip install omegaconf iopath timm webdataset
+pip install salesforce-lavis --no-deps
+```
+
+### CUDA version mismatch
+
+**Error**: `RuntimeError: CUDA error: no kernel image is available`
+
+**Solutions**:
+```bash
+# Check CUDA version
+nvcc --version
+python -c "import torch; print(torch.version.cuda)"
+
+# Install matching PyTorch
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
+
+# For CUDA 11.8
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
+```
+
+## Model Loading Issues
+
+### Out of memory during load
+
+**Error**: `torch.cuda.OutOfMemoryError` during model loading
+
+**Solutions**:
+```python
+# Use quantization
+from transformers import BitsAndBytesConfig
+
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
+model = Blip2ForConditionalGeneration.from_pretrained(
+    "Salesforce/blip2-opt-2.7b",
+    quantization_config=quantization_config,
+    device_map="auto"
+)
+
+# Or 4-bit quantization
+quantization_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_compute_dtype=torch.float16
+)
+
+# Use smaller model
+model = Blip2ForConditionalGeneration.from_pretrained(
+    "Salesforce/blip2-opt-2.7b",  # Instead of 6.7b or flan-t5-xxl
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+
+# Offload to CPU
+model = Blip2ForConditionalGeneration.from_pretrained(
+    "Salesforce/blip2-opt-6.7b",
+    device_map="auto",
+    offload_folder="offload"
+)
+```
+
+### Model download fails
+
+**Error**: Connection errors or incomplete downloads
+
+**Solutions**:
+```python
+# Set cache directory
+import os
+os.environ["HF_HOME"] = "/path/to/cache"
+
+# Resume download
+from huggingface_hub import snapshot_download
+snapshot_download(
+    "Salesforce/blip2-opt-2.7b",
+    resume_download=True
+)
+
+# Use local files only after download
+model = Blip2ForConditionalGeneration.from_pretrained(
+    "Salesforce/blip2-opt-2.7b",
+    local_files_only=True
+)
+```
+
+### Weight loading errors
+
+**Error**: `RuntimeError: Error(s) in loading state_dict`
+
+**Solutions**:
+```python
+# Ignore mismatched weights
+model = Blip2ForConditionalGeneration.from_pretrained(
+    "Salesforce/blip2-opt-2.7b",
+    ignore_mismatched_sizes=True
+)
+
+# Check model architecture matches checkpoint
+from transformers import AutoConfig
+config = AutoConfig.from_pretrained("Salesforce/blip2-opt-2.7b")
+print(config.text_config.model_type)  # Should be 'opt'
+```
+
+## Inference Issues
+
+### Image format errors
+
+**Error**: `ValueError: Unable to create tensor`
+
+**Solutions**:
+```python
+from PIL import Image
+
+# Ensure RGB format
+image = Image.open("image.jpg").convert("RGB")
+
+# Handle different formats
+def load_image(path):
+    image = Image.open(path)
+
+    # Convert RGBA to RGB
+    if image.mode == "RGBA":
+        background = Image.new("RGB", image.size, (255, 255, 255))
+        background.paste(image, mask=image.split()[3])
+        image = background
+    elif image.mode != "RGB":
+        image = image.convert("RGB")
+
+    return image
+
+# Handle URL images
+import requests
+from io import BytesIO
+
+def load_image_from_url(url):
+    response = requests.get(url)
+    image = Image.open(BytesIO(response.content))
+    return image.convert("RGB")
+```
+
+### Empty or nonsensical output
+
+**Problem**: Model returns empty string or gibberish
+
+**Solutions**:
+```python
+# Check input preprocessing
+inputs = processor(images=image, return_tensors="pt")
+print(f"Pixel values shape: {inputs['pixel_values'].shape}")
+# Should be [1, 3, 224, 224] for single image
+
+# Ensure correct dtype
+inputs = inputs.to("cuda", torch.float16)
+
+# Use better generation parameters
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=100,
+    min_length=10,
+    num_beams=5,
+    do_sample=False  # Deterministic for debugging
+)
+
+# Check decoder starting tokens
+print(f"Generated IDs: {generated_ids}")
+```
+
+### Slow generation
+
+**Problem**: Generation takes too long
+
+**Solutions**:
+```python
+# Reduce max_new_tokens
+generated_ids = model.generate(**inputs, max_new_tokens=30)
+
+# Use greedy decoding (faster than beam search)
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=50,
+    num_beams=1,
+    do_sample=False
+)
+
+# Enable model compilation (PyTorch 2.0+)
+model = torch.compile(model)
+
+# Use Flash Attention
+model = Blip2ForConditionalGeneration.from_pretrained(
+    "Salesforce/blip2-opt-2.7b",
+    torch_dtype=torch.float16,
+    attn_implementation="flash_attention_2",
+    device_map="auto"
+)
+```
+
+### Batch processing errors
+
+**Error**: Dimension mismatch in batch processing
+
+**Solutions**:
+```python
+# Ensure consistent image sizes with padding
+inputs = processor(
+    images=images,
+    return_tensors="pt",
+    padding=True
+)
+
+# Handle variable size images
+from torchvision import transforms
+
+transform = transforms.Compose([
+    transforms.Resize((224, 224)),
+    transforms.ToTensor(),
+])
+
+# Ensure all images are same size before processing
+images = [transform(img) for img in images]
+
+# For text inputs, use padding
+inputs = processor(
+    images=images,
+    text=questions,
+    return_tensors="pt",
+    padding="max_length",
+    max_length=32,
+    truncation=True
+)
+```
+
+## Memory Issues
+
+### CUDA out of memory
+
+**Error**: `torch.cuda.OutOfMemoryError: CUDA out of memory`
+
+**Solutions**:
+```python
+# Clear cache before inference
+torch.cuda.empty_cache()
+
+# Use smaller batch size
+batch_size = 1  # Start with 1
+
+# Process sequentially
+results = []
+for image in images:
+    inputs = processor(images=image, return_tensors="pt").to("cuda", torch.float16)
+    generated_ids = model.generate(**inputs, max_new_tokens=50)
+    results.append(processor.decode(generated_ids[0], skip_special_tokens=True))
+    torch.cuda.empty_cache()
+
+# Use gradient checkpointing
+model.gradient_checkpointing_enable()
+
+# Monitor memory
+print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
+print(f"Cached: {torch.cuda.memory_reserved() / 1e9:.2f} GB")
+```
+
+### Memory leak during batch processing
+
+**Problem**: Memory grows over time
+
+**Solutions**:
+```python
+import gc
+
+# Delete tensors explicitly
+del inputs, generated_ids
+gc.collect()
+torch.cuda.empty_cache()
+
+# Use context manager
+with torch.inference_mode():
+    inputs = processor(images=image, return_tensors="pt").to("cuda", torch.float16)
+    generated_ids = model.generate(**inputs, max_new_tokens=50)
+    caption = processor.decode(generated_ids[0], skip_special_tokens=True)
+
+# Move to CPU after inference
+caption = processor.decode(generated_ids.cpu()[0], skip_special_tokens=True)
+```
+
+## Quality Issues
+
+### Poor caption quality
+
+**Problem**: Captions are generic or inaccurate
+
+**Solutions**:
+```python
+# Use larger model
+model = Blip2ForConditionalGeneration.from_pretrained(
+    "Salesforce/blip2-flan-t5-xl",  # Better quality than OPT
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+
+# Use prompts for better captions
+inputs = processor(
+    images=image,
+    text="a detailed description of the image:",
+    return_tensors="pt"
+)
+
+# Increase diversity with sampling
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=100,
+    num_beams=5,
+    num_return_sequences=3,  # Generate multiple
+    temperature=0.9,
+    do_sample=True
+)
+
+# Select best from multiple candidates
+```
+
+### VQA hallucinations
+
+**Problem**: Model makes up information not in image
+
+**Solutions**:
+```python
+# Use more specific questions
+# Instead of "What is happening?"
+# Ask "Is there a person in this image?"
+
+# Lower temperature
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=30,
+    temperature=0.3,  # More focused
+    do_sample=True
+)
+
+# Use beam search (more deterministic)
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=30,
+    num_beams=5,
+    do_sample=False
+)
+
+# Add constraints
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=30,
+    no_repeat_ngram_size=3,
+)
+```
+
+### Incorrect colors/objects
+
+**Problem**: Model identifies wrong colors or objects
+
+**Solutions**:
+```python
+# Ensure image is RGB not BGR
+import cv2
+image_cv = cv2.imread("image.jpg")
+image_rgb = cv2.cvtColor(image_cv, cv2.COLOR_BGR2RGB)
+image = Image.fromarray(image_rgb)
+
+# Check image quality
+print(f"Image size: {image.size}")
+print(f"Image mode: {image.mode}")
+
+# Use higher resolution if possible (but processor resizes to 224x224)
+
+# Ask more specific questions
+# Instead of "What color is it?"
+# Ask "Is the car red or blue?"
+```
+
+## Processor Issues
+
+### Tokenizer warnings
+
+**Warning**: `Asking to pad but the tokenizer does not have a padding token`
+
+**Solutions**:
+```python
+# Set padding token
+processor.tokenizer.pad_token = processor.tokenizer.eos_token
+
+# Or specify during processing
+inputs = processor(
+    images=image,
+    text=question,
+    return_tensors="pt",
+    padding="max_length",
+    max_length=32
+)
+```
+
+### Image normalization issues
+
+**Problem**: Unexpected results due to normalization
+
+**Solutions**:
+```python
+# Check processor's image normalization
+print(processor.image_processor.image_mean)
+print(processor.image_processor.image_std)
+
+# Manual normalization if needed
+from torchvision import transforms
+
+normalize = transforms.Normalize(
+    mean=processor.image_processor.image_mean,
+    std=processor.image_processor.image_std
+)
+
+# Or use raw pixel values
+inputs = processor(
+    images=image,
+    return_tensors="pt",
+    do_normalize=False  # Skip normalization
+)
+```
+
+## LAVIS-Specific Issues
+
+### Config not found
+
+**Error**: `ConfigError: Config file not found`
+
+**Solutions**:
+```python
+# Use registry properly
+from lavis.common.registry import registry
+from lavis.models import load_model_and_preprocess
+
+# Check available models
+print(registry.list_models())
+
+# Load with explicit config
+model, vis_processors, txt_processors = load_model_and_preprocess(
+    name="blip2_opt",
+    model_type="pretrain_opt2.7b",
+    is_eval=True,
+    device="cuda"
+)
+```
+
+### Dataset loading errors
+
+**Error**: `Dataset not found` or download issues
+
+**Solutions**:
+```python
+from lavis.datasets.builders import load_dataset
+
+# Set download directory
+import os
+os.environ["LAVIS_DATASETS_ROOT"] = "/path/to/datasets"
+
+# Download manually first
+# Then load with local files
+dataset = load_dataset("coco_caption", split="val")
+```
+
+## Common Error Messages
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `CUDA out of memory` | Model too large | Use quantization or smaller model |
+| `Unable to create tensor` | Invalid image format | Convert to RGB PIL Image |
+| `padding_side must be` | Tokenizer config | Set pad_token explicitly |
+| `Expected 4D input` | Wrong tensor shape | Add batch dimension with unsqueeze(0) |
+| `device mismatch` | Tensors on different devices | Move all to same device |
+| `half() not implemented` | CPU doesn't support FP16 | Use float32 on CPU |
+
+## Getting Help
+
+1. **HuggingFace Forums**: https://discuss.huggingface.co
+2. **LAVIS GitHub Issues**: https://github.com/salesforce/LAVIS/issues
+3. **Paper**: https://arxiv.org/abs/2301.12597
+4. **Model Card**: https://huggingface.co/Salesforce/blip2-opt-2.7b
+
+### Reporting Issues
+
+Include:
+- Python version
+- transformers/lavis version
+- PyTorch and CUDA versions
+- GPU model and VRAM
+- Full error traceback
+- Minimal reproducible code
+- Image resolution and format