@synsci/cli-darwin-x64 1.1.49

package/bin/skills/gptq/references/calibration.md

@@ -0,0 +1,337 @@
+# GPTQ Calibration Guide
+
+Complete guide to calibration data selection and quantization process.
+
+## Calibration Data Selection
+
+### Why calibration matters
+
+Calibration data is used to:
+1. **Compute weight importance** (Hessian matrix)
+2. **Minimize quantization error** for important weights
+3. **Preserve model accuracy** after quantization
+
+**Impact**:
+- Good calibration: <1.5% perplexity increase
+- Poor calibration: 5-10% perplexity increase
+- No calibration: Model may output gibberish
+
+### Dataset size
+
+**Recommended**:
+- **128-256 samples** of 512 tokens each
+- Total: 65K-131K tokens
+
+**More is not always better**:
+- <64 samples: Underfitting (poor quality)
+- 128-256 samples: Sweet spot
+- >512 samples: Diminishing returns, slower quantization
+
+### Dataset selection by domain
+
+**General purpose models (GPT, Llama)**:
+```python
+from datasets import load_dataset
+
+# C4 dataset (recommended for general models)
+dataset = load_dataset("c4", split="train", streaming=True)
+calibration_data = [
+    tokenizer(example["text"])["input_ids"][:512]
+    for example in dataset.take(128)
+]
+```
+
+**Code models (CodeLlama, StarCoder)**:
+```python
+# The Stack dataset
+dataset = load_dataset("bigcode/the-stack", split="train", streaming=True)
+calibration_data = [
+    tokenizer(example["content"])["input_ids"][:512]
+    for example in dataset.take(128)
+    if example["lang"] == "Python"  # Or your target language
+]
+```
+
+**Chat models**:
+```python
+# ShareGPT or Alpaca format
+dataset = load_dataset("anon8231489123/ShareGPT_Vicuna_unfiltered", split="train")
+
+calibration_data = []
+for example in dataset.select(range(128)):
+    # Format as conversation
+    conversation = tokenizer.apply_chat_template(
+        example["conversations"],
+        tokenize=True,
+        max_length=512
+    )
+    calibration_data.append(conversation)
+```
+
+**Domain-specific (medical, legal)**:
+```python
+# Use domain-specific text
+dataset = load_dataset("medical_dataset", split="train")
+calibration_data = [
+    tokenizer(example["text"])["input_ids"][:512]
+    for example in dataset.take(256)  # More samples for niche domains
+]
+```
+
+## Quantization Process
+
+### Basic quantization
+
+```python
+from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
+from transformers import AutoTokenizer
+from datasets import load_dataset
+
+# 1. Load model
+model_name = "meta-llama/Llama-2-7b-hf"
+model = AutoGPTQForCausalLM.from_pretrained(
+    model_name,
+    quantize_config=BaseQuantizeConfig(
+        bits=4,
+        group_size=128,
+        desc_act=False
+    )
+)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# 2. Prepare calibration data
+dataset = load_dataset("c4", split="train", streaming=True)
+calibration_data = [
+    tokenizer(example["text"])["input_ids"][:512]
+    for example in dataset.take(128)
+]
+
+# 3. Quantize
+model.quantize(calibration_data)
+
+# 4. Save
+model.save_quantized("llama-2-7b-gptq")
+```
+
+**Time**: ~10-30 minutes for 7B model on A100
+
+### Advanced configuration
+
+```python
+config = BaseQuantizeConfig(
+    bits=4,                    # 3, 4, or 8 bits
+    group_size=128,            # 32, 64, 128, or -1 (per-column)
+    desc_act=False,            # Activation order (True = better accuracy, slower)
+    damp_percent=0.01,         # Dampening (0.001-0.1, default 0.01)
+    static_groups=False,       # Static quantization
+    sym=True,                  # Symmetric quantization
+    true_sequential=True,      # Sequential quantization (more accurate)
+    model_seqlen=2048          # Model sequence length
+)
+```
+
+**Parameter tuning**:
+- `damp_percent`: Lower = more accurate, slower. Try 0.005-0.02.
+- `desc_act=True`: 0.5-1% better accuracy, 20-30% slower inference
+- `group_size=32`: Better accuracy, slightly larger model
+
+### Multi-GPU quantization
+
+```python
+# Quantize on multiple GPUs (faster)
+model = AutoGPTQForCausalLM.from_pretrained(
+    model_name,
+    quantize_config=config,
+    device_map="auto",         # Distribute across GPUs
+    max_memory={0: "40GB", 1: "40GB"}
+)
+
+model.quantize(calibration_data)
+```
+
+## Quality Evaluation
+
+### Perplexity testing
+
+```python
+from datasets import load_dataset
+import torch
+
+# Load test dataset
+test_dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
+test_text = "\n\n".join(test_dataset["text"])
+
+# Tokenize
+encodings = tokenizer(test_text, return_tensors="pt")
+max_length = model.seqlen
+
+# Calculate perplexity
+nlls = []
+for i in range(0, encodings.input_ids.size(1), max_length):
+    begin_loc = i
+    end_loc = min(i + max_length, encodings.input_ids.size(1))
+    input_ids = encodings.input_ids[:, begin_loc:end_loc].to("cuda")
+
+    with torch.no_grad():
+        outputs = model(input_ids, labels=input_ids)
+        nll = outputs.loss
+
+    nlls.append(nll)
+
+ppl = torch.exp(torch.stack(nlls).mean())
+print(f"Perplexity: {ppl.item():.2f}")
+```
+
+**Quality targets**:
+- <1.5% increase: Excellent
+- 1.5-3% increase: Good
+- 3-5% increase: Acceptable for some use cases
+- >5% increase: Poor, redo calibration
+
+### Benchmark evaluation
+
+```python
+from lm_eval import evaluator
+
+# Evaluate on standard benchmarks
+results = evaluator.simple_evaluate(
+    model=model,
+    tasks=["hellaswag", "mmlu", "arc_challenge"],
+    num_fewshot=5
+)
+
+print(results["results"])
+
+# Compare to baseline FP16 scores
+```
+
+## Optimization Tips
+
+### Improving accuracy
+
+**1. Use more calibration samples**:
+```python
+# Try 256 or 512 samples
+calibration_data = [... for example in dataset.take(256)]
+```
+
+**2. Use domain-specific data**:
+```python
+# Match your use case
+if code_model:
+    dataset = load_dataset("bigcode/the-stack")
+elif chat_model:
+    dataset = load_dataset("ShareGPT")
+```
+
+**3. Enable activation reordering**:
+```python
+config = BaseQuantizeConfig(
+    bits=4,
+    group_size=128,
+    desc_act=True  # Better accuracy, slower inference
+)
+```
+
+**4. Use smaller group size**:
+```python
+config = BaseQuantizeConfig(
+    bits=4,
+    group_size=32,  # vs 128
+    desc_act=False
+)
+```
+
+### Reducing quantization time
+
+**1. Use fewer samples**:
+```python
+# 64-128 samples usually sufficient
+calibration_data = [... for example in dataset.take(64)]
+```
+
+**2. Disable activation ordering**:
+```python
+config = BaseQuantizeConfig(
+    desc_act=False  # Faster quantization
+)
+```
+
+**3. Use multi-GPU**:
+```python
+model = AutoGPTQForCausalLM.from_pretrained(
+    model_name,
+    device_map="auto"  # Parallelize across GPUs
+)
+```
+
+## Troubleshooting
+
+### Poor quality after quantization
+
+**Symptom**: >5% perplexity increase or gibberish output
+
+**Solutions**:
+1. **Check calibration data**:
+   ```python
+   # Verify data is representative
+   for sample in calibration_data[:5]:
+       print(tokenizer.decode(sample))
+   ```
+
+2. **Try more samples**:
+   ```python
+   calibration_data = [... for example in dataset.take(256)]
+   ```
+
+3. **Use domain-specific data**:
+   ```python
+   # Match your model's use case
+   dataset = load_dataset("domain_specific_dataset")
+   ```
+
+4. **Adjust dampening**:
+   ```python
+   config = BaseQuantizeConfig(damp_percent=0.005)  # Lower dampening
+   ```
+
+### Quantization OOM
+
+**Solutions**:
+1. **Reduce batch size**:
+   ```python
+   model.quantize(calibration_data, batch_size=1)  # Default: auto
+   ```
+
+2. **Use CPU offloading**:
+   ```python
+   model = AutoGPTQForCausalLM.from_pretrained(
+       model_name,
+       device_map="auto",
+       max_memory={"cpu": "100GB"}
+   )
+   ```
+
+3. **Quantize on larger GPU** or use multi-GPU
+
+### Slow quantization
+
+**Typical times** (7B model):
+- Single A100: 10-15 minutes
+- Single RTX 4090: 20-30 minutes
+- CPU: 2-4 hours (not recommended)
+
+**Speedup**:
+- Use fewer samples (64 vs 256)
+- Disable `desc_act`
+- Use multi-GPU
+
+## Best Practices
+
+1. **Use C4 dataset for general models** - well-balanced, diverse
+2. **Match domain** - code models need code data, chat needs conversations
+3. **Start with 128 samples** - good balance of speed and quality
+4. **Test perplexity** - always verify quality before deployment
+5. **Compare kernels** - try ExLlama, Marlin, Triton for speed
+6. **Save multiple versions** - try group_size 32, 128, 256
+7. **Document settings** - save quantize_config.json for reproducibility

package/bin/skills/gptq/references/integration.md

@@ -0,0 +1,129 @@
+# GPTQ Integration Guide
+
+Integration with transformers, PEFT, vLLM, and other frameworks.
+
+## Transformers Integration
+
+### Auto-detection
+```python
+from transformers import AutoModelForCausalLM
+
+# Automatically detects and loads GPTQ model
+model = AutoModelForCausalLM.from_pretrained(
+    "TheBloke/Llama-2-13B-GPTQ",
+    device_map="auto"
+)
+```
+
+### Manual loading
+```python
+from auto_gptq import AutoGPTQForCausalLM
+
+model = AutoGPTQForCausalLM.from_quantized(
+    "TheBloke/Llama-2-13B-GPTQ",
+    device="cuda:0",
+    use_exllama=True
+)
+```
+
+## QLoRA Fine-Tuning
+
+```python
+from transformers import AutoModelForCausalLM, TrainingArguments
+from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model
+from trl import SFTTrainer
+
+# Load GPTQ model
+model = AutoModelForCausalLM.from_pretrained(
+    "TheBloke/Llama-2-70B-GPTQ",
+    device_map="auto"
+)
+
+# Prepare for training
+model = prepare_model_for_kbit_training(model)
+
+# LoRA config
+lora_config = LoraConfig(
+    r=16,
+    lora_alpha=32,
+    target_modules=["q_proj", "v_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM"
+)
+
+model = get_peft_model(model, lora_config)
+
+# Train (70B model on single A100!)
+trainer = SFTTrainer(
+    model=model,
+    train_dataset=dataset,
+    max_seq_length=2048,
+    args=TrainingArguments(
+        per_device_train_batch_size=1,
+        gradient_accumulation_steps=16,
+        learning_rate=2e-4,
+        num_train_epochs=3,
+        output_dir="./results"
+    )
+)
+
+trainer.train()
+```
+
+## vLLM Integration
+
+```python
+from vllm import LLM, SamplingParams
+
+# Load GPTQ model in vLLM
+llm = LLM(
+    model="TheBloke/Llama-2-70B-GPTQ",
+    quantization="gptq",
+    dtype="float16",
+    gpu_memory_utilization=0.95
+)
+
+# Generate
+sampling_params = SamplingParams(
+    temperature=0.7,
+    top_p=0.9,
+    max_tokens=200
+)
+
+outputs = llm.generate(["Explain AI"], sampling_params)
+```
+
+## Text Generation Inference (TGI)
+
+```bash
+# Docker with GPTQ support
+docker run --gpus all -p 8080:80 \
+    -v $PWD/data:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id TheBloke/Llama-2-70B-GPTQ \
+    --quantize gptq
+```
+
+## LangChain Integration
+
+```python
+from langchain.llms import HuggingFacePipeline
+from transformers import AutoTokenizer, pipeline
+
+tokenizer = AutoTokenizer.from_pretrained("TheBloke/Llama-2-13B-GPTQ")
+model = AutoModelForCausalLM.from_pretrained(
+    "TheBloke/Llama-2-13B-GPTQ",
+    device_map="auto"
+)
+
+pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=200)
+llm = HuggingFacePipeline(pipeline=pipe)
+
+# Use in LangChain
+from langchain.chains import LLMChain
+from langchain.prompts import PromptTemplate
+
+chain = LLMChain(llm=llm, prompt=PromptTemplate(...))
+result = chain.run(input="...")
+```

package/bin/skills/gptq/references/troubleshooting.md

@@ -0,0 +1,95 @@
+# GPTQ Troubleshooting Guide
+
+Common issues and solutions for GPTQ quantization and inference.
+
+## Installation Issues
+
+### CUDA mismatch
+```bash
+# Check CUDA version
+nvcc --version
+python -c "import torch; print(torch.version.cuda)"
+
+# Install matching version
+pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/  # CUDA 11.8
+```
+
+### Build errors
+```bash
+# Install build dependencies
+pip install auto-gptq --no-build-isolation
+
+# On Ubuntu
+sudo apt-get install python3-dev
+```
+
+## Runtime Issues
+
+### Slow inference
+```python
+# Try different backends
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    use_exllama=True  # Fastest (try v1 or v2)
+)
+
+# Or Marlin (Ampere+ GPUs)
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    use_marlin=True
+)
+```
+
+### OOM during inference
+```python
+# Reduce batch size
+outputs = model.generate(**inputs, batch_size=1)
+
+# Use CPU offloading
+model = AutoGPTQForCausalLM.from_quantized(
+    model_name,
+    device_map="auto",
+    max_memory={"cpu": "100GB"}
+)
+
+# Reduce context
+model.seqlen = 1024  # Instead of 2048
+```
+
+### Poor quality outputs
+```python
+# Requantize with better calibration
+# 1. Use more samples (256 instead of 128)
+# 2. Use domain-specific data
+# 3. Lower dampening: damp_percent=0.005
+# 4. Enable desc_act=True
+```
+
+## Quantization Issues
+
+### Very slow quantization
+```bash
+# Expected times (7B model):
+# - A100: 10-15 min
+# - RTX 4090: 20-30 min
+# - CPU: 2-4 hours
+
+# Speed up:
+# 1. Use GPU
+# 2. Reduce samples (64 instead of 256)
+# 3. Disable desc_act
+# 4. Use multi-GPU
+```
+
+### Quantization crashes
+```python
+# Reduce memory usage
+model = AutoGPTQForCausalLM.from_pretrained(
+    model_name,
+    device_map="auto",
+    max_memory={"cpu": "100GB"}  # Offload to CPU
+)
+
+# Or quantize layer-by-layer (slower but works)
+model.quantize(calibration_data, batch_size=1)
+```

package/bin/skills/grpo-rl-training/README.md

@@ -0,0 +1,97 @@
+# GRPO/RL Training Skill
+
+**Expert-level guidance for Group Relative Policy Optimization with TRL**
+
+## 📁 Skill Structure
+
+```
+grpo-rl-training/
+├── SKILL.md                              # Main skill documentation (READ THIS FIRST)
+├── README.md                             # This file
+├── templates/
+│   └── basic_grpo_training.py            # Production-ready training template
+└── examples/
+    └── reward_functions_library.py       # 20+ reward function examples
+```
+
+## 🚀 Quick Start
+
+1. **Read SKILL.md** - Comprehensive guide with all concepts and patterns
+2. **Copy `templates/basic_grpo_training.py`** - Start with working code
+3. **Browse `examples/reward_functions_library.py`** - Pick reward functions for your task
+4. **Modify for your use case** - Adapt dataset, rewards, and config
+
+## 💡 What's Inside
+
+### SKILL.md (Main Documentation)
+- Core GRPO concepts and algorithm fundamentals
+- Complete implementation workflow (dataset → rewards → training → deployment)
+- 10+ reward function examples with code
+- Hyperparameter tuning guide
+- Training insights (loss behavior, metrics, debugging)
+- Troubleshooting guide
+- Production best practices
+
+### Templates
+- **basic_grpo_training.py**: Minimal, production-ready training script
+  - Uses Qwen 2.5 1.5B Instruct
+  - 3 reward functions (format + correctness)
+  - LoRA for efficient training
+  - Fully documented and ready to run
+
+### Examples
+- **reward_functions_library.py**: 20+ battle-tested reward functions
+  - Correctness rewards (exact match, fuzzy match, numeric, code execution)
+  - Format rewards (XML, JSON, strict/soft)
+  - Length rewards (ideal length, min/max)
+  - Style rewards (reasoning quality, citations, repetition penalty)
+  - Combined rewards (multi-objective optimization)
+  - Preset collections for common tasks
+
+## 📖 Usage for Agents
+
+When this skill is loaded in your agent's context:
+
+1. **Always read SKILL.md first** before implementing
+2. **Start simple** - Use length-based reward to validate setup
+3. **Build incrementally** - Add one reward function at a time
+4. **Reference examples** - Copy patterns from reward_functions_library.py
+5. **Monitor training** - Watch reward metrics (not loss!)
+
+## 🎯 Common Use Cases
+
+| Task Type | Recommended Rewards | Template |
+|-----------|---------------------|----------|
+| Math reasoning | `MATH_REASONING_REWARDS` preset | basic_grpo_training.py |
+| Code generation | `CODE_GENERATION_REWARDS` preset | Modify dataset in template |
+| Summarization | `SUMMARIZATION_REWARDS` preset | Adjust prompts + rewards |
+| Q&A | `QA_REWARDS` preset | Use fuzzy match + citations |
+
+## ⚠️ Critical Reminders
+
+- **Loss goes UP during training** - This is normal (it's KL divergence)
+- **Use 3-5 reward functions** - Single rewards often fail
+- **Test rewards before training** - Debug each function independently
+- **Monitor reward_std** - Should stay > 0.1 (avoid mode collapse)
+- **Start with num_generations=4-8** - Scale up if GPU allows
+
+## 🔗 External Resources
+
+- [TRL Documentation](https://huggingface.co/docs/trl)
+- [DeepSeek R1 Paper](https://arxiv.org/abs/2501.12948)
+- [Open R1 Implementation](https://github.com/huggingface/open-r1)
+- [Unsloth (2-3x faster)](https://docs.unsloth.ai/)
+
+## 📝 Version
+
+**v1.0.0** - Initial release (January 2025)
+
+## 👨‍💻 Maintained By
+
+Synthetic Sciences
+For questions or improvements, see https://orchestra.com
+
+---
+
+**License:** MIT
+**Last Updated:** January 2025