EvoScientist 0.0.1.dev4__py3-none-any.whl → 0.1.0rc1__py3-none-any.whl

EvoScientist/skills/bitsandbytes/references/quantization-formats.md

@@ -1,447 +0,0 @@
-# Quantization Formats
-
-Complete guide to INT8, NF4, FP4 quantization formats, double quantization, and custom configurations in bitsandbytes.
-
-## Overview
-
-bitsandbytes supports multiple quantization formats:
-- **INT8**: 8-bit integer quantization (LLM.int8())
-- **NF4**: 4-bit NormalFloat (for normally distributed weights)
-- **FP4**: 4-bit FloatPoint (for uniformly distributed weights)
-- **Double Quantization**: Quantize the quantization constants
-
-## INT8 Quantization
-
-### LLM.int8() Algorithm
-
-LLM.int8() uses mixed 8-bit/16-bit matrix multiplication:
-- Most features (>99.9%) computed in INT8
-- Outlier features (>threshold) computed in FP16
-- Results combined for final output
-
-**Memory**: 50% reduction (2 bytes → 1 byte per parameter)
-**Accuracy**: <0.5% degradation
-
-### Configuration
-
-```python
-from transformers import BitsAndBytesConfig
-
-config = BitsAndBytesConfig(
-    load_in_8bit=True,
-    llm_int8_threshold=6.0,  # Outlier threshold
-    llm_int8_has_fp16_weight=False,  # Use INT8 storage
-    llm_int8_skip_modules=["lm_head"]  # Skip certain layers
-)
-```
-
-### Parameters Explained
-
-**`llm_int8_threshold`** (default: 6.0):
-- Activations with magnitude > threshold are kept in FP16
-- Lower = more FP16 (slower but more accurate)
-- Higher = more INT8 (faster but less accurate)
-
-```python
-# Conservative (more accurate)
-llm_int8_threshold=5.0
-
-# Aggressive (faster)
-llm_int8_threshold=8.0
-```
-
-**`llm_int8_has_fp16_weight`** (default: False):
-- `False`: Store weights in INT8 (50% memory savings)
-- `True`: Store in FP16, quantize only during computation (no memory savings)
-
-**`llm_int8_skip_modules`**:
-```python
-# Skip specific layers (keep in FP16)
-llm_int8_skip_modules=["lm_head", "embed_tokens"]
-```
-
-### Example
-
-```python
-from transformers import AutoModelForCausalLM
-
-model = AutoModelForCausalLM.from_pretrained(
-    "meta-llama/Llama-2-13b-hf",
-    quantization_config=config,
-    device_map="auto"
-)
-
-# Memory: 26GB (FP16) → 13GB (INT8)
-```
-
-### When to Use INT8
-
-✅ **Use INT8 when**:
-- Need high accuracy (<0.5% loss)
-- Model fits with 50% reduction
-- Have Turing+ GPU (tensor cores)
-
-❌ **Don't use when**:
-- Need maximum memory savings (use 4-bit)
-- Inference speed critical (use GPTQ/AWQ)
-
-## 4-Bit Quantization
-
-### NormalFloat4 (NF4)
-
-Optimized for normally distributed weights (most neural networks).
-
-**How it works**:
-- Bins chosen to minimize quantization error for normal distribution
-- Asymmetric quantization bins
-- Better for transformer weights
-
-**Configuration**:
-```python
-config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-    bnb_4bit_quant_type="nf4"  # NormalFloat4
-)
-```
-
-**Memory**: 75% reduction (2 bytes → 0.5 bytes per parameter)
-
-### FloatPoint4 (FP4)
-
-Standard 4-bit floating point for uniform distributions.
-
-**How it works**:
-- Symmetric quantization bins
-- Better for weights with broader dynamic range
-- Less common for transformers
-
-**Configuration**:
-```python
-config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-    bnb_4bit_quant_type="fp4"  # FloatPoint4
-)
-```
-
-### NF4 vs FP4 Comparison
-
-| Aspect | NF4 | FP4 |
-|--------|-----|-----|
-| Distribution | Normal | Uniform |
-| Typical use | **Transformers** | CNNs, unusual architectures |
-| Accuracy | **Better for LLMs** | Worse for LLMs |
-| Speed | Same | Same |
-| Recommendation | ✅ Default | Use only if NF4 fails |
-
-**Rule of thumb**: Always use NF4 for transformers.
-
-### Example Comparison
-
-```python
-# NF4 (recommended)
-nf4_config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_quant_type="nf4"
-)
-
-# FP4 (alternative)
-fp4_config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_quant_type="fp4"
-)
-
-# Load and compare
-model_nf4 = AutoModelForCausalLM.from_pretrained(
-    "meta-llama/Llama-2-7b-hf",
-    quantization_config=nf4_config
-)
-
-model_fp4 = AutoModelForCausalLM.from_pretrained(
-    "meta-llama/Llama-2-7b-hf",
-    quantization_config=fp4_config
-)
-
-# Typical results on MMLU:
-# NF4: 45.2%
-# FP4: 43.8%
-# FP16: 45.9%
-```
-
-## Compute Dtype
-
-The `bnb_4bit_compute_dtype` controls the precision used for actual computation.
-
-### Options
-
-**torch.bfloat16** (recommended):
-```python
-bnb_4bit_compute_dtype=torch.bfloat16
-```
-- Good balance of speed and accuracy
-- Recommended for A100/H100
-- Prevents numerical instability
-
-**torch.float16**:
-```python
-bnb_4bit_compute_dtype=torch.float16
-```
-- Slightly faster than BF16
-- Risk of overflow/underflow
-- Use only if BF16 unavailable
-
-**torch.float32**:
-```python
-bnb_4bit_compute_dtype=torch.float32
-```
-- Most accurate
-- Slowest (no tensor core acceleration)
-- Debugging only
-
-### Performance Comparison
-
-| Dtype | Speed | Accuracy | Memory |
-|-------|-------|----------|--------|
-| FP32 | 1× (baseline) | 100% | 4 bytes |
-| FP16 | 3-4× | 99.5% | 2 bytes |
-| BF16 | 3-4× | **99.8%** | 2 bytes |
-
-**Recommendation**: Always use `torch.bfloat16` if supported.
-
-## Double Quantization
-
-Quantize the quantization constants for additional memory savings.
-
-### How It Works
-
-Standard 4-bit quantization stores:
-- 4-bit quantized weights
-- FP32 scaling factors (4 bytes per block)
-
-Double quantization:
-- 4-bit quantized weights
-- **INT8 quantized scaling factors** (1 byte per block)
-
-**Additional savings**: ~2-3% memory reduction
-
-### Configuration
-
-```python
-config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_use_double_quant=True  # Enable double quantization
-)
-```
-
-### Example
-
-```python
-# Without double quant
-model_single = AutoModelForCausalLM.from_pretrained(
-    "meta-llama/Llama-2-70b-hf",
-    quantization_config=BitsAndBytesConfig(
-        load_in_4bit=True,
-        bnb_4bit_use_double_quant=False
-    )
-)
-# Memory: ~36GB
-
-# With double quant
-model_double = AutoModelForCausalLM.from_pretrained(
-    "meta-llama/Llama-2-70b-hf",
-    quantization_config=BitsAndBytesConfig(
-        load_in_4bit=True,
-        bnb_4bit_use_double_quant=True
-    )
-)
-# Memory: ~35GB (saves ~1GB)
-```
-
-**Accuracy impact**: Negligible (<0.1%)
-
-**Recommendation**: Always enable for maximum memory savings.
-
-## Quantization Storage
-
-Controls storage dtype for quantized weights (important for FSDP).
-
-### Configuration
-
-```python
-config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_quant_storage=torch.bfloat16  # Storage dtype
-)
-```
-
-### When to Use
-
-**Default (uint8)**:
-- Single GPU training/inference
-- No special requirements
-
-**torch.bfloat16** (for FSDP):
-```python
-bnb_4bit_quant_storage=torch.bfloat16
-```
-- **Required for FSDP+QLoRA**
-- Ensures 4-bit layers wrapped like regular layers
-- Enables proper model sharding
-
-### Example: FSDP Configuration
-
-```python
-# CRITICAL: Set quant_storage for FSDP
-fsdp_config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_use_double_quant=True,
-    bnb_4bit_quant_storage=torch.bfloat16  # Must match torch_dtype!
-)
-
-model = AutoModelForCausalLM.from_pretrained(
-    "meta-llama/Llama-2-70b-hf",
-    quantization_config=fsdp_config,
-    torch_dtype=torch.bfloat16  # Must match quant_storage!
-)
-```
-
-## Recommended Configurations
-
-### Production Inference (Best Accuracy)
-
-```python
-BitsAndBytesConfig(
-    load_in_8bit=True,
-    llm_int8_threshold=6.0
-)
-```
-
-**Use case**: Maximum accuracy with 50% memory savings
-
-### Production Inference (Maximum Memory Savings)
-
-```python
-BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_use_double_quant=True
-)
-```
-
-**Use case**: 75% memory reduction with <1% accuracy loss
-
-### QLoRA Training (Single GPU)
-
-```python
-BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_use_double_quant=True
-)
-```
-
-**Use case**: Fine-tune 70B on RTX 3090
-
-### FSDP + QLoRA (Multi-GPU)
-
-```python
-BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_use_double_quant=True,
-    bnb_4bit_quant_storage=torch.bfloat16  # CRITICAL!
-)
-```
-
-**Use case**: Fine-tune 405B on 8×H100
-
-## Advanced: Block-wise Quantization
-
-bitsandbytes uses block-wise quantization:
-- Weights divided into blocks (typically 64 or 128 elements)
-- Each block has own scaling factor
-- Better accuracy than tensor-wise quantization
-
-**Block size** (automatically determined):
-```python
-# Typical block sizes
-# 4-bit: 64 elements per block
-# 8-bit: 64 elements per block
-```
-
-**Cannot be configured** (internal implementation detail).
-
-## Quantization Quality Metrics
-
-### Perplexity (Lower is Better)
-
-| Model | FP16 | INT8 | NF4 | NF4+DQ |
-|-------|------|------|-----|--------|
-| Llama 2 7B | 5.12 | 5.14 | 5.18 | 5.19 |
-| Llama 2 13B | 4.88 | 4.90 | 4.93 | 4.94 |
-| Llama 2 70B | 3.32 | 3.33 | 3.35 | 3.36 |
-
-**Conclusion**: <1% degradation for all quantization methods
-
-### MMLU Accuracy (Higher is Better)
-
-| Model | FP16 | INT8 | NF4 | FP4 |
-|-------|------|------|-----|-----|
-| Llama 2 7B | 45.9% | 45.7% | 45.2% | 43.8% |
-| Llama 2 13B | 54.8% | 54.6% | 54.1% | 52.9% |
-| Llama 2 70B | 68.9% | 68.7% | 68.4% | 67.2% |
-
-**Conclusion**: NF4 is significantly better than FP4 for transformers
-
-## Troubleshooting
-
-### "Quantization failed" Error
-
-Try different quant type:
-```python
-# If NF4 fails
-bnb_4bit_quant_type="fp4"
-```
-
-### Numerical Instability
-
-Use BF16 compute:
-```python
-bnb_4bit_compute_dtype=torch.bfloat16
-```
-
-### Poor Quality with 4-bit
-
-1. Try 8-bit instead:
-   ```python
-   load_in_8bit=True
-   ```
-
-2. Enable double quantization:
-   ```python
-   bnb_4bit_use_double_quant=True
-   ```
-
-3. Use BF16 compute dtype
-
-### FSDP Errors
-
-Ensure quant_storage matches torch_dtype:
-```python
-bnb_4bit_quant_storage=torch.bfloat16
-torch_dtype=torch.bfloat16  # Must match!
-```
-
-## References
-
-- LLM.int8() paper: "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (2022)
-- QLoRA paper: "QLoRA: Efficient Finetuning of Quantized LLMs" (2023)
-- bitsandbytes GitHub: https://github.com/bitsandbytes-foundation/bitsandbytes
-- HuggingFace quantization docs: https://huggingface.co/docs/transformers/quantization/bitsandbytes

EvoScientist/skills/find-skills/SKILL.md

@@ -1,133 +0,0 @@
----
-name: find-skills
-description: Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", or express interest in extending capabilities. Uses a non-interactive installer script suitable for automated agents.
----
-
-# Find Skills
-
-This skill helps you discover and install skills from the open agent skills ecosystem.
-
-## When to Use This Skill
-
-Use this skill when the user:
-
-- Asks "how do I do X" where X might be a common task with an existing skill
-- Says "find a skill for X" or "is there a skill for X"
-- Wants to search for tools, templates, or workflows
-- Expresses interest in extending agent capabilities
-- Mentions they wish they had help with a specific domain (design, testing, deployment, etc.)
-
-## Step 1: Search for Skills
-
-Use `npx -y skills find` with a relevant keyword to search the ecosystem:
-
-```bash
-npx -y skills find [query]
-```
-
-Examples:
-- User asks "help me with React performance" → `npx -y skills find react performance`
-- User asks "is there a skill for PR reviews?" → `npx -y skills find pr review`
-- User asks "I need to create a changelog" → `npx -y skills find changelog`
-
-The search results will show installable skills like:
-
-```
-vercel-labs/agent-skills@vercel-react-best-practices
-└ https://skills.sh/vercel-labs/agent-skills/vercel-react-best-practices
-```
-
-Browse all available skills at: https://skills.sh/
-
-## Step 2: Present Options
-
-When you find relevant skills, present them to the user with:
-1. The skill name and what it does
-2. A link to learn more on skills.sh
-
-Ask the user which skill(s) they want to install. All skills are installed to `./skills/` in the current working directory.
-
-## Step 3: Install with the Script
-
-**IMPORTANT: Do NOT use `npx -y skills add` for installation** — it requires interactive prompts.
-
-Use the bundled installer script instead:
-
-```bash
-python /skills/find-skills/scripts/install_skill.py --url <github_url>
-```
-
-### Install Commands
-
-**From a GitHub URL** (most common — copy the URL from search results):
-```bash
-python /skills/find-skills/scripts/install_skill.py \
-  --url https://github.com/owner/repo/tree/main/skill-name
-```
-
-**From skills.sh shorthand** (owner/repo@skill):
-```bash
-python /skills/find-skills/scripts/install_skill.py \
-  --url vercel-labs/agent-skills@vercel-react-best-practices
-```
-
-**From repo + path** (install specific skills from a multi-skill repo):
-```bash
-# Single skill
-python /skills/find-skills/scripts/install_skill.py \
-  --repo owner/repo --path skill-name
-
-# Multiple skills from same repo
-python /skills/find-skills/scripts/install_skill.py \
-  --repo owner/repo --path skill-a --path skill-b
-```
-
-**With a specific git branch or tag**:
-```bash
-python /skills/find-skills/scripts/install_skill.py \
-  --repo owner/repo --path skill-name --ref v2.0
-```
-
-### Installer Options
-
-| Option | Description |
-|--------|-------------|
-| `--url` | GitHub URL or owner/repo@skill shorthand |
-| `--repo` | GitHub repo (owner/repo format) |
-| `--path` | Path to skill inside repo (repeatable) |
-| `--ref` | Git branch or tag |
-| `--dest` | Custom destination directory (default: `./skills`) |
-
-## Step 4: Confirm Installation
-
-After installation, verify by listing the skills directory:
-
-```bash
-ls /skills/   # all skills (system + user merged)
-```
-
-Then read the installed skill's SKILL.md to confirm it loaded correctly:
-
-```bash
-read_file /skills/<skill-name>/SKILL.md
-```
-
-## Common Skill Categories
-
-| Category | Example Queries |
-|----------|----------------|
-| Web Development | react, nextjs, typescript, css, tailwind |
-| Testing | testing, jest, playwright, e2e |
-| DevOps | deploy, docker, kubernetes, ci-cd |
-| Documentation | docs, readme, changelog, api-docs |
-| Code Quality | review, lint, refactor, best-practices |
-| Design | ui, ux, design-system, accessibility |
-| Productivity | workflow, automation, git |
-
-## When No Skills Are Found
-
-If no relevant skills exist:
-
-1. Acknowledge that no existing skill was found
-2. Offer to help with the task directly using your general capabilities
-3. Mention the user could create their own skill with `npx -y skills init`