opencode-skills-collection 4.0.68 → 4.0.70

package/bundled-skills/llm-fine-tuning/SKILL.md

@@ -0,0 +1,329 @@
+---
+name: llm-fine-tuning
+description: Set up infrastructure for fine-tuning LLMs with QLoRA, LoRA, and full
+  fine-tuning using Hugging Face TRL, Axolotl, and distributed training with DeepSpeed
+  or FSDP.
+category: devops
+risk: critical
+source: https://github.com/BagelHole/DevOps-Security-Agent-Skills
+source_repo: BagelHole/DevOps-Security-Agent-Skills
+source_type: community
+date_added: '2026-09-20'
+license: MIT
+license_source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/main/LICENSE
+compatibility: Requires the relevant OS/platform tooling and privileged access where
+  noted. Docs-only; helper scripts and templates not bundled.
+metadata:
+  author: devops-skills
+  version: '1.0'
+---
+
+# LLM Fine-Tuning Infrastructure
+
+Train and fine-tune open-source LLMs efficiently — from LoRA on a single GPU to distributed full fine-tuning across multi-node clusters.
+
+## When to Use This Skill
+
+Use this skill when:
+- Fine-tuning an LLM on domain-specific data (legal, medical, code, support)
+- Running QLoRA to fine-tune 70B models on consumer GPUs
+- Setting up distributed training with DeepSpeed or FSDP
+- Exporting fine-tuned adapters for production serving
+- Implementing RLHF, DPO, or instruction tuning pipelines
+
+## Prerequisites
+
+- NVIDIA GPU(s) with 24GB+ VRAM (RTX 4090 / A100 / H100)
+- CUDA 12.1+ and `nvidia-smi` working
+- Python 3.10+ with `pip`
+- Hugging Face account and `HF_TOKEN` for gated models
+- 500GB+ disk for model weights and training data
+
+## Quick Start: QLoRA Fine-Tuning
+
+```bash
+pip install transformers datasets trl peft bitsandbytes accelerate
+
+python - <<'EOF'
+from datasets import load_dataset
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+from peft import LoraConfig, get_peft_model
+from trl import SFTTrainer, SFTConfig
+import torch
+
+model_id = "meta-llama/Llama-3.1-8B-Instruct"
+
+# 4-bit quantization (QLoRA)
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16,
+    bnb_4bit_use_double_quant=True,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, quantization_config=bnb_config, device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+# LoRA configuration
+peft_config = LoraConfig(
+    r=16,                    # rank
+    lora_alpha=32,
+    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
+                    "gate_proj", "up_proj", "down_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+
+dataset = load_dataset("your-org/your-dataset", split="train")
+
+trainer = SFTTrainer(
+    model=model,
+    args=SFTConfig(
+        output_dir="./output",
+        num_train_epochs=3,
+        per_device_train_batch_size=2,
+        gradient_accumulation_steps=8,
+        learning_rate=2e-4,
+        bf16=True,
+        logging_steps=10,
+        save_strategy="epoch",
+        report_to="wandb",
+    ),
+    train_dataset=dataset,
+    peft_config=peft_config,
+    processing_class=tokenizer,
+)
+trainer.train()
+trainer.save_model("./fine-tuned-model")
+EOF
+```
+
+## Axolotl (Production Fine-Tuning Framework)
+
+```yaml
+# config.yaml — Axolotl QLoRA config for Llama 3.1
+base_model: meta-llama/Llama-3.1-8B-Instruct
+model_type: LlamaForCausalLM
+tokenizer_type: PreTrainedTokenizerFast
+
+load_in_4bit: true
+adapter: qlora
+lora_r: 32
+lora_alpha: 64
+lora_dropout: 0.05
+lora_target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  - gate_proj
+  - up_proj
+  - down_proj
+
+datasets:
+  - path: your-org/your-dataset
+    type: alpaca              # or sharegpt, chat_template, etc.
+
+dataset_prepared_path: ./prepared-data
+val_set_size: 0.05
+output_dir: ./output
+
+sequence_len: 4096
+sample_packing: true         # pack multiple short samples for efficiency
+
+micro_batch_size: 2
+gradient_accumulation_steps: 8
+num_epochs: 3
+learning_rate: 2e-4
+optimizer: adamw_bnb_8bit
+lr_scheduler: cosine
+warmup_ratio: 0.05
+
+bf16: true
+flash_attention: true
+
+logging_steps: 10
+eval_steps: 100
+save_steps: 200
+wandb_project: my-fine-tune
+```
+
+```bash
+# Run with Axolotl
+pip install axolotl[flash-attn,deepspeed]
+accelerate launch -m axolotl.cli.train config.yaml
+```
+
+## Distributed Training with DeepSpeed
+
+```json
+// deepspeed_zero3.json — ZeRO Stage 3 (split optimizer + gradients + params)
+{
+  "zero_optimization": {
+    "stage": 3,
+    "offload_optimizer": {"device": "cpu", "pin_memory": true},
+    "offload_param": {"device": "cpu", "pin_memory": true},
+    "overlap_comm": true,
+    "contiguous_gradients": true,
+    "sub_group_size": 1e9,
+    "reduce_bucket_size": "auto",
+    "stage3_prefetch_bucket_size": "auto",
+    "stage3_param_persistence_threshold": "auto",
+    "stage3_max_live_parameters": 1e9,
+    "stage3_max_reuse_distance": 1e9,
+    "gather_16bit_weights_on_model_save": true
+  },
+  "bf16": {"enabled": true},
+  "gradient_clipping": 1.0,
+  "train_batch_size": "auto",
+  "train_micro_batch_size_per_gpu": "auto"
+}
+```
+
+```bash
+# Launch 4-GPU DeepSpeed training
+deepspeed --num_gpus=4 train.py \
+  --deepspeed deepspeed_zero3.json \
+  --model_name meta-llama/Llama-3.1-70B-Instruct \
+  --output_dir ./output
+```
+
+## DPO / RLHF Alignment
+
+```python
+from trl import DPOTrainer, DPOConfig
+from datasets import load_dataset
+
+# Dataset format: {"prompt": ..., "chosen": ..., "rejected": ...}
+dataset = load_dataset("your-org/preference-data")
+
+trainer = DPOTrainer(
+    model=model,
+    ref_model=None,           # None = implicit reference with peft
+    args=DPOConfig(
+        output_dir="./dpo-output",
+        beta=0.1,             # KL divergence weight
+        num_train_epochs=1,
+        per_device_train_batch_size=1,
+        gradient_accumulation_steps=16,
+        learning_rate=5e-7,
+        bf16=True,
+    ),
+    train_dataset=dataset["train"],
+    peft_config=peft_config,
+    processing_class=tokenizer,
+)
+trainer.train()
+```
+
+## Merging LoRA Adapters for Deployment
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM
+
+# Load base model in full precision
+base_model = AutoModelForCausalLM.from_pretrained(
+    "meta-llama/Llama-3.1-8B-Instruct",
+    torch_dtype=torch.bfloat16,
+    device_map="cpu",
+)
+
+# Load and merge LoRA adapter
+model = PeftModel.from_pretrained(base_model, "./fine-tuned-model")
+merged_model = model.merge_and_unload()
+
+# Save merged model (ready for vLLM serving)
+merged_model.save_pretrained("./merged-model", safe_serialization=True)
+tokenizer.save_pretrained("./merged-model")
+
+# Push to Hugging Face Hub
+merged_model.push_to_hub("your-org/your-fine-tuned-model")
+```
+
+## Kubernetes Training Job
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: llm-fine-tune
+spec:
+  template:
+    spec:
+      restartPolicy: OnFailure
+      nodeSelector:
+        nvidia.com/gpu.product: A100-SXM4-80GB
+      containers:
+      - name: trainer
+        image: nvcr.io/nvidia/pytorch:24.05-py3
+        command: ["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"]
+        resources:
+          limits:
+            nvidia.com/gpu: "4"
+            memory: "320Gi"
+          requests:
+            nvidia.com/gpu: "4"
+        volumeMounts:
+        - name: config
+          mountPath: /config
+        - name: model-cache
+          mountPath: /root/.cache/huggingface
+        - name: output
+          mountPath: /output
+        env:
+        - name: HUGGING_FACE_HUB_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: hf-token
+              key: token
+        - name: WANDB_API_KEY
+          valueFrom:
+            secretKeyRef:
+              name: wandb-token
+              key: key
+      volumes:
+      - name: config
+        configMap:
+          name: axolotl-config
+      - name: model-cache
+        persistentVolumeClaim:
+          claimName: model-cache-pvc
+      - name: output
+        persistentVolumeClaim:
+          claimName: training-output-pvc
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `CUDA out of memory` | Batch too large | Reduce `micro_batch_size`; increase `gradient_accumulation_steps` |
+| Training loss NaN | Learning rate too high | Lower LR to `1e-4` or `5e-5`; add warmup |
+| Slow training | No Flash Attention | Install `flash-attn`; enable `flash_attention: true` |
+| Poor fine-tune quality | Bad data formatting | Validate dataset format; check `sample_packing` compatibility |
+| Adapter merge errors | Mixed quantization | Merge in bf16 on CPU, not in 4-bit |
+
+## Best Practices
+
+- Use Flash Attention 2 — it's 2–4× faster and uses less memory.
+- Monitor training loss/eval loss via W&B or MLflow; overfit = more dropout or less data.
+- Validate with a held-out eval set (5–10%); MMLU or custom evals for quality gates.
+- Start with LoRA r=16 before increasing — higher rank = more parameters, diminishing returns.
+- Use `sample_packing` in Axolotl to maximize GPU utilization on short sequences.
+
+## Related Skills
+
+- vllm-server (`vllm-server`) - Serve fine-tuned models
+- gpu-server-management (`gpu-server-management`) - GPU setup
+- llm-inference-scaling (`llm-inference-scaling`) - Deploy at scale
+- ai-pipeline-orchestration (`ai-pipeline-orchestration`) - Training pipelines
+
+## Limitations
+
+- Infrastructure commands can disrupt services: confirm target host/scope and have backups/snapshots before mutating state.
+- Docs-only import: upstream scripts and templates not bundled.
+

package/bundled-skills/llm-gateway/SKILL.md

@@ -0,0 +1,282 @@
+---
+name: llm-gateway
+description: Deploy an API gateway for LLM traffic with load balancing, rate limiting,
+  key management, semantic caching, fallback routing, and cost tracking.
+category: devops
+risk: critical
+source: https://github.com/BagelHole/DevOps-Security-Agent-Skills
+source_repo: BagelHole/DevOps-Security-Agent-Skills
+source_type: community
+date_added: '2026-09-20'
+license: MIT
+license_source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/main/LICENSE
+compatibility: Requires the relevant OS/platform tooling and privileged access where
+  noted. Docs-only; helper scripts and templates not bundled.
+metadata:
+  author: devops-skills
+  version: '1.0'
+---
+
+# LLM Gateway
+
+A unified API gateway that routes LLM requests across providers and self-hosted models — with rate limiting, cost tracking, caching, and failover.
+
+## When to Use This Skill
+
+Use this skill when:
+- Running multiple LLM backends (OpenAI, Anthropic, vLLM, Ollama) behind a single endpoint
+- Enforcing per-team or per-user rate limits and spend budgets
+- Implementing automatic fallback when a provider is down
+- Adding semantic caching to reduce API costs by 20–50%
+- Centralizing API key management instead of distributing keys to every app
+
+## Prerequisites
+
+- Docker and Docker Compose
+- A PostgreSQL or SQLite database (for LiteLLM state)
+- LLM API keys (OpenAI, Anthropic, etc.) or self-hosted vLLM endpoints
+- Optional: Redis for caching and rate limiting
+
+## LiteLLM Proxy — Quick Start
+
+LiteLLM is the de facto open-source LLM gateway with OpenAI-compatible API.
+
+```bash
+# Run with Docker
+docker run -d \
+  --name litellm-proxy \
+  -p 4000:4000 \
+  -e OPENAI_API_KEY=$OPENAI_API_KEY \
+  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
+  -v $(pwd)/litellm-config.yaml:/app/config.yaml \
+  ghcr.io/berriai/litellm:main-latest \
+  --config /app/config.yaml \
+  --detailed_debug
+```
+
+## LiteLLM Configuration
+
+```yaml
+# litellm-config.yaml
+model_list:
+  # OpenAI models
+  - model_name: gpt-4o
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+      rpm: 10000
+      tpm: 2000000
+
+  - model_name: gpt-4o-mini
+    litellm_params:
+      model: openai/gpt-4o-mini
+      api_key: os.environ/OPENAI_API_KEY
+
+  # Anthropic
+  - model_name: claude-sonnet-4-6
+    litellm_params:
+      model: anthropic/claude-sonnet-4-6
+      api_key: os.environ/ANTHROPIC_API_KEY
+
+  # Self-hosted vLLM instances (load balanced)
+  - model_name: llama-3.1-8b
+    litellm_params:
+      model: openai/meta-llama/Llama-3.1-8B-Instruct
+      api_base: http://vllm-1:8000/v1
+      api_key: fake                    # vLLM key
+  - model_name: llama-3.1-8b
+    litellm_params:
+      model: openai/meta-llama/Llama-3.1-8B-Instruct
+      api_base: http://vllm-2:8000/v1  # second replica — auto load balanced
+      api_key: fake
+
+  # Fallback: cheap model if primary fails
+  - model_name: gpt-4o
+    litellm_params:
+      model: openai/gpt-4o-mini        # fallback to cheaper model
+      api_key: os.environ/OPENAI_API_KEY
+
+router_settings:
+  routing_strategy: least-busy         # or: latency-based, simple-shuffle
+  num_retries: 3
+  retry_after: 5
+  allowed_fails: 2
+  cooldown_time: 60
+
+  # Fallback configuration
+  fallbacks:
+    - gpt-4o: [claude-sonnet-4-6]
+    - claude-sonnet-4-6: [gpt-4o]
+
+litellm_settings:
+  # Semantic caching
+  cache: true
+  cache_params:
+    type: redis
+    host: redis
+    port: 6379
+    similarity_threshold: 0.90        # cache if >90% semantic similarity
+
+  # Logging
+  success_callback: ["langfuse"]
+  failure_callback: ["langfuse"]
+  langfuse_public_key: os.environ/LANGFUSE_PUBLIC_KEY
+  langfuse_secret_key: os.environ/LANGFUSE_SECRET_KEY
+
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  database_url: postgresql://litellm:password@postgres:5432/litellm
+  store_model_in_db: true
+```
+
+## Docker Compose: Full Gateway Stack
+
+```yaml
+services:
+  litellm:
+    image: ghcr.io/berriai/litellm:main-latest
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    volumes:
+      - ./litellm-config.yaml:/app/config.yaml
+    ports:
+      - "4000:4000"
+    environment:
+      - OPENAI_API_KEY=${OPENAI_API_KEY}
+      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
+      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
+      - DATABASE_URL=postgresql://litellm:password@postgres:5432/litellm
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_started
+    restart: unless-stopped
+
+  postgres:
+    image: postgres:16-alpine
+    environment:
+      POSTGRES_DB: litellm
+      POSTGRES_USER: litellm
+      POSTGRES_PASSWORD: password
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      retries: 5
+    restart: unless-stopped
+
+  redis:
+    image: redis:7-alpine
+    command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
+    volumes:
+      - redis-data:/data
+    restart: unless-stopped
+
+volumes:
+  postgres-data:
+  redis-data:
+```
+
+## Virtual Keys & Rate Limiting
+
+```bash
+# Create a virtual API key for a team (via LiteLLM API)
+curl -X POST http://localhost:4000/key/generate \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "team_id": "team-backend",
+    "key_alias": "backend-team-key",
+    "models": ["gpt-4o-mini", "llama-3.1-8b"],
+    "max_budget": 100,              # USD limit
+    "budget_duration": "monthly",
+    "rpm_limit": 100,               # requests per minute
+    "tpm_limit": 500000             # tokens per minute
+  }'
+
+# View spend
+curl http://localhost:4000/spend/keys \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
+```
+
+## Nginx Load Balancer (Alternative/Complement)
+
+```nginx
+# nginx.conf — round-robin across vLLM replicas
+upstream vllm_backends {
+    least_conn;
+    server vllm-1:8000 max_fails=3 fail_timeout=30s;
+    server vllm-2:8000 max_fails=3 fail_timeout=30s;
+    server vllm-3:8000 max_fails=3 fail_timeout=30s;
+    keepalive 32;
+}
+
+server {
+    listen 80;
+    server_name llm-api.internal;
+
+    # Rate limiting
+    limit_req_zone $http_authorization zone=per_key:10m rate=100r/m;
+    limit_req zone=per_key burst=20 nodelay;
+
+    location /v1/ {
+        proxy_pass http://vllm_backends;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_read_timeout 300s;        # long timeout for streaming
+        proxy_buffering off;            # required for SSE streaming
+        proxy_cache_bypass 1;
+    }
+}
+```
+
+## Monitoring Gateway Health
+
+```bash
+# Check LiteLLM health
+curl http://localhost:4000/health
+
+# Model-level health
+curl http://localhost:4000/health/liveliness
+
+# Spend by model
+curl http://localhost:4000/spend/models \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
+
+# Active virtual keys
+curl http://localhost:4000/key/list \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `ConnectionRefusedError` to backend | Backend not reachable | Check `api_base` URL; verify backend is healthy |
+| Rate limit errors (429) | Budget/RPM exceeded | Increase limits or rotate to fallback model |
+| Slow streaming responses | `proxy_buffering` enabled | Set `proxy_buffering off` in Nginx |
+| Cache miss rate high | Threshold too strict | Lower `similarity_threshold` to `0.85` |
+| Postgres connection errors | DB not ready | Add `depends_on` with `condition: service_healthy` |
+
+## Best Practices
+
+- Use virtual keys per team/app — never expose raw provider API keys.
+- Enable `cache: true` with Redis for repeated or similar queries; can cut costs 30–50%.
+- Set `num_retries: 3` with fallbacks to handle provider outages gracefully.
+- Log all requests to Langfuse or OpenTelemetry for cost attribution and debugging.
+- Use `least-busy` routing strategy for self-hosted models to avoid GPU saturation.
+
+## Related Skills
+
+- vllm-server (`vllm-server`) - Backend inference server
+- llm-inference-scaling (`llm-inference-scaling`) - Auto-scaling backends
+- llm-caching (`llm-caching`) - Semantic cache patterns
+- llm-cost-optimization (`llm-cost-optimization`) - Cost management
+
+## Limitations
+
+- Infrastructure commands can disrupt services: confirm target host/scope and have backups/snapshots before mutating state.
+- Docs-only import: upstream scripts and templates not bundled.
+