opencode-skills-collection 4.0.68 → 4.0.70

package/bundled-skills/gpu-kubernetes-operations/SKILL.md

@@ -0,0 +1,468 @@
+---
+name: gpu-kubernetes-operations
+description: Operate GPU-backed Kubernetes clusters for AI inference and training
+  with scheduling, autoscaling, node health, MIG partitioning, and cost controls.
+category: devops
+risk: critical
+source: https://github.com/BagelHole/DevOps-Security-Agent-Skills
+source_repo: BagelHole/DevOps-Security-Agent-Skills
+source_type: community
+date_added: '2026-09-20'
+license: MIT
+license_source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/main/LICENSE
+compatibility: Requires the relevant OS/platform tooling and privileged access where
+  noted. Docs-only; helper scripts and templates not bundled.
+metadata:
+  author: devops-skills
+  version: '1.0'
+---
+
+# GPU Kubernetes Operations
+
+Run resilient and cost-efficient GPU clusters for production AI workloads.
+
+## When to Use This Skill
+
+- Setting up GPU node pools in Kubernetes for AI inference or training
+- Configuring NVIDIA device plugin and GPU operator
+- Implementing MIG partitioning to share GPUs across workloads
+- Building GPU-aware autoscaling policies
+- Monitoring GPU health with DCGM and Prometheus
+- Troubleshooting GPU scheduling, driver, or OOM issues
+
+## Prerequisites
+
+- Kubernetes 1.28+ cluster with GPU-capable nodes
+- NVIDIA GPUs (A10, L4, A100, H100, or similar)
+- NVIDIA drivers installed on nodes (535+ recommended)
+- Helm 3 for operator and plugin installation
+- Prometheus stack for metrics collection
+
+## NVIDIA GPU Operator Installation
+
+The GPU Operator automates driver, toolkit, device plugin, and DCGM deployment.
+
+```bash
+# Add NVIDIA Helm repo
+helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
+helm repo update
+
+# Install GPU Operator
+helm install gpu-operator nvidia/gpu-operator \
+  --namespace gpu-operator \
+  --create-namespace \
+  --set driver.enabled=true \
+  --set toolkit.enabled=true \
+  --set devicePlugin.enabled=true \
+  --set dcgmExporter.enabled=true \
+  --set migManager.enabled=true \
+  --set nodeStatusExporter.enabled=true \
+  --version v24.3.0
+
+# Verify installation
+kubectl get pods -n gpu-operator
+kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'
+```
+
+## NVIDIA Device Plugin (Standalone)
+
+If not using the GPU Operator, deploy the device plugin directly.
+
+```yaml
+# nvidia-device-plugin.yaml
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: nvidia-device-plugin
+  namespace: kube-system
+spec:
+  selector:
+    matchLabels:
+      name: nvidia-device-plugin
+  template:
+    metadata:
+      labels:
+        name: nvidia-device-plugin
+    spec:
+      tolerations:
+        - key: nvidia.com/gpu
+          operator: Exists
+          effect: NoSchedule
+      priorityClassName: system-node-critical
+      containers:
+        - name: nvidia-device-plugin
+          image: nvcr.io/nvidia/k8s-device-plugin:v0.15.0
+          securityContext:
+            privileged: true
+          env:
+            - name: FAIL_ON_INIT_ERROR
+              value: "false"
+            - name: DEVICE_SPLIT_COUNT
+              value: "1"
+            - name: DEVICE_LIST_STRATEGY
+              value: "envvar"
+          volumeMounts:
+            - name: device-plugin
+              mountPath: /var/lib/kubelet/device-plugins
+      volumes:
+        - name: device-plugin
+          hostPath:
+            path: /var/lib/kubelet/device-plugins
+```
+
+## MIG (Multi-Instance GPU) Partitioning
+
+MIG allows a single A100 or H100 to be split into isolated GPU instances.
+
+```yaml
+# mig-config.yaml - ConfigMap for MIG Manager
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: mig-parted-config
+  namespace: gpu-operator
+data:
+  config.yaml: |
+    version: v1
+    mig-configs:
+      # 7 small instances for inference microservices
+      all-1g.10gb:
+        - devices: all
+          mig-enabled: true
+          mig-devices:
+            "1g.10gb": 7
+
+      # 3 medium instances for mid-size models
+      all-2g.20gb:
+        - devices: all
+          mig-enabled: true
+          mig-devices:
+            "2g.20gb": 3
+
+      # Mixed: 1 large + 2 small
+      mixed-inference:
+        - devices: all
+          mig-enabled: true
+          mig-devices:
+            "3g.40gb": 1
+            "1g.10gb": 4
+
+      # Full GPU for training (no partitioning)
+      all-disabled:
+        - devices: all
+          mig-enabled: false
+```
+
+```bash
+# Apply MIG profile to a node
+kubectl label nodes gpu-node-01 nvidia.com/mig.config=all-1g.10gb --overwrite
+
+# Verify MIG instances
+kubectl exec -it nvidia-device-plugin-xxxxx -n kube-system -- nvidia-smi mig -lgi
+
+# Check available MIG resources
+kubectl get nodes gpu-node-01 -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com")))'
+```
+
+### Requesting MIG Slices in Pods
+
+```yaml
+# pod-with-mig.yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: inference-small
+spec:
+  containers:
+    - name: model
+      image: registry.internal/vllm-server:latest
+      resources:
+        limits:
+          nvidia.com/mig-1g.10gb: 1
+      # For medium slice:
+      # nvidia.com/mig-2g.20gb: 1
+      # For large slice:
+      # nvidia.com/mig-3g.40gb: 1
+```
+
+## GPU Time-Slicing
+
+For GPUs that do not support MIG (A10, L4), use time-slicing to share a GPU.
+
+```yaml
+# time-slicing-config.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: time-slicing-config
+  namespace: gpu-operator
+data:
+  any: |-
+    version: v1
+    flags:
+      migStrategy: none
+    sharing:
+      timeSlicing:
+        renameByDefault: false
+        failRequestsGreaterThanOne: false
+        resources:
+          - name: nvidia.com/gpu
+            replicas: 4
+```
+
+```bash
+# Apply time-slicing config
+kubectl patch clusterpolicy/cluster-policy \
+  --type merge \
+  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'
+
+# After applying, each physical GPU appears as 4 virtual GPUs
+kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'
+# Output: "4" per physical GPU
+```
+
+## DCGM Monitoring
+
+```yaml
+# dcgm-servicemonitor.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: dcgm-exporter
+  namespace: gpu-operator
+  labels:
+    release: prometheus
+spec:
+  selector:
+    matchLabels:
+      app: nvidia-dcgm-exporter
+  endpoints:
+    - port: gpu-metrics
+      interval: 15s
+      path: /metrics
+```
+
+### Key DCGM Metrics and Alert Rules
+
+```yaml
+# gpu-alerts.yaml
+groups:
+  - name: gpu-health
+    rules:
+      - alert: GPUHighTemperature
+        expr: DCGM_FI_DEV_GPU_TEMP > 85
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "GPU {{ $labels.gpu }} temperature above 85C on {{ $labels.node }}"
+
+      - alert: GPUMemoryPressure
+        expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE) > 0.90
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "GPU memory above 90% on {{ $labels.node }} GPU {{ $labels.gpu }}"
+
+      - alert: GPUECCErrors
+        expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0
+        labels:
+          severity: critical
+        annotations:
+          summary: "Double-bit ECC errors detected on {{ $labels.node }} GPU {{ $labels.gpu }}"
+
+      - alert: GPUXidErrors
+        expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
+        labels:
+          severity: warning
+        annotations:
+          summary: "Xid error on {{ $labels.node }} GPU {{ $labels.gpu }}: {{ $labels.xid }}"
+
+      - alert: GPULowUtilization
+        expr: DCGM_FI_DEV_GPU_UTIL < 10 and on(pod) kube_pod_status_phase{phase="Running"} == 1
+        for: 30m
+        labels:
+          severity: info
+        annotations:
+          summary: "GPU underutilized on {{ $labels.node }} - consider rightsizing"
+
+      - alert: GPUDriverMismatch
+        expr: count(count by (driver_version)(DCGM_FI_DRIVER_VERSION)) > 1
+        labels:
+          severity: warning
+        annotations:
+          summary: "Multiple GPU driver versions detected across cluster"
+```
+
+## GPU Node Pool Configuration
+
+```yaml
+# gpu-nodepool.yaml
+apiVersion: v1
+kind: Node
+metadata:
+  labels:
+    gpu-type: a100
+    gpu-memory: "80gb"
+    gpu-mig-capable: "true"
+    node-role: gpu-inference
+spec:
+  taints:
+    - key: nvidia.com/gpu
+      value: "true"
+      effect: NoSchedule
+---
+# Inference deployment with GPU scheduling
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llm-inference
+  namespace: ai-serving
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: llm-inference
+  template:
+    metadata:
+      labels:
+        app: llm-inference
+    spec:
+      tolerations:
+        - key: nvidia.com/gpu
+          operator: Exists
+          effect: NoSchedule
+      nodeSelector:
+        gpu-type: a100
+      affinity:
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+            - weight: 100
+              podAffinityTerm:
+                labelSelector:
+                  matchLabels:
+                    app: llm-inference
+                topologyKey: kubernetes.io/hostname
+      containers:
+        - name: vllm
+          image: registry.internal/vllm-server:0.4.1
+          resources:
+            requests:
+              nvidia.com/gpu: 1
+              cpu: "4"
+              memory: "32Gi"
+            limits:
+              nvidia.com/gpu: 1
+              cpu: "8"
+              memory: "64Gi"
+          env:
+            - name: CUDA_VISIBLE_DEVICES
+              value: "all"
+```
+
+## GPU Autoscaling
+
+```yaml
+# gpu-hpa.yaml
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: llm-inference-hpa
+  namespace: ai-serving
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: llm-inference
+  minReplicas: 2
+  maxReplicas: 8
+  metrics:
+    - type: Pods
+      pods:
+        metric:
+          name: DCGM_FI_DEV_GPU_UTIL
+        target:
+          type: AverageValue
+          averageValue: "75"
+    - type: Pods
+      pods:
+        metric:
+          name: inference_queue_depth
+        target:
+          type: AverageValue
+          averageValue: "10"
+  behavior:
+    scaleUp:
+      stabilizationWindowSeconds: 60
+      policies:
+        - type: Pods
+          value: 2
+          periodSeconds: 120
+    scaleDown:
+      stabilizationWindowSeconds: 300
+      policies:
+        - type: Pods
+          value: 1
+          periodSeconds: 300
+---
+# Cluster Autoscaler config for GPU node pools
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cluster-autoscaler-config
+  namespace: kube-system
+data:
+  config: |
+    expander: priority
+    scale-down-delay-after-add: 10m
+    scale-down-unneeded-time: 10m
+    skip-nodes-with-local-storage: false
+    balance-similar-node-groups: true
+    expendable-pods-priority-cutoff: -10
+    gpu-total:
+      - min: 2
+        max: 16
+        gpu: nvidia.com/gpu
+```
+
+## Scheduling Patterns
+
+- Use node affinity by GPU type (A10/L4/A100/H100).
+- Separate latency-critical inference from batch training.
+- Pin model replicas with anti-affinity for availability.
+- Reserve headroom for failover and rolling updates.
+
+## Cost Optimization
+
+- Prefer MIG slices for smaller inference services.
+- Schedule batch jobs in off-peak windows.
+- Route low-priority traffic to cheaper model tiers.
+- Use spot/preemptible instances for training workloads.
+- Monitor GPU utilization and rightsize deployments.
+
+## Troubleshooting
+
+| Symptom | Check | Fix |
+|---------|-------|-----|
+| Pod stuck in Pending | `kubectl describe pod` for GPU resource events | Verify node has allocatable GPUs, check taints/tolerations |
+| CUDA OOM during inference | Model too large for GPU memory | Reduce batch size, use quantization, or use MIG slice |
+| DCGM metrics missing | ServiceMonitor labels matching | Verify DCGM exporter pod is running and scrape config |
+| Driver mismatch after upgrade | `nvidia-smi` on each node | Cordon node, drain, upgrade driver, uncordon |
+| GPU not detected | Device plugin pod logs | Restart device plugin, check NVIDIA container toolkit |
+| Time-slicing not working | ConfigMap applied but no extra GPUs | Restart device plugin pods after config change |
+| ECC errors increasing | `nvidia-smi -q -d ECC` | Schedule node drain and hardware replacement |
+
+## Related Skills
+
+- llm-inference-scaling (`llm-inference-scaling`) - Autoscale inference workloads
+- model-serving-kubernetes (`model-serving-kubernetes`) - Production model serving patterns
+- gpu-server-management (`gpu-server-management`) - Host-level GPU management fundamentals
+- multi-tenant-llm-hosting (`multi-tenant-llm-hosting`) - Multi-tenant GPU sharing
+- llm-cost-optimization (`llm-cost-optimization`) - Cost optimization strategies
+
+## Limitations
+
+- Infrastructure commands can disrupt services: confirm target host/scope and have backups/snapshots before mutating state.
+- Docs-only import: upstream scripts and templates not bundled.
+

package/bundled-skills/gpu-server-management/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: gpu-server-management
+description: Set up and manage NVIDIA GPU servers for AI workloads
+category: devops
+risk: critical
+source: https://github.com/BagelHole/DevOps-Security-Agent-Skills
+source_repo: BagelHole/DevOps-Security-Agent-Skills
+source_type: community
+date_added: '2026-09-20'
+license: MIT
+license_source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/main/LICENSE
+compatibility: Requires the relevant OS/platform tooling and privileged access where
+  noted. Docs-only; helper scripts and templates not bundled.
+metadata:
+  author: devops-skills
+  version: '1.0'
+---
+
+# GPU Server Management
+
+Provision, configure, and monitor NVIDIA GPU servers for AI inference and training workloads.
+
+## When to Use This Skill
+
+Use this skill when:
+- Setting up a new GPU server for LLM inference or model training
+- Installing or upgrading NVIDIA drivers and CUDA toolkit
+- Configuring Docker with NVIDIA Container Toolkit for GPU workloads
+- Partitioning A100/H100 GPUs with MIG for multi-tenant workloads
+- Troubleshooting GPU errors, driver issues, or thermal throttling
+
+## Prerequisites
+
+- Ubuntu 22.04 LTS (recommended) or RHEL 8/9
+- NVIDIA GPU (A10G, A100, H100, RTX 4090, or L40S recommended)
+- Root or sudo access
+- Internet access for package downloads
+
+## Driver Installation (Ubuntu)
+
+```bash
+# Remove old drivers
+sudo apt purge -y 'nvidia*' 'cuda*' 'libcuda*'
+sudo apt autoremove -y
+
+# Add NVIDIA package repository
+distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
+curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
+  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
+
+curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
+  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
+  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
+
+sudo apt update
+
+# Install latest driver (560.x as of 2025)
+sudo apt install -y nvidia-driver-560 cuda-toolkit-12-6
+
+# Install NVIDIA Container Toolkit (Docker GPU support)
+sudo apt install -y nvidia-container-toolkit
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+
+# Verify
+nvidia-smi
+nvcc --version
+docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
+```
+
+## Post-Install Configuration
+
+```bash
+# Enable persistence mode (reduces driver initialization latency)
+sudo nvidia-smi -pm 1
+
+# Set power limit (reduce heat/noise on inference servers)
+sudo nvidia-smi -pl 350          # watts; check TDP for your GPU model
+
+# Disable ECC on inference servers (frees ~6% VRAM, less safe)
+sudo nvidia-smi --ecc-config=0   # requires reboot
+
+# Enable P2P for multi-GPU NVLink training
+sudo nvidia-smi topo -m          # check NVLink topology
+```
+
+## GPU Health Monitoring
+
+```bash
+# Real-time monitoring (like htop for GPUs)
+watch -n 1 nvidia-smi
+
+# Detailed stats
+nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,\
+utilization.memory,memory.used,memory.free,power.draw,clocks.current.graphics \
+--format=csv --loop=1
+
+# DCGM — production monitoring daemon (for clusters)
+sudo apt install -y datacenter-gpu-manager
+sudo systemctl start dcgm
+dcgmi discovery -l                # list GPUs
+dcgmi diag -r 1                  # quick health check
+dcgmi diag -r 3                  # full diagnostic (takes ~20 min)
+
+# Check GPU errors (XID errors — important for stability)
+sudo dmesg | grep -i "NVRM\|nvidia\|XID"
+nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total \
+  --format=csv,noheader
+```
+
+## Prometheus GPU Metrics (DCGM Exporter)
+
+```bash
+# Deploy DCGM Exporter for Prometheus scraping
+docker run -d \
+  --name dcgm-exporter \
+  --gpus all \
+  --cap-add SYS_ADMIN \
+  -p 9400:9400 \
+  --restart unless-stopped \
+  nvcr.io/nvidia/k8s/dcgm-exporter:latest
+
+# Key metrics exposed:
+# DCGM_FI_DEV_GPU_UTIL          - GPU utilization %
+# DCGM_FI_DEV_MEM_COPY_UTIL     - Memory bandwidth utilization
+# DCGM_FI_DEV_FB_USED           - Framebuffer memory used (MB)
+# DCGM_FI_DEV_SM_CLOCK          - SM clock speed (MHz)
+# DCGM_FI_DEV_GPU_TEMP          - Temperature (°C)
+# DCGM_FI_DEV_POWER_USAGE       - Power draw (W)
+# DCGM_FI_DEV_XID_ERRORS        - XID error count (0 = healthy)
+```
+
+## MIG Partitioning (A100/H100)
+
+MIG (Multi-Instance GPU) allows slicing one GPU into isolated smaller GPUs.
+
+```bash
+# Enable MIG mode (requires reboot or restart of all processes)
+sudo nvidia-smi -mig 1
+sudo systemctl restart nvidia-persistenced
+
+# List available MIG profiles (A100 80GB example)
+nvidia-smi mig -lgip
+# 1g.10gb   — 1 slice,  10GB (max 7 instances)
+# 2g.20gb   — 2 slices, 20GB (max 3 instances)
+# 3g.40gb   — 3 slices, 40GB (max 2 instances)
+# 7g.80gb   — full GPU, 80GB (max 1 instance)
+
+# Create MIG instances (e.g., 3× 2g.20gb + 1× 2g.20gb = multi-tenant)
+sudo nvidia-smi mig -cgi 2g.20gb,2g.20gb,2g.20gb,2g.20gb -C
+
+# List created instances
+nvidia-smi mig -lgi
+nvidia-smi mig -lcgi
+
+# Use in Docker
+docker run --gpus '"device=MIG-GPU-xxx/0/0"' ...
+
+# Disable MIG
+sudo nvidia-smi mig -i 0 -dci
+sudo nvidia-smi mig -i 0 -dgi
+sudo nvidia-smi -mig 0
+```
+
+## Kernel & OS Tuning for GPU Servers
+
+```bash
+# Increase file descriptor limits
+echo '* soft nofile 1048576' | sudo tee -a /etc/security/limits.conf
+echo '* hard nofile 1048576' | sudo tee -a /etc/security/limits.conf
+
+# Disable transparent huge pages (reduces latency jitter)
+echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
+echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
+
+# Persist via rc.local or systemd unit:
+cat <<'EOF' | sudo tee /etc/rc.local
+#!/bin/bash
+echo never > /sys/kernel/mm/transparent_hugepage/enabled
+echo never > /sys/kernel/mm/transparent_hugepage/defrag
+nvidia-smi -pm 1
+exit 0
+EOF
+sudo chmod +x /etc/rc.local
+
+# PCIe performance mode
+sudo nvidia-smi --auto-boost-default=0
+sudo nvidia-smi --auto-boost-permission=0
+```
+
+## Multi-GPU Topology Check
+
+```bash
+# Check NVLink and PCIe topology
+nvidia-smi topo -m
+# Output shows interconnect type:
+# NV4 = NVLink 4.0 (H100 SXM)
+# NV2 = NVLink 2.0 (A100 SXM)
+# PHB = PCIe bus (slower; avoid for tensor parallel training)
+# PIX = same PCIe switch (fast)
+
+# Bandwidth test between GPUs
+/usr/local/cuda/samples/bin/x86_64/linux/release/p2pBandwidthLatencyTest
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `nvidia-smi: command not found` | Driver not installed | Follow driver installation steps above |
+| Driver version mismatch | CUDA/driver incompatibility | Check compatibility matrix at developer.nvidia.com |
+| GPU temperature >85°C | Poor airflow or fan failure | Check `nvidia-smi -q -d TEMPERATURE`; reseat cooler |
+| XID 79 errors | GPU hardware error | Run `dcgmi diag -r 3`; may need GPU replacement |
+| `failed to open device` in container | Container toolkit not configured | Run `nvidia-ctk runtime configure --runtime=docker` |
+| Low PCIe bandwidth | Wrong slot or power limit | Check `nvidia-smi -q | grep PCIe`; use x16 slot |
+
+## Best Practices
+
+- Always enable persistence mode (`nvidia-smi -pm 1`) — reduces first-request latency.
+- Monitor XID errors; persistent XID 79/94 indicates hardware failure.
+- For training: use NVLink-connected GPUs; for inference: PCIe is usually fine.
+- Set up DCGM alerts on temperature >80°C and power draw near TDP.
+- Use MIG for multi-tenant inference to provide GPU isolation between models.
+
+## Related Skills
+
+- vllm-server (`vllm-server`) - LLM inference on GPUs
+- llm-fine-tuning (`llm-fine-tuning`) - GPU training setup
+- linux-hardening (`linux-hardening`) - Secure the host OS
+- prometheus-grafana (`prometheus-grafana`) - Metrics dashboards
+
+## Limitations
+
+- Infrastructure commands can disrupt services: confirm target host/scope and have backups/snapshots before mutating state.
+- Docs-only import: upstream scripts and templates not bundled.
+