@synsci/cli-darwin-x64 1.1.49

package/bin/skills/sglang/references/deployment.md

@@ -0,0 +1,490 @@
+# Production Deployment Guide
+
+Complete guide to deploying SGLang in production environments.
+
+## Server Deployment
+
+### Basic server
+
+```bash
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+    --host 0.0.0.0 \
+    --port 30000 \
+    --mem-fraction-static 0.9
+```
+
+### Multi-GPU (Tensor Parallelism)
+
+```bash
+# Llama 3-70B on 4 GPUs
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-70B-Instruct \
+    --tp 4 \
+    --port 30000
+```
+
+### Quantization
+
+```bash
+# FP8 quantization (H100)
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-70B-Instruct \
+    --quantization fp8 \
+    --tp 4
+
+# INT4 AWQ quantization
+python -m sglang.launch_server \
+    --model-path TheBloke/Llama-2-70B-AWQ \
+    --quantization awq \
+    --tp 2
+
+# INT4 GPTQ quantization
+python -m sglang.launch_server \
+    --model-path TheBloke/Llama-2-70B-GPTQ \
+    --quantization gptq \
+    --tp 2
+```
+
+## Docker Deployment
+
+### Dockerfile
+
+```dockerfile
+FROM nvidia/cuda:12.1.0-devel-ubuntu22.04
+
+# Install Python
+RUN apt-get update && apt-get install -y python3.10 python3-pip git
+
+# Install SGLang
+RUN pip3 install "sglang[all]" flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
+
+# Copy model (or download at runtime)
+WORKDIR /app
+
+# Expose port
+EXPOSE 30000
+
+# Start server
+CMD ["python3", "-m", "sglang.launch_server", \
+     "--model-path", "meta-llama/Meta-Llama-3-8B-Instruct", \
+     "--host", "0.0.0.0", \
+     "--port", "30000"]
+```
+
+### Build and run
+
+```bash
+# Build image
+docker build -t sglang:latest .
+
+# Run with GPU
+docker run --gpus all -p 30000:30000 sglang:latest
+
+# Run with specific GPUs
+docker run --gpus '"device=0,1,2,3"' -p 30000:30000 sglang:latest
+
+# Run with custom model
+docker run --gpus all -p 30000:30000 \
+    -e MODEL_PATH="meta-llama/Meta-Llama-3-70B-Instruct" \
+    -e TP_SIZE="4" \
+    sglang:latest
+```
+
+## Kubernetes Deployment
+
+### Deployment YAML
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sglang-llama3-70b
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: sglang
+  template:
+    metadata:
+      labels:
+        app: sglang
+    spec:
+      containers:
+      - name: sglang
+        image: sglang:latest
+        command:
+          - python3
+          - -m
+          - sglang.launch_server
+          - --model-path=meta-llama/Meta-Llama-3-70B-Instruct
+          - --tp=4
+          - --host=0.0.0.0
+          - --port=30000
+          - --mem-fraction-static=0.9
+        ports:
+        - containerPort: 30000
+          name: http
+        resources:
+          limits:
+            nvidia.com/gpu: 4
+        livenessProbe:
+          httpGet:
+            path: /health
+            port: 30000
+          initialDelaySeconds: 60
+          periodSeconds: 10
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 30000
+          initialDelaySeconds: 30
+          periodSeconds: 5
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: sglang-service
+spec:
+  selector:
+    app: sglang
+  ports:
+  - port: 80
+    targetPort: 30000
+  type: LoadBalancer
+```
+
+## Monitoring
+
+### Health checks
+
+```bash
+# Health endpoint
+curl http://localhost:30000/health
+
+# Model info
+curl http://localhost:30000/v1/models
+
+# Server stats
+curl http://localhost:30000/stats
+```
+
+### Prometheus metrics
+
+```bash
+# Start server with metrics
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+    --enable-metrics
+
+# Metrics endpoint
+curl http://localhost:30000/metrics
+
+# Key metrics:
+# - sglang_request_total
+# - sglang_request_duration_seconds
+# - sglang_tokens_generated_total
+# - sglang_active_requests
+# - sglang_queue_size
+# - sglang_radix_cache_hit_rate
+# - sglang_gpu_memory_used_bytes
+```
+
+### Logging
+
+```bash
+# Enable debug logging
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+    --log-level debug
+
+# Log to file
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+    --log-file /var/log/sglang.log
+```
+
+## Load Balancing
+
+### NGINX configuration
+
+```nginx
+upstream sglang_backend {
+    least_conn;  # Route to least busy instance
+    server sglang-1:30000 max_fails=3 fail_timeout=30s;
+    server sglang-2:30000 max_fails=3 fail_timeout=30s;
+    server sglang-3:30000 max_fails=3 fail_timeout=30s;
+}
+
+server {
+    listen 80;
+
+    location / {
+        proxy_pass http://sglang_backend;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+        proxy_read_timeout 300s;
+        proxy_connect_timeout 10s;
+
+        # For streaming
+        proxy_buffering off;
+        proxy_cache off;
+    }
+
+    location /metrics {
+        proxy_pass http://sglang_backend/metrics;
+    }
+}
+```
+
+## Autoscaling
+
+### HPA based on GPU utilization
+
+```yaml
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: sglang-hpa
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: sglang-llama3-70b
+  minReplicas: 2
+  maxReplicas: 10
+  metrics:
+  - type: Pods
+    pods:
+      metric:
+        name: nvidia_gpu_duty_cycle
+      target:
+        type: AverageValue
+        averageValue: "80"  # Scale when GPU >80%
+```
+
+### HPA based on active requests
+
+```yaml
+metrics:
+- type: Pods
+  pods:
+    metric:
+      name: sglang_active_requests
+    target:
+      type: AverageValue
+      averageValue: "50"  # Scale when >50 active requests per pod
+```
+
+## Performance Tuning
+
+### Memory optimization
+
+```bash
+# Reduce memory usage
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-70B-Instruct \
+    --tp 4 \
+    --mem-fraction-static 0.85 \  # Use 85% of GPU memory
+    --max-radix-cache-len 8192    # Limit cache to 8K tokens
+```
+
+### Throughput optimization
+
+```bash
+# Maximize throughput
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+    --mem-fraction-static 0.95 \  # More memory for batching
+    --max-radix-cache-len 16384 \ # Larger cache
+    --max-running-requests 256    # More concurrent requests
+```
+
+### Latency optimization
+
+```bash
+# Minimize latency
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+    --max-running-requests 32 \   # Fewer concurrent (less queueing)
+    --schedule-policy fcfs         # First-come first-served
+```
+
+## Multi-Node Deployment
+
+### Ray cluster setup
+
+```bash
+# Head node
+ray start --head --port=6379
+
+# Worker nodes
+ray start --address='head-node:6379'
+
+# Launch server across cluster
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-405B-Instruct \
+    --tp 8 \
+    --num-nodes 2  # Use 2 nodes (8 GPUs each)
+```
+
+## Security
+
+### API authentication
+
+```bash
+# Start with API key
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+    --api-key YOUR_SECRET_KEY
+
+# Client request
+curl http://localhost:30000/v1/chat/completions \
+  -H "Authorization: Bearer YOUR_SECRET_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"model": "default", "messages": [...]}'
+```
+
+### Network policies (Kubernetes)
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: sglang-policy
+spec:
+  podSelector:
+    matchLabels:
+      app: sglang
+  policyTypes:
+  - Ingress
+  ingress:
+  - from:
+    - podSelector:
+        matchLabels:
+          app: api-gateway  # Only allow from gateway
+    ports:
+    - protocol: TCP
+      port: 30000
+```
+
+## Troubleshooting
+
+### High memory usage
+
+**Check**:
+```bash
+nvidia-smi
+curl http://localhost:30000/stats | grep cache
+```
+
+**Solutions**:
+```bash
+# Reduce cache size
+--max-radix-cache-len 4096
+
+# Reduce memory fraction
+--mem-fraction-static 0.75
+
+# Enable quantization
+--quantization fp8
+```
+
+### Low throughput
+
+**Check**:
+```bash
+curl http://localhost:30000/stats | grep queue_size
+```
+
+**Solutions**:
+```bash
+# Increase batch size
+--max-running-requests 256
+
+# Add more GPUs
+--tp 4  # Increase tensor parallelism
+
+# Check cache hit rate (should be >70%)
+curl http://localhost:30000/stats | grep cache_hit_rate
+```
+
+### High latency
+
+**Check**:
+```bash
+curl http://localhost:30000/metrics | grep duration
+```
+
+**Solutions**:
+```bash
+# Reduce concurrent requests
+--max-running-requests 32
+
+# Use FCFS scheduling (no batching delay)
+--schedule-policy fcfs
+
+# Add more replicas (horizontal scaling)
+```
+
+### OOM errors
+
+**Solutions**:
+```bash
+# Reduce batch size
+--max-running-requests 128
+
+# Reduce cache
+--max-radix-cache-len 2048
+
+# Enable quantization
+--quantization awq
+
+# Increase tensor parallelism
+--tp 8
+```
+
+## Best Practices
+
+1. **Use RadixAttention** - Enabled by default, 5-10× speedup for agents
+2. **Monitor cache hit rate** - Target >70% for agent/few-shot workloads
+3. **Set health checks** - Use `/health` endpoint for k8s probes
+4. **Enable metrics** - Monitor with Prometheus + Grafana
+5. **Use load balancing** - Distribute load across replicas
+6. **Tune memory** - Start with `--mem-fraction-static 0.9`, adjust based on OOM
+7. **Use quantization** - FP8 on H100, AWQ/GPTQ on A100
+8. **Set up autoscaling** - Scale based on GPU utilization or active requests
+9. **Log to persistent storage** - Use `--log-file` for debugging
+10. **Test before production** - Run load tests with expected traffic patterns
+
+## Cost Optimization
+
+### GPU selection
+
+**A100 80GB** ($3-4/hour):
+- Llama 3-70B with FP8 (TP=4)
+- Throughput: 10,000-15,000 tok/s
+- Cost per 1M tokens: $0.20-0.30
+
+**H100 80GB** ($6-8/hour):
+- Llama 3-70B with FP8 (TP=4)
+- Throughput: 20,000-30,000 tok/s
+- Cost per 1M tokens: $0.15-0.25 (2× faster)
+
+**L4** ($0.50-1/hour):
+- Llama 3-8B
+- Throughput: 1,500-2,500 tok/s
+- Cost per 1M tokens: $0.20-0.40
+
+### Batching for cost efficiency
+
+**Low batch (batch=1)**:
+- Throughput: 1,000 tok/s
+- Cost: $3/hour ÷ 1M tok/hour = $3/M tokens
+
+**High batch (batch=128)**:
+- Throughput: 8,000 tok/s
+- Cost: $3/hour ÷ 8M tok/hour = $0.375/M tokens
+- **8× cost reduction**
+
+**Recommendation**: Target batch size 64-256 for optimal cost/latency.