claude-code-pack 1.0.0

package/skills/senior-ml-engineer/references/llm_integration_guide.md

@@ -0,0 +1,317 @@
+# LLM Integration Guide
+
+Production patterns for integrating Large Language Models into applications.
+
+---
+
+## Table of Contents
+
+- [API Integration Patterns](#api-integration-patterns)
+- [Prompt Engineering](#prompt-engineering)
+- [Token Optimization](#token-optimization)
+- [Cost Management](#cost-management)
+- [Error Handling](#error-handling)
+
+---
+
+## API Integration Patterns
+
+### Provider Abstraction Layer
+
+```python
+from abc import ABC, abstractmethod
+from typing import List, Dict, Any
+
+class LLMProvider(ABC):
+    """Abstract base class for LLM providers."""
+
+    @abstractmethod
+    def complete(self, prompt: str, **kwargs) -> str:
+        pass
+
+    @abstractmethod
+    def chat(self, messages: List[Dict], **kwargs) -> str:
+        pass
+
+class OpenAIProvider(LLMProvider):
+    def __init__(self, api_key: str, model: str = "gpt-4"):
+        self.client = OpenAI(api_key=api_key)
+        self.model = model
+
+    def complete(self, prompt: str, **kwargs) -> str:
+        response = self.client.completions.create(
+            model=self.model,
+            prompt=prompt,
+            **kwargs
+        )
+        return response.choices[0].text
+
+class AnthropicProvider(LLMProvider):
+    def __init__(self, api_key: str, model: str = "claude-3-opus"):
+        self.client = Anthropic(api_key=api_key)
+        self.model = model
+
+    def chat(self, messages: List[Dict], **kwargs) -> str:
+        response = self.client.messages.create(
+            model=self.model,
+            messages=messages,
+            **kwargs
+        )
+        return response.content[0].text
+```
+
+### Retry and Fallback Strategy
+
+```python
+import time
+from tenacity import retry, stop_after_attempt, wait_exponential
+
+@retry(
+    stop=stop_after_attempt(3),
+    wait=wait_exponential(multiplier=1, min=1, max=10)
+)
+def call_llm_with_retry(provider: LLMProvider, prompt: str) -> str:
+    """Call LLM with exponential backoff retry."""
+    return provider.complete(prompt)
+
+def call_with_fallback(
+    primary: LLMProvider,
+    fallback: LLMProvider,
+    prompt: str
+) -> str:
+    """Try primary provider, fall back on failure."""
+    try:
+        return call_llm_with_retry(primary, prompt)
+    except Exception as e:
+        logger.warning(f"Primary provider failed: {e}, using fallback")
+        return call_llm_with_retry(fallback, prompt)
+```
+
+---
+
+## Prompt Engineering
+
+### Prompt Templates
+
+| Pattern | Use Case | Structure |
+|---------|----------|-----------|
+| Zero-shot | Simple tasks | Task description + input |
+| Few-shot | Complex tasks | Examples + task + input |
+| Chain-of-thought | Reasoning | "Think step by step" + task |
+| Role-based | Specialized output | System role + task |
+
+### Few-Shot Template
+
+```python
+FEW_SHOT_TEMPLATE = """
+You are a sentiment classifier. Classify the sentiment as positive, negative, or neutral.
+
+Examples:
+Input: "This product is amazing, I love it!"
+Output: positive
+
+Input: "Terrible experience, waste of money."
+Output: negative
+
+Input: "The product arrived on time."
+Output: neutral
+
+Now classify:
+Input: "{user_input}"
+Output:"""
+
+def classify_sentiment(text: str, provider: LLMProvider) -> str:
+    prompt = FEW_SHOT_TEMPLATE.format(user_input=text)
+    response = provider.complete(prompt, max_tokens=10, temperature=0)
+    return response.strip().lower()
+```
+
+### System Prompts for Consistency
+
+```python
+SYSTEM_PROMPT = """You are a helpful assistant that answers questions about our product.
+
+Guidelines:
+- Be concise and direct
+- Use bullet points for lists
+- If unsure, say "I don't have that information"
+- Never make up information
+- Keep responses under 200 words
+
+Product context:
+{product_context}
+"""
+
+def create_chat_messages(user_query: str, context: str) -> List[Dict]:
+    return [
+        {"role": "system", "content": SYSTEM_PROMPT.format(product_context=context)},
+        {"role": "user", "content": user_query}
+    ]
+```
+
+---
+
+## Token Optimization
+
+### Token Counting
+
+```python
+import tiktoken
+
+def count_tokens(text: str, model: str = "gpt-4") -> int:
+    """Count tokens for a given text and model."""
+    encoding = tiktoken.encoding_for_model(model)
+    return len(encoding.encode(text))
+
+def truncate_to_token_limit(text: str, max_tokens: int, model: str = "gpt-4") -> str:
+    """Truncate text to fit within token limit."""
+    encoding = tiktoken.encoding_for_model(model)
+    tokens = encoding.encode(text)
+
+    if len(tokens) <= max_tokens:
+        return text
+
+    return encoding.decode(tokens[:max_tokens])
+```
+
+### Context Window Management
+
+| Model | Context Window | Effective Limit |
+|-------|----------------|-----------------|
+| GPT-4 | 8,192 | ~6,000 (leave room for response) |
+| GPT-4-32k | 32,768 | ~28,000 |
+| Claude 3 | 200,000 | ~180,000 |
+| Llama 3 | 8,192 | ~6,000 |
+
+### Chunking Strategy
+
+```python
+def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 100) -> List[str]:
+    """Split text into overlapping chunks."""
+    chunks = []
+    start = 0
+
+    while start < len(text):
+        end = start + chunk_size
+        chunk = text[start:end]
+        chunks.append(chunk)
+        start = end - overlap
+
+    return chunks
+```
+
+---
+
+## Cost Management
+
+### Cost Calculation
+
+| Provider | Input Cost | Output Cost | Example (1K tokens) |
+|----------|------------|-------------|---------------------|
+| GPT-4 | $0.03/1K | $0.06/1K | $0.09 |
+| GPT-3.5 | $0.0005/1K | $0.0015/1K | $0.002 |
+| Claude 3 Opus | $0.015/1K | $0.075/1K | $0.09 |
+| Claude 3 Haiku | $0.00025/1K | $0.00125/1K | $0.0015 |
+
+### Cost Tracking
+
+```python
+from dataclasses import dataclass
+from typing import Optional
+
+@dataclass
+class LLMUsage:
+    input_tokens: int
+    output_tokens: int
+    model: str
+    cost: float
+
+def calculate_cost(
+    input_tokens: int,
+    output_tokens: int,
+    model: str
+) -> float:
+    """Calculate cost based on token usage."""
+    PRICING = {
+        "gpt-4": {"input": 0.03, "output": 0.06},
+        "gpt-3.5-turbo": {"input": 0.0005, "output": 0.0015},
+        "claude-3-opus": {"input": 0.015, "output": 0.075},
+    }
+
+    prices = PRICING.get(model, {"input": 0.01, "output": 0.03})
+
+    input_cost = (input_tokens / 1000) * prices["input"]
+    output_cost = (output_tokens / 1000) * prices["output"]
+
+    return input_cost + output_cost
+```
+
+### Cost Optimization Strategies
+
+1. **Use smaller models for simple tasks** - GPT-3.5 for classification, GPT-4 for reasoning
+2. **Cache common responses** - Store results for repeated queries
+3. **Batch requests** - Combine multiple items in single prompt
+4. **Truncate context** - Only include relevant information
+5. **Set max_tokens limit** - Prevent runaway responses
+
+---
+
+## Error Handling
+
+### Common Error Types
+
+| Error | Cause | Handling |
+|-------|-------|----------|
+| RateLimitError | Too many requests | Exponential backoff |
+| InvalidRequestError | Bad input | Validate before sending |
+| AuthenticationError | Invalid API key | Check credentials |
+| ServiceUnavailable | Provider down | Fallback to alternative |
+| ContextLengthExceeded | Input too long | Truncate or chunk |
+
+### Error Handling Pattern
+
+```python
+from openai import RateLimitError, APIError
+
+def safe_llm_call(provider: LLMProvider, prompt: str, max_retries: int = 3) -> str:
+    """Safely call LLM with comprehensive error handling."""
+    for attempt in range(max_retries):
+        try:
+            return provider.complete(prompt)
+
+        except RateLimitError:
+            wait_time = 2 ** attempt
+            logger.warning(f"Rate limited, waiting {wait_time}s")
+            time.sleep(wait_time)
+
+        except APIError as e:
+            if e.status_code >= 500:
+                logger.warning(f"Server error: {e}, retrying...")
+                time.sleep(1)
+            else:
+                raise
+
+    raise Exception(f"Failed after {max_retries} attempts")
+```
+
+### Response Validation
+
+```python
+import json
+from pydantic import BaseModel, ValidationError
+
+class StructuredResponse(BaseModel):
+    answer: str
+    confidence: float
+    sources: List[str]
+
+def parse_structured_response(response: str) -> StructuredResponse:
+    """Parse and validate LLM JSON response."""
+    try:
+        data = json.loads(response)
+        return StructuredResponse(**data)
+    except json.JSONDecodeError:
+        raise ValueError("Response is not valid JSON")
+    except ValidationError as e:
+        raise ValueError(f"Response validation failed: {e}")
+```

package/skills/senior-ml-engineer/references/mlops_production_patterns.md

@@ -0,0 +1,265 @@
+# MLOps Production Patterns
+
+Production ML infrastructure patterns for model deployment, monitoring, and lifecycle management.
+
+---
+
+## Table of Contents
+
+- [Model Deployment Pipeline](#model-deployment-pipeline)
+- [Feature Store Architecture](#feature-store-architecture)
+- [Model Monitoring](#model-monitoring)
+- [A/B Testing Infrastructure](#ab-testing-infrastructure)
+- [Automated Retraining](#automated-retraining)
+
+---
+
+## Model Deployment Pipeline
+
+### Deployment Workflow
+
+1. Export trained model to standardized format (ONNX, TorchScript, SavedModel)
+2. Package model with dependencies in Docker container
+3. Deploy to staging environment
+4. Run integration tests against staging
+5. Deploy canary (5% traffic) to production
+6. Monitor latency and error rates for 1 hour
+7. Promote to full production if metrics pass
+8. **Validation:** p95 latency < 100ms, error rate < 0.1%
+
+### Container Structure
+
+```dockerfile
+FROM python:3.11-slim
+
+# Install dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy model artifacts
+COPY model/ /app/model/
+COPY src/ /app/src/
+
+# Health check endpoint
+HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1
+
+EXPOSE 8080
+CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]
+```
+
+### Model Serving Options
+
+| Option | Latency | Throughput | Use Case |
+|--------|---------|------------|----------|
+| FastAPI + Uvicorn | Low | Medium | REST APIs, small models |
+| Triton Inference Server | Very Low | Very High | GPU inference, batching |
+| TensorFlow Serving | Low | High | TensorFlow models |
+| TorchServe | Low | High | PyTorch models |
+| Ray Serve | Medium | High | Complex pipelines, multi-model |
+
+### Kubernetes Deployment
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: model-serving
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: model-serving
+  template:
+    spec:
+      containers:
+      - name: model
+        image: model:v1.0.0
+        resources:
+          requests:
+            memory: "2Gi"
+            cpu: "1"
+          limits:
+            memory: "4Gi"
+            cpu: "2"
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 10
+          periodSeconds: 5
+```
+
+---
+
+## Feature Store Architecture
+
+### Feature Store Components
+
+| Component | Purpose | Tools |
+|-----------|---------|-------|
+| Offline Store | Training data, batch features | BigQuery, Snowflake, S3 |
+| Online Store | Low-latency serving | Redis, DynamoDB, Feast |
+| Feature Registry | Metadata, lineage | Feast, Tecton, Hopsworks |
+| Transformation | Feature engineering | Spark, Flink, dbt |
+
+### Feature Pipeline Workflow
+
+1. Define feature schema in registry
+2. Implement transformation logic (SQL or Python)
+3. Backfill historical features to offline store
+4. Schedule incremental updates
+5. Materialize to online store for serving
+6. Monitor feature freshness and quality
+7. **Validation:** Feature values within expected ranges, no nulls in required fields
+
+### Feature Definition Example
+
+```python
+from feast import Entity, Feature, FeatureView, FileSource
+
+user = Entity(name="user_id", value_type=ValueType.INT64)
+
+user_features = FeatureView(
+    name="user_features",
+    entities=["user_id"],
+    ttl=timedelta(days=1),
+    features=[
+        Feature(name="purchase_count_30d", dtype=ValueType.INT64),
+        Feature(name="avg_order_value", dtype=ValueType.FLOAT),
+        Feature(name="days_since_last_purchase", dtype=ValueType.INT64),
+    ],
+    online=True,
+    source=FileSource(path="data/user_features.parquet"),
+)
+```
+
+---
+
+## Model Monitoring
+
+### Monitoring Dimensions
+
+| Dimension | Metrics | Alert Threshold |
+|-----------|---------|-----------------|
+| Latency | p50, p95, p99 | p95 > 100ms |
+| Throughput | requests/sec | < 80% baseline |
+| Errors | error rate, 5xx count | > 0.1% |
+| Data Drift | PSI, KS statistic | PSI > 0.2 |
+| Model Drift | accuracy, AUC decay | > 5% drop |
+
+### Data Drift Detection
+
+```python
+from scipy.stats import ks_2samp
+import numpy as np
+
+def detect_drift(reference: np.array, current: np.array, threshold: float = 0.05):
+    """Detect distribution drift using Kolmogorov-Smirnov test."""
+    statistic, p_value = ks_2samp(reference, current)
+
+    drift_detected = p_value < threshold
+
+    return {
+        "drift_detected": drift_detected,
+        "ks_statistic": statistic,
+        "p_value": p_value,
+        "threshold": threshold
+    }
+```
+
+### Monitoring Dashboard Metrics
+
+**Infrastructure:**
+- Request latency (p50, p95, p99)
+- Requests per second
+- Error rate by type
+- CPU/memory utilization
+- GPU utilization (if applicable)
+
+**Model Performance:**
+- Prediction distribution
+- Feature value distributions
+- Model output confidence
+- Ground truth vs predictions (when available)
+
+---
+
+## A/B Testing Infrastructure
+
+### Experiment Workflow
+
+1. Define experiment hypothesis and success metrics
+2. Calculate required sample size for statistical power
+3. Configure traffic split (control vs treatment)
+4. Deploy treatment model alongside control
+5. Route traffic based on user/session hash
+6. Collect metrics for both variants
+7. Run statistical significance test
+8. **Validation:** p-value < 0.05, minimum sample size reached
+
+### Traffic Splitting
+
+```python
+import hashlib
+
+def get_variant(user_id: str, experiment: str, control_pct: float = 0.5) -> str:
+    """Deterministic traffic splitting based on user ID."""
+    hash_input = f"{user_id}:{experiment}"
+    hash_value = int(hashlib.md5(hash_input.encode()).hexdigest(), 16)
+    bucket = (hash_value % 100) / 100.0
+
+    return "control" if bucket < control_pct else "treatment"
+```
+
+### Metrics Collection
+
+| Metric Type | Examples | Collection Method |
+|-------------|----------|-------------------|
+| Primary | Conversion rate, revenue | Event logging |
+| Secondary | Latency, engagement | Request logs |
+| Guardrail | Error rate, crashes | Monitoring system |
+
+---
+
+## Automated Retraining
+
+### Retraining Triggers
+
+| Trigger | Detection Method | Action |
+|---------|------------------|--------|
+| Scheduled | Cron (weekly/monthly) | Full retrain |
+| Performance drop | Accuracy < threshold | Immediate retrain |
+| Data drift | PSI > 0.2 | Evaluate, then retrain |
+| New data volume | X new samples | Incremental update |
+
+### Retraining Pipeline
+
+1. Trigger detection (schedule, drift, performance)
+2. Fetch latest training data from feature store
+3. Run training job with hyperparameter config
+4. Evaluate model on holdout set
+5. Compare against production model
+6. If improved: register new model version
+7. Deploy to staging for validation
+8. Promote to production via canary
+9. **Validation:** New model outperforms baseline on key metrics
+
+### MLflow Model Registry Integration
+
+```python
+import mlflow
+
+def register_model(model, metrics: dict, model_name: str):
+    """Register trained model with MLflow."""
+    with mlflow.start_run():
+        # Log metrics
+        for name, value in metrics.items():
+            mlflow.log_metric(name, value)
+
+        # Log model
+        mlflow.sklearn.log_model(model, "model")
+
+        # Register in model registry
+        model_uri = f"runs:/{mlflow.active_run().info.run_id}/model"
+        mlflow.register_model(model_uri, model_name)
+```