@diottodev/runic 1.0.0 → 1.1.0

package/engineering-team/skills/senior-ml-engineer/references/mlops_production_patterns.md

@@ -0,0 +1,265 @@
+# MLOps Production Patterns
+
+Production ML infrastructure patterns for model deployment, monitoring, and lifecycle management.
+
+---
+
+## Table of Contents
+
+- [Model Deployment Pipeline](#model-deployment-pipeline)
+- [Feature Store Architecture](#feature-store-architecture)
+- [Model Monitoring](#model-monitoring)
+- [A/B Testing Infrastructure](#ab-testing-infrastructure)
+- [Automated Retraining](#automated-retraining)
+
+---
+
+## Model Deployment Pipeline
+
+### Deployment Workflow
+
+1. Export trained model to standardized format (ONNX, TorchScript, SavedModel)
+2. Package model with dependencies in Docker container
+3. Deploy to staging environment
+4. Run integration tests against staging
+5. Deploy canary (5% traffic) to production
+6. Monitor latency and error rates for 1 hour
+7. Promote to full production if metrics pass
+8. **Validation:** p95 latency < 100ms, error rate < 0.1%
+
+### Container Structure
+
+```dockerfile
+FROM python:3.11-slim
+
+# Install dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy model artifacts
+COPY model/ /app/model/
+COPY src/ /app/src/
+
+# Health check endpoint
+HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1
+
+EXPOSE 8080
+CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]
+```
+
+### Model Serving Options
+
+| Option | Latency | Throughput | Use Case |
+|--------|---------|------------|----------|
+| FastAPI + Uvicorn | Low | Medium | REST APIs, small models |
+| Triton Inference Server | Very Low | Very High | GPU inference, batching |
+| TensorFlow Serving | Low | High | TensorFlow models |
+| TorchServe | Low | High | PyTorch models |
+| Ray Serve | Medium | High | Complex pipelines, multi-model |
+
+### Kubernetes Deployment
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: model-serving
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: model-serving
+  template:
+    spec:
+      containers:
+      - name: model
+        image: model:v1.0.0
+        resources:
+          requests:
+            memory: "2Gi"
+            cpu: "1"
+          limits:
+            memory: "4Gi"
+            cpu: "2"
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 10
+          periodSeconds: 5
+```
+
+---
+
+## Feature Store Architecture
+
+### Feature Store Components
+
+| Component | Purpose | Tools |
+|-----------|---------|-------|
+| Offline Store | Training data, batch features | BigQuery, Snowflake, S3 |
+| Online Store | Low-latency serving | Redis, DynamoDB, Feast |
+| Feature Registry | Metadata, lineage | Feast, Tecton, Hopsworks |
+| Transformation | Feature engineering | Spark, Flink, dbt |
+
+### Feature Pipeline Workflow
+
+1. Define feature schema in registry
+2. Implement transformation logic (SQL or Python)
+3. Backfill historical features to offline store
+4. Schedule incremental updates
+5. Materialize to online store for serving
+6. Monitor feature freshness and quality
+7. **Validation:** Feature values within expected ranges, no nulls in required fields
+
+### Feature Definition Example
+
+```python
+from feast import Entity, Feature, FeatureView, FileSource
+
+user = Entity(name="user_id", value_type=ValueType.INT64)
+
+user_features = FeatureView(
+    name="user_features",
+    entities=["user_id"],
+    ttl=timedelta(days=1),
+    features=[
+        Feature(name="purchase_count_30d", dtype=ValueType.INT64),
+        Feature(name="avg_order_value", dtype=ValueType.FLOAT),
+        Feature(name="days_since_last_purchase", dtype=ValueType.INT64),
+    ],
+    online=True,
+    source=FileSource(path="data/user_features.parquet"),
+)
+```
+
+---
+
+## Model Monitoring
+
+### Monitoring Dimensions
+
+| Dimension | Metrics | Alert Threshold |
+|-----------|---------|-----------------|
+| Latency | p50, p95, p99 | p95 > 100ms |
+| Throughput | requests/sec | < 80% baseline |
+| Errors | error rate, 5xx count | > 0.1% |
+| Data Drift | PSI, KS statistic | PSI > 0.2 |
+| Model Drift | accuracy, AUC decay | > 5% drop |
+
+### Data Drift Detection
+
+```python
+from scipy.stats import ks_2samp
+import numpy as np
+
+def detect_drift(reference: np.array, current: np.array, threshold: float = 0.05):
+    """Detect distribution drift using Kolmogorov-Smirnov test."""
+    statistic, p_value = ks_2samp(reference, current)
+
+    drift_detected = p_value < threshold
+
+    return {
+        "drift_detected": drift_detected,
+        "ks_statistic": statistic,
+        "p_value": p_value,
+        "threshold": threshold
+    }
+```
+
+### Monitoring Dashboard Metrics
+
+**Infrastructure:**
+- Request latency (p50, p95, p99)
+- Requests per second
+- Error rate by type
+- CPU/memory utilization
+- GPU utilization (if applicable)
+
+**Model Performance:**
+- Prediction distribution
+- Feature value distributions
+- Model output confidence
+- Ground truth vs predictions (when available)
+
+---
+
+## A/B Testing Infrastructure
+
+### Experiment Workflow
+
+1. Define experiment hypothesis and success metrics
+2. Calculate required sample size for statistical power
+3. Configure traffic split (control vs treatment)
+4. Deploy treatment model alongside control
+5. Route traffic based on user/session hash
+6. Collect metrics for both variants
+7. Run statistical significance test
+8. **Validation:** p-value < 0.05, minimum sample size reached
+
+### Traffic Splitting
+
+```python
+import hashlib
+
+def get_variant(user_id: str, experiment: str, control_pct: float = 0.5) -> str:
+    """Deterministic traffic splitting based on user ID."""
+    hash_input = f"{user_id}:{experiment}"
+    hash_value = int(hashlib.md5(hash_input.encode()).hexdigest(), 16)
+    bucket = (hash_value % 100) / 100.0
+
+    return "control" if bucket < control_pct else "treatment"
+```
+
+### Metrics Collection
+
+| Metric Type | Examples | Collection Method |
+|-------------|----------|-------------------|
+| Primary | Conversion rate, revenue | Event logging |
+| Secondary | Latency, engagement | Request logs |
+| Guardrail | Error rate, crashes | Monitoring system |
+
+---
+
+## Automated Retraining
+
+### Retraining Triggers
+
+| Trigger | Detection Method | Action |
+|---------|------------------|--------|
+| Scheduled | Cron (weekly/monthly) | Full retrain |
+| Performance drop | Accuracy < threshold | Immediate retrain |
+| Data drift | PSI > 0.2 | Evaluate, then retrain |
+| New data volume | X new samples | Incremental update |
+
+### Retraining Pipeline
+
+1. Trigger detection (schedule, drift, performance)
+2. Fetch latest training data from feature store
+3. Run training job with hyperparameter config
+4. Evaluate model on holdout set
+5. Compare against production model
+6. If improved: register new model version
+7. Deploy to staging for validation
+8. Promote to production via canary
+9. **Validation:** New model outperforms baseline on key metrics
+
+### MLflow Model Registry Integration
+
+```python
+import mlflow
+
+def register_model(model, metrics: dict, model_name: str):
+    """Register trained model with MLflow."""
+    with mlflow.start_run():
+        # Log metrics
+        for name, value in metrics.items():
+            mlflow.log_metric(name, value)
+
+        # Log model
+        mlflow.sklearn.log_model(model, "model")
+
+        # Register in model registry
+        model_uri = f"runs:/{mlflow.active_run().info.run_id}/model"
+        mlflow.register_model(model_uri, model_name)
+```

package/engineering-team/skills/senior-ml-engineer/references/rag_system_architecture.md

@@ -0,0 +1,371 @@
+# RAG System Architecture
+
+Retrieval-Augmented Generation patterns for production applications.
+
+---
+
+## Table of Contents
+
+- [RAG Pipeline Architecture](#rag-pipeline-architecture)
+- [Vector Database Selection](#vector-database-selection)
+- [Chunking Strategies](#chunking-strategies)
+- [Embedding Models](#embedding-models)
+- [Retrieval Optimization](#retrieval-optimization)
+
+---
+
+## RAG Pipeline Architecture
+
+### Basic RAG Flow
+
+1. Receive user query
+2. Generate query embedding
+3. Search vector database for relevant chunks
+4. Rerank retrieved chunks by relevance
+5. Format context with retrieved chunks
+6. Send prompt to LLM with context
+7. Return generated response
+8. **Validation:** Response references retrieved context, no hallucinations
+
+### Pipeline Components
+
+```python
+from dataclasses import dataclass
+from typing import List
+
+@dataclass
+class Document:
+    content: str
+    metadata: dict
+    embedding: List[float] = None
+
+@dataclass
+class RetrievalResult:
+    document: Document
+    score: float
+
+class RAGPipeline:
+    def __init__(
+        self,
+        embedder: Embedder,
+        vector_store: VectorStore,
+        llm: LLMProvider,
+        reranker: Reranker = None
+    ):
+        self.embedder = embedder
+        self.vector_store = vector_store
+        self.llm = llm
+        self.reranker = reranker
+
+    def query(self, question: str, top_k: int = 5) -> str:
+        # 1. Embed query
+        query_embedding = self.embedder.embed(question)
+
+        # 2. Retrieve relevant documents
+        results = self.vector_store.search(query_embedding, top_k=top_k * 2)
+
+        # 3. Rerank if available
+        if self.reranker:
+            results = self.reranker.rerank(question, results)[:top_k]
+        else:
+            results = results[:top_k]
+
+        # 4. Build context
+        context = self._build_context(results)
+
+        # 5. Generate response
+        prompt = self._build_prompt(question, context)
+        return self.llm.complete(prompt)
+
+    def _build_context(self, results: List[RetrievalResult]) -> str:
+        return "\n\n".join([
+            f"[Source {i+1}]: {r.document.content}"
+            for i, r in enumerate(results)
+        ])
+
+    def _build_prompt(self, question: str, context: str) -> str:
+        return f"""Answer the question based on the context provided.
+
+Context:
+{context}
+
+Question: {question}
+
+Answer:"""
+```
+
+---
+
+## Vector Database Selection
+
+### Comparison Matrix
+
+| Database | Hosting | Scale | Latency | Cost | Best For |
+|----------|---------|-------|---------|------|----------|
+| Pinecone | Managed | High | Low | $$ | Production, managed |
+| Weaviate | Both | High | Low | $ | Hybrid search |
+| Qdrant | Both | High | Very Low | $ | Performance-critical |
+| Chroma | Self-hosted | Medium | Low | Free | Prototyping |
+| pgvector | Self-hosted | Medium | Medium | Free | Existing Postgres |
+| Milvus | Both | Very High | Low | $ | Large-scale |
+
+### Pinecone Integration
+
+```python
+import pinecone
+
+class PineconeVectorStore:
+    def __init__(self, api_key: str, environment: str, index_name: str):
+        pinecone.init(api_key=api_key, environment=environment)
+        self.index = pinecone.Index(index_name)
+
+    def upsert(self, documents: List[Document], batch_size: int = 100):
+        """Upsert documents in batches."""
+        vectors = [
+            (doc.metadata["id"], doc.embedding, doc.metadata)
+            for doc in documents
+        ]
+
+        for i in range(0, len(vectors), batch_size):
+            batch = vectors[i:i + batch_size]
+            self.index.upsert(vectors=batch)
+
+    def search(self, embedding: List[float], top_k: int = 5) -> List[RetrievalResult]:
+        """Search for similar vectors."""
+        results = self.index.query(
+            vector=embedding,
+            top_k=top_k,
+            include_metadata=True
+        )
+
+        return [
+            RetrievalResult(
+                document=Document(
+                    content=match.metadata.get("content", ""),
+                    metadata=match.metadata
+                ),
+                score=match.score
+            )
+            for match in results.matches
+        ]
+```
+
+---
+
+## Chunking Strategies
+
+### Strategy Comparison
+
+| Strategy | Chunk Size | Overlap | Best For |
+|----------|------------|---------|----------|
+| Fixed | 500-1000 tokens | 50-100 | General text |
+| Sentence | 3-5 sentences | 1 sentence | Structured text |
+| Paragraph | Natural breaks | None | Documents with clear structure |
+| Semantic | Variable | Based on meaning | Research papers |
+| Recursive | Hierarchical | Parent-child | Long documents |
+
+### Recursive Character Splitter
+
+```python
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+def create_chunks(
+    text: str,
+    chunk_size: int = 1000,
+    chunk_overlap: int = 100
+) -> List[str]:
+    """Split text using recursive character splitting."""
+    splitter = RecursiveCharacterTextSplitter(
+        chunk_size=chunk_size,
+        chunk_overlap=chunk_overlap,
+        separators=["\n\n", "\n", ". ", " ", ""]
+    )
+
+    return splitter.split_text(text)
+```
+
+### Semantic Chunking
+
+```python
+from sentence_transformers import SentenceTransformer
+import numpy as np
+
+def semantic_chunk(
+    sentences: List[str],
+    embedder: SentenceTransformer,
+    threshold: float = 0.7
+) -> List[List[str]]:
+    """Group sentences by semantic similarity."""
+    embeddings = embedder.encode(sentences)
+
+    chunks = []
+    current_chunk = [sentences[0]]
+    current_embedding = embeddings[0]
+
+    for i in range(1, len(sentences)):
+        similarity = np.dot(current_embedding, embeddings[i]) / (
+            np.linalg.norm(current_embedding) * np.linalg.norm(embeddings[i])
+        )
+
+        if similarity >= threshold:
+            current_chunk.append(sentences[i])
+            current_embedding = np.mean(
+                [current_embedding, embeddings[i]], axis=0
+            )
+        else:
+            chunks.append(current_chunk)
+            current_chunk = [sentences[i]]
+            current_embedding = embeddings[i]
+
+    chunks.append(current_chunk)
+    return chunks
+```
+
+---
+
+## Embedding Models
+
+### Model Comparison
+
+| Model | Dimensions | Quality | Speed | Cost |
+|-------|------------|---------|-------|------|
+| text-embedding-3-large | 3072 | Excellent | Medium | $0.13/1M |
+| text-embedding-3-small | 1536 | Good | Fast | $0.02/1M |
+| BGE-large | 1024 | Excellent | Medium | Free |
+| all-MiniLM-L6-v2 | 384 | Good | Very Fast | Free |
+| Cohere embed-v3 | 1024 | Excellent | Medium | $0.10/1M |
+
+### Embedding with Caching
+
+```python
+import hashlib
+from functools import lru_cache
+
+class CachedEmbedder:
+    def __init__(self, model_name: str = "text-embedding-3-small"):
+        self.client = OpenAI()
+        self.model = model_name
+        self._cache = {}
+
+    def embed(self, text: str) -> List[float]:
+        """Embed text with caching."""
+        cache_key = hashlib.md5(text.encode()).hexdigest()
+
+        if cache_key in self._cache:
+            return self._cache[cache_key]
+
+        response = self.client.embeddings.create(
+            model=self.model,
+            input=text
+        )
+
+        embedding = response.data[0].embedding
+        self._cache[cache_key] = embedding
+
+        return embedding
+
+    def embed_batch(self, texts: List[str]) -> List[List[float]]:
+        """Embed multiple texts efficiently."""
+        response = self.client.embeddings.create(
+            model=self.model,
+            input=texts
+        )
+
+        return [item.embedding for item in response.data]
+```
+
+---
+
+## Retrieval Optimization
+
+### Hybrid Search
+
+Combine dense (vector) and sparse (keyword) retrieval:
+
+```python
+from rank_bm25 import BM25Okapi
+
+class HybridRetriever:
+    def __init__(
+        self,
+        vector_store: VectorStore,
+        documents: List[Document],
+        alpha: float = 0.5
+    ):
+        self.vector_store = vector_store
+        self.alpha = alpha  # Weight for vector search
+
+        # Build BM25 index
+        tokenized = [doc.content.lower().split() for doc in documents]
+        self.bm25 = BM25Okapi(tokenized)
+        self.documents = documents
+
+    def search(self, query: str, query_embedding: List[float], top_k: int = 5):
+        # Vector search
+        vector_results = self.vector_store.search(query_embedding, top_k=top_k * 2)
+
+        # BM25 search
+        tokenized_query = query.lower().split()
+        bm25_scores = self.bm25.get_scores(tokenized_query)
+
+        # Combine scores
+        combined = {}
+        for result in vector_results:
+            doc_id = result.document.metadata["id"]
+            combined[doc_id] = self.alpha * result.score
+
+        for i, score in enumerate(bm25_scores):
+            doc_id = self.documents[i].metadata["id"]
+            if doc_id in combined:
+                combined[doc_id] += (1 - self.alpha) * score
+            else:
+                combined[doc_id] = (1 - self.alpha) * score
+
+        # Sort and return top_k
+        sorted_ids = sorted(combined.keys(), key=lambda x: combined[x], reverse=True)
+        return sorted_ids[:top_k]
+```
+
+### Reranking
+
+```python
+from sentence_transformers import CrossEncoder
+
+class Reranker:
+    def __init__(self, model_name: str = "cross-encoder/ms-marco-MiniLM-L-12-v2"):
+        self.model = CrossEncoder(model_name)
+
+    def rerank(
+        self,
+        query: str,
+        results: List[RetrievalResult],
+        top_k: int = 5
+    ) -> List[RetrievalResult]:
+        """Rerank results using cross-encoder."""
+        pairs = [(query, r.document.content) for r in results]
+        scores = self.model.predict(pairs)
+
+        # Update scores and sort
+        for i, score in enumerate(scores):
+            results[i].score = float(score)
+
+        return sorted(results, key=lambda x: x.score, reverse=True)[:top_k]
+```
+
+### Query Expansion
+
+```python
+def expand_query(query: str, llm: LLMProvider) -> List[str]:
+    """Generate query variations for better retrieval."""
+    prompt = f"""Generate 3 alternative phrasings of this question for search.
+Return only the questions, one per line.
+
+Original: {query}
+
+Alternatives:"""
+
+    response = llm.complete(prompt, max_tokens=150)
+    alternatives = [q.strip() for q in response.strip().split("\n") if q.strip()]
+
+    return [query] + alternatives[:3]
+```