omgkit 2.20.0 → 2.21.1

package/plugin/skills/ml-systems/data-eng/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: data-engineering
+description: ML data engineering covering data pipelines, data quality, collection strategies, storage, and versioning for machine learning systems.
+---
+
+# Data Engineering for ML
+
+Building robust data infrastructure for ML systems.
+
+## Data Pipeline Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                   ML DATA PIPELINE                           │
+├─────────────────────────────────────────────────────────────┤
+│                                                              │
+│  COLLECTION  →  VALIDATION  →  PROCESSING  →  STORAGE       │
+│     ↓              ↓              ↓              ↓          │
+│  Sources      Schema Check    Transform      Data Lake      │
+│  APIs         Quality Check   Normalize      Feature Store  │
+│  DBs          Statistics      Encode         Model Registry │
+│                                                              │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Data Collection
+
+```python
+from dataclasses import dataclass
+from typing import List, Dict
+
+@dataclass
+class DataSource:
+    name: str
+    type: str  # database, api, file, stream
+    connection: Dict
+
+class DataCollector:
+    def __init__(self, sources: List[DataSource]):
+        self.sources = sources
+
+    def collect(self, source_name: str) -> pd.DataFrame:
+        source = next(s for s in self.sources if s.name == source_name)
+
+        if source.type == "database":
+            return pd.read_sql(source.connection["query"],
+                             source.connection["conn"])
+        elif source.type == "api":
+            response = requests.get(source.connection["url"])
+            return pd.DataFrame(response.json())
+        elif source.type == "file":
+            return pd.read_parquet(source.connection["path"])
+```
+
+## Data Quality
+
+```python
+import great_expectations as ge
+
+def validate_data(df: pd.DataFrame, expectations_path: str) -> bool:
+    ge_df = ge.from_pandas(df)
+
+    # Schema validation
+    assert ge_df.expect_column_to_exist("user_id").success
+    assert ge_df.expect_column_values_to_not_be_null("user_id").success
+    assert ge_df.expect_column_values_to_be_unique("user_id").success
+
+    # Value validation
+    assert ge_df.expect_column_values_to_be_between(
+        "age", min_value=0, max_value=150
+    ).success
+
+    # Statistical validation
+    assert ge_df.expect_column_mean_to_be_between(
+        "purchase_amount", min_value=0, max_value=10000
+    ).success
+
+    return True
+```
+
+## Data Versioning
+
+```python
+# DVC for data versioning
+# dvc init
+# dvc add data/processed/
+
+import dvc.api
+
+# Load specific version
+data_url = dvc.api.get_url(
+    path='data/processed/train.parquet',
+    repo='https://github.com/org/repo',
+    rev='v1.2.0'
+)
+
+# Track changes
+def version_data(data_path: str, message: str):
+    import subprocess
+    subprocess.run(["dvc", "add", data_path])
+    subprocess.run(["git", "add", f"{data_path}.dvc"])
+    subprocess.run(["git", "commit", "-m", message])
+    subprocess.run(["dvc", "push"])
+```
+
+## Data Storage Patterns
+
+| Pattern | Use Case | Technology |
+|---------|----------|------------|
+| Data Lake | Raw storage | S3, GCS, ADLS |
+| Data Warehouse | Analytics | Snowflake, BigQuery |
+| Feature Store | ML features | Feast, Tecton |
+| Vector Store | Embeddings | Pinecone, Weaviate |
+
+## Commands
+- `/omgdata:collect` - Data collection
+- `/omgdata:validate` - Data validation
+- `/omgdata:version` - Version data
+
+## Best Practices
+
+1. Validate data at every stage
+2. Version all data assets
+3. Document data schemas
+4. Monitor data quality metrics
+5. Implement data lineage tracking

package/plugin/skills/ml-systems/deep-learning-primer/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: deep-learning-primer
+description: Deep learning foundations including neural network basics, backpropagation, optimization, regularization, and training best practices.
+---
+
+# Deep Learning Primer
+
+Essential deep learning concepts for ML systems.
+
+## Neural Network Basics
+
+```python
+import torch
+import torch.nn as nn
+
+class MLP(nn.Module):
+    def __init__(self, input_dim, hidden_dim, output_dim):
+        super().__init__()
+        self.layers = nn.Sequential(
+            nn.Linear(input_dim, hidden_dim),
+            nn.ReLU(),
+            nn.Dropout(0.2),
+            nn.Linear(hidden_dim, hidden_dim),
+            nn.ReLU(),
+            nn.Dropout(0.2),
+            nn.Linear(hidden_dim, output_dim)
+        )
+
+    def forward(self, x):
+        return self.layers(x)
+```
+
+## Activation Functions
+
+| Function | Use Case | Output Range |
+|----------|----------|--------------|
+| ReLU | Hidden layers | [0, ∞) |
+| LeakyReLU | Prevent dead neurons | (-∞, ∞) |
+| GELU | Transformers | (-∞, ∞) |
+| Sigmoid | Binary classification | (0, 1) |
+| Softmax | Multi-class | (0, 1) |
+| Tanh | Centered output | (-1, 1) |
+
+## Loss Functions
+
+```python
+# Classification
+cross_entropy = nn.CrossEntropyLoss()
+binary_ce = nn.BCEWithLogitsLoss()
+focal_loss = lambda p, y, gamma=2: -((1-p)**gamma * y * torch.log(p))
+
+# Regression
+mse = nn.MSELoss()
+mae = nn.L1Loss()
+huber = nn.HuberLoss()
+
+# Custom loss
+def contrastive_loss(embeddings, labels, margin=1.0):
+    distances = torch.cdist(embeddings, embeddings)
+    same_class = labels.unsqueeze(0) == labels.unsqueeze(1)
+    loss = same_class * distances + (~same_class) * F.relu(margin - distances)
+    return loss.mean()
+```
+
+## Optimization
+
+```python
+# Optimizers
+optimizer = torch.optim.AdamW(
+    model.parameters(),
+    lr=1e-3,
+    weight_decay=0.01
+)
+
+# Learning rate scheduling
+scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
+    optimizer,
+    T_max=num_epochs
+)
+
+# Training loop
+for epoch in range(num_epochs):
+    for batch in dataloader:
+        optimizer.zero_grad()
+        loss = criterion(model(batch.x), batch.y)
+        loss.backward()
+        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+        optimizer.step()
+    scheduler.step()
+```
+
+## Regularization Techniques
+
+```python
+# Dropout
+nn.Dropout(p=0.5)
+
+# Weight decay (L2)
+optimizer = AdamW(params, weight_decay=0.01)
+
+# Batch normalization
+nn.BatchNorm1d(num_features)
+
+# Layer normalization (for transformers)
+nn.LayerNorm(hidden_dim)
+
+# Early stopping
+class EarlyStopping:
+    def __init__(self, patience=5):
+        self.patience = patience
+        self.counter = 0
+        self.best_score = None
+
+    def __call__(self, val_loss):
+        if self.best_score is None or val_loss < self.best_score:
+            self.best_score = val_loss
+            self.counter = 0
+        else:
+            self.counter += 1
+        return self.counter >= self.patience
+```
+
+## Initialization
+
+```python
+def init_weights(module):
+    if isinstance(module, nn.Linear):
+        nn.init.xavier_uniform_(module.weight)
+        if module.bias is not None:
+            nn.init.zeros_(module.bias)
+    elif isinstance(module, nn.Embedding):
+        nn.init.normal_(module.weight, std=0.02)
+
+model.apply(init_weights)
+```
+
+## Best Practices
+
+1. Start with Adam optimizer, lr=1e-3
+2. Use learning rate warmup
+3. Apply gradient clipping
+4. Monitor training/validation gap
+5. Use mixed precision training

package/plugin/skills/ml-systems/deployment-paradigms/SKILL.md

@@ -0,0 +1,148 @@
+---
+name: deployment-paradigms
+description: ML deployment paradigms including batch vs real-time inference, online vs offline serving, edge deployment, and serverless ML.
+---
+
+# Deployment Paradigms
+
+Understanding ML deployment patterns and trade-offs.
+
+## Deployment Modes
+
+### Batch Inference
+```python
+# Process all data at once
+def batch_inference(model, data_path, output_path):
+    data = pd.read_parquet(data_path)
+    predictions = model.predict(data)
+    predictions.to_parquet(output_path)
+
+# Schedule: Daily/Hourly
+# Airflow DAG example
+with DAG('batch_inference', schedule_interval='@daily') as dag:
+    inference_task = PythonOperator(
+        task_id='run_inference',
+        python_callable=batch_inference
+    )
+```
+
+### Real-time Inference
+```python
+from fastapi import FastAPI
+import torch
+
+app = FastAPI()
+model = torch.jit.load("model.pt")
+
+@app.post("/predict")
+async def predict(request: PredictRequest):
+    features = preprocess(request.data)
+    with torch.no_grad():
+        prediction = model(features)
+    return {"prediction": prediction.tolist()}
+```
+
+### Streaming Inference
+```python
+from kafka import KafkaConsumer, KafkaProducer
+
+consumer = KafkaConsumer('input-topic')
+producer = KafkaProducer()
+
+for message in consumer:
+    data = deserialize(message.value)
+    prediction = model.predict(data)
+    producer.send('output-topic', serialize(prediction))
+```
+
+## Serving Patterns
+
+### Online Serving
+- Sub-second latency
+- Feature store for features
+- Model caching
+- Auto-scaling
+
+### Offline Serving
+- Batch processing
+- Precomputed predictions
+- Lower cost
+- Higher throughput
+
+### Hybrid Serving
+```python
+class HybridPredictor:
+    def __init__(self, cache_ttl=3600):
+        self.cache = {}
+        self.cache_ttl = cache_ttl
+        self.model = load_model()
+
+    def predict(self, user_id, context):
+        # Check precomputed cache
+        cache_key = f"{user_id}:{hash(context)}"
+        if cache_key in self.cache:
+            return self.cache[cache_key]
+
+        # Compute real-time
+        prediction = self.model.predict(context)
+        self.cache[cache_key] = prediction
+        return prediction
+```
+
+## Edge Deployment
+
+```python
+# TFLite for mobile/embedded
+import tensorflow as tf
+
+converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
+converter.optimizations = [tf.lite.Optimize.DEFAULT]
+tflite_model = converter.convert()
+
+# ONNX for cross-platform
+import onnx
+torch.onnx.export(model, dummy_input, "model.onnx")
+
+# CoreML for iOS
+import coremltools as ct
+mlmodel = ct.convert(model, inputs=[ct.TensorType(shape=(1, 3, 224, 224))])
+```
+
+## Serverless ML
+
+```python
+# AWS Lambda
+import json
+import boto3
+
+def lambda_handler(event, context):
+    runtime = boto3.client('runtime.sagemaker')
+    response = runtime.invoke_endpoint(
+        EndpointName='my-model',
+        ContentType='application/json',
+        Body=json.dumps(event['body'])
+    )
+    return json.loads(response['Body'].read())
+```
+
+## Deployment Comparison
+
+| Pattern | Latency | Cost | Complexity | Use Case |
+|---------|---------|------|------------|----------|
+| Batch | High | Low | Low | Reports, ETL |
+| Real-time | Low | High | Medium | User-facing |
+| Streaming | Medium | Medium | High | Event-driven |
+| Edge | Very Low | Low | High | Offline, IoT |
+| Serverless | Variable | Pay-per-use | Low | Sporadic traffic |
+
+## Commands
+- `/omgdeploy:serve` - Deploy serving
+- `/omgdeploy:edge` - Edge deployment
+
+## Best Practices
+
+1. Match paradigm to requirements
+2. Consider latency vs cost trade-offs
+3. Plan for scaling
+4. Test under realistic conditions
+5. Monitor deployed models

package/plugin/skills/ml-systems/dnn-architectures/SKILL.md

@@ -0,0 +1,128 @@
+---
+name: dnn-architectures
+description: Deep neural network architectures including CNNs, RNNs, Transformers, and modern architectures for vision, NLP, and multimodal tasks.
+---
+
+# DNN Architectures
+
+Modern deep neural network architectures.
+
+## Convolutional Neural Networks
+
+```python
+import torch.nn as nn
+
+class CNN(nn.Module):
+    def __init__(self, num_classes=10):
+        super().__init__()
+        self.features = nn.Sequential(
+            nn.Conv2d(3, 64, kernel_size=3, padding=1),
+            nn.BatchNorm2d(64),
+            nn.ReLU(),
+            nn.MaxPool2d(2),
+            nn.Conv2d(64, 128, kernel_size=3, padding=1),
+            nn.BatchNorm2d(128),
+            nn.ReLU(),
+            nn.MaxPool2d(2),
+            nn.Conv2d(128, 256, kernel_size=3, padding=1),
+            nn.BatchNorm2d(256),
+            nn.ReLU(),
+            nn.AdaptiveAvgPool2d(1)
+        )
+        self.classifier = nn.Linear(256, num_classes)
+
+    def forward(self, x):
+        x = self.features(x)
+        x = x.view(x.size(0), -1)
+        return self.classifier(x)
+```
+
+## Transformer Architecture
+
+```python
+class TransformerBlock(nn.Module):
+    def __init__(self, d_model, n_heads, d_ff, dropout=0.1):
+        super().__init__()
+        self.attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout)
+        self.ff = nn.Sequential(
+            nn.Linear(d_model, d_ff),
+            nn.GELU(),
+            nn.Dropout(dropout),
+            nn.Linear(d_ff, d_model)
+        )
+        self.ln1 = nn.LayerNorm(d_model)
+        self.ln2 = nn.LayerNorm(d_model)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, x, mask=None):
+        # Self-attention with residual
+        attn_out, _ = self.attn(x, x, x, attn_mask=mask)
+        x = self.ln1(x + self.dropout(attn_out))
+        # Feedforward with residual
+        ff_out = self.ff(x)
+        x = self.ln2(x + self.dropout(ff_out))
+        return x
+```
+
+## Vision Transformer (ViT)
+
+```python
+class ViT(nn.Module):
+    def __init__(self, image_size, patch_size, num_classes, d_model, n_heads, n_layers):
+        super().__init__()
+        num_patches = (image_size // patch_size) ** 2
+        self.patch_embed = nn.Conv2d(3, d_model, patch_size, patch_size)
+        self.cls_token = nn.Parameter(torch.zeros(1, 1, d_model))
+        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, d_model))
+        self.transformer = nn.ModuleList([
+            TransformerBlock(d_model, n_heads, d_model * 4)
+            for _ in range(n_layers)
+        ])
+        self.head = nn.Linear(d_model, num_classes)
+
+    def forward(self, x):
+        patches = self.patch_embed(x).flatten(2).transpose(1, 2)
+        cls_tokens = self.cls_token.expand(x.size(0), -1, -1)
+        x = torch.cat([cls_tokens, patches], dim=1)
+        x = x + self.pos_embed
+        for block in self.transformer:
+            x = block(x)
+        return self.head(x[:, 0])
+```
+
+## Architecture Comparison
+
+| Architecture | Best For | Params | Inference |
+|--------------|----------|--------|-----------|
+| ResNet | Image classification | 25M | Fast |
+| EfficientNet | Efficient vision | 5-66M | Efficient |
+| ViT | Vision + scale | 86-632M | GPU optimized |
+| BERT | NLP understanding | 110-340M | Moderate |
+| GPT | Text generation | 117M-175B | Heavy |
+| T5 | Seq2seq tasks | 60M-11B | Heavy |
+
+## Modern Architectures
+
+```python
+# Using pretrained models
+from transformers import AutoModel
+
+# Vision
+vit = AutoModel.from_pretrained("google/vit-base-patch16-224")
+clip = AutoModel.from_pretrained("openai/clip-vit-base-patch32")
+
+# NLP
+bert = AutoModel.from_pretrained("bert-base-uncased")
+llama = AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf")
+
+# Multimodal
+blip = AutoModel.from_pretrained("Salesforce/blip-image-captioning-base")
+```
+
+## Best Practices
+
+1. Use pretrained models when possible
+2. Match architecture to task
+3. Consider compute budget
+4. Scale model size with data size
+5. Monitor memory usage