omgkit 2.20.0 → 2.21.1

package/plugin/skills/ml-systems/ml-systems-fundamentals/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: ml-systems-fundamentals
+description: Core ML systems concepts including ML lifecycle, system architecture, requirements, and design principles for production ML.
+---
+
+# ML Systems Fundamentals
+
+Foundation concepts for building production ML systems.
+
+## ML System Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    ML SYSTEM ARCHITECTURE                    │
+├─────────────────────────────────────────────────────────────┤
+│                                                              │
+│  DATA LAYER                                                  │
+│  ├── Data Collection    ├── Data Storage                    │
+│  ├── Data Processing    └── Feature Store                   │
+│                                                              │
+│  MODEL LAYER                                                 │
+│  ├── Training Pipeline  ├── Experiment Tracking              │
+│  ├── Model Registry     └── Evaluation                      │
+│                                                              │
+│  SERVING LAYER                                               │
+│  ├── Model Serving      ├── Feature Serving                 │
+│  ├── Prediction Cache   └── Load Balancing                  │
+│                                                              │
+│  MONITORING LAYER                                            │
+│  ├── Data Monitoring    ├── Model Monitoring                │
+│  ├── System Metrics     └── Alerting                        │
+│                                                              │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## ML Lifecycle
+
+1. **Problem Definition** - Business goal → ML task
+2. **Data Collection** - Gather relevant data
+3. **Data Processing** - Clean, transform, validate
+4. **Feature Engineering** - Create informative features
+5. **Model Development** - Train, tune, evaluate
+6. **Deployment** - Serve predictions
+7. **Monitoring** - Track performance
+8. **Iteration** - Improve based on feedback
+
+## System Requirements
+
+### Reliability
+- Handle failures gracefully
+- Maintain prediction quality
+- Provide consistent latency
+
+### Scalability
+- Handle growing data
+- Support more requests
+- Enable parallel training
+
+### Maintainability
+- Easy to update models
+- Clear documentation
+- Reproducible experiments
+
+### Adaptability
+- Respond to data changes
+- Support new features
+- Enable quick iterations
+
+## Design Principles
+
+```python
+# 1. Start Simple
+baseline = LogisticRegression()
+baseline.fit(X_train, y_train)
+print(f"Baseline: {baseline.score(X_test, y_test)}")
+
+# 2. Data Quality > Model Complexity
+def validate_data(df):
+    assert df.isnull().sum().sum() == 0
+    assert df.duplicated().sum() == 0
+    return True
+
+# 3. Version Everything
+import mlflow
+mlflow.log_param("model_version", "1.0.0")
+mlflow.log_artifact("data/processed/")
+
+# 4. Monitor Continuously
+def check_drift(reference, current):
+    return ks_2samp(reference, current).pvalue < 0.05
+```
+
+## Commands
+- `/omgml:init` - Initialize ML project
+- `/omgml:status` - Project status
+
+## Best Practices
+
+1. Define clear success metrics
+2. Establish baselines early
+3. Invest in data quality
+4. Automate everything possible
+5. Monitor production models

package/plugin/skills/ml-systems/ml-workflow/SKILL.md

@@ -0,0 +1,162 @@
+---
+name: ml-workflow
+description: ML development workflow covering experiment design, baseline establishment, iterative improvement, and experiment tracking best practices.
+---
+
+# ML Workflow
+
+Systematic approach to ML model development.
+
+## Development Lifecycle
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                  ML DEVELOPMENT WORKFLOW                     │
+├─────────────────────────────────────────────────────────────┤
+│                                                              │
+│  1. PROBLEM      2. BASELINE     3. EXPERIMENT              │
+│     SETUP           MODEL           ITERATE                  │
+│     ↓               ↓               ↓                       │
+│  Define metrics  Simple model   Hypothesis                  │
+│  Success criteria Benchmark     Test ideas                  │
+│  Constraints     Comparison     Track results               │
+│                                                              │
+│  4. EVALUATE     5. VALIDATE    6. DEPLOY                   │
+│     ↓               ↓               ↓                       │
+│  Full metrics    Production    Ship to prod                │
+│  Error analysis  validation    Monitor                     │
+│  Fairness        A/B test      Iterate                     │
+│                                                              │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Experiment Design
+
+```python
+import mlflow
+from dataclasses import dataclass
+
+@dataclass
+class Experiment:
+    name: str
+    hypothesis: str
+    metrics: list
+    success_criteria: dict
+
+experiment = Experiment(
+    name="feature_engineering_v2",
+    hypothesis="Adding temporal features improves prediction",
+    metrics=["accuracy", "f1", "latency_ms"],
+    success_criteria={"f1": 0.85, "latency_ms": 50}
+)
+
+# Track experiment
+mlflow.set_experiment(experiment.name)
+with mlflow.start_run():
+    mlflow.log_param("hypothesis", experiment.hypothesis)
+    # ... training code ...
+    mlflow.log_metrics(results)
+```
+
+## Baseline Models
+
+```python
+from sklearn.dummy import DummyClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.ensemble import RandomForestClassifier
+
+baselines = {
+    "majority": DummyClassifier(strategy="most_frequent"),
+    "logistic": LogisticRegression(),
+    "random_forest": RandomForestClassifier(n_estimators=100)
+}
+
+results = {}
+for name, model in baselines.items():
+    model.fit(X_train, y_train)
+    y_pred = model.predict(X_test)
+    results[name] = {
+        "accuracy": accuracy_score(y_test, y_pred),
+        "f1": f1_score(y_test, y_pred, average="macro")
+    }
+
+# Best baseline
+best = max(results.items(), key=lambda x: x[1]["f1"])
+print(f"Best baseline: {best[0]} with F1={best[1]['f1']:.3f}")
+```
+
+## Experiment Tracking
+
+```python
+import mlflow
+import mlflow.pytorch
+
+# Start experiment
+mlflow.set_tracking_uri("http://mlflow.example.com")
+mlflow.set_experiment("churn_prediction")
+
+with mlflow.start_run(run_name="xgboost_v3"):
+    # Log parameters
+    mlflow.log_params({
+        "model_type": "xgboost",
+        "max_depth": 6,
+        "learning_rate": 0.1
+    })
+
+    # Train model
+    model = train_model(X_train, y_train, params)
+
+    # Log metrics
+    mlflow.log_metrics({
+        "train_accuracy": train_acc,
+        "val_accuracy": val_acc,
+        "f1_score": f1
+    })
+
+    # Log model
+    mlflow.sklearn.log_model(model, "model")
+
+    # Log artifacts
+    mlflow.log_artifact("feature_importance.png")
+```
+
+## Iterative Improvement
+
+```python
+class ExperimentIterator:
+    def __init__(self, baseline_metrics):
+        self.baseline = baseline_metrics
+        self.experiments = []
+
+    def run_experiment(self, name, model_fn, hypothesis):
+        with mlflow.start_run(run_name=name):
+            mlflow.log_param("hypothesis", hypothesis)
+            model, metrics = model_fn()
+            mlflow.log_metrics(metrics)
+
+            improvement = {k: metrics[k] - self.baseline[k]
+                          for k in metrics}
+            mlflow.log_metrics({f"{k}_improvement": v
+                              for k, v in improvement.items()})
+
+            self.experiments.append({
+                "name": name,
+                "hypothesis": hypothesis,
+                "metrics": metrics,
+                "improvement": improvement
+            })
+
+            return model, metrics
+```
+
+## Commands
+- `/omgml:init` - Initialize project
+- `/omgtrain:baseline` - Train baselines
+
+## Best Practices
+
+1. Always start with a baseline
+2. Change one thing at a time
+3. Track all experiments
+4. Document hypotheses
+5. Validate before deploying

package/plugin/skills/ml-systems/mlops/SKILL.md

@@ -0,0 +1,386 @@
+---
+name: mlops
+description: MLOps practices including CI/CD for ML, experiment tracking, model monitoring, pipeline orchestration, and production ML operations.
+---
+
+# MLOps
+
+Production ML operations and automation.
+
+## MLOps Maturity Model
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                   MLOPS MATURITY LEVELS                      │
+├─────────────────────────────────────────────────────────────┤
+│                                                              │
+│  LEVEL 0           LEVEL 1           LEVEL 2                │
+│  Manual            ML Pipeline       CI/CD for ML           │
+│  ───────           ──────────        ──────────             │
+│  Notebooks         Automated         Automated              │
+│  Manual deploy     training          retraining             │
+│  No monitoring     Basic pipeline    Full automation        │
+│                                                              │
+│  Components:                                                 │
+│  ├── Version Control (Git, DVC)                             │
+│  ├── Experiment Tracking (MLflow, W&B)                      │
+│  ├── Feature Store (Feast, Tecton)                          │
+│  ├── Model Registry (MLflow, Sagemaker)                     │
+│  ├── Orchestration (Airflow, Kubeflow)                      │
+│  └── Monitoring (Prometheus, Evidently)                     │
+│                                                              │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Experiment Tracking
+
+### MLflow Integration
+```python
+import mlflow
+from mlflow.tracking import MlflowClient
+
+# Set tracking server
+mlflow.set_tracking_uri("http://mlflow.example.com:5000")
+mlflow.set_experiment("churn_prediction")
+
+# Start run with context manager
+with mlflow.start_run(run_name="xgboost_v2") as run:
+    # Log parameters
+    mlflow.log_params({
+        "model_type": "xgboost",
+        "learning_rate": 0.1,
+        "max_depth": 6,
+        "n_estimators": 100
+    })
+
+    # Train model
+    model = train_model(params)
+
+    # Log metrics
+    mlflow.log_metrics({
+        "accuracy": 0.92,
+        "f1_score": 0.89,
+        "auc_roc": 0.95
+    })
+
+    # Log artifacts
+    mlflow.log_artifact("feature_importance.png")
+    mlflow.log_artifact("confusion_matrix.png")
+
+    # Log model
+    mlflow.sklearn.log_model(
+        model,
+        "model",
+        registered_model_name="churn_predictor"
+    )
+
+    # Log custom metrics over time
+    for epoch in range(100):
+        mlflow.log_metric("loss", train_loss, step=epoch)
+
+# Compare runs
+client = MlflowClient()
+runs = client.search_runs(
+    experiment_ids=["1"],
+    filter_string="metrics.f1_score > 0.85",
+    order_by=["metrics.f1_score DESC"]
+)
+```
+
+### Weights & Biases
+```python
+import wandb
+
+wandb.init(
+    project="ml-project",
+    config={
+        "learning_rate": 0.001,
+        "architecture": "ResNet50",
+        "epochs": 100
+    }
+)
+
+# Log metrics
+for epoch in range(100):
+    wandb.log({
+        "epoch": epoch,
+        "loss": train_loss,
+        "val_loss": val_loss,
+        "accuracy": accuracy
+    })
+
+# Log images
+wandb.log({"examples": [wandb.Image(img, caption=label) for img, label in samples]})
+
+# Log model
+wandb.save("model.pt")
+
+# Hyperparameter sweeps
+sweep_config = {
+    "method": "bayes",
+    "metric": {"name": "val_loss", "goal": "minimize"},
+    "parameters": {
+        "learning_rate": {"min": 0.0001, "max": 0.1},
+        "batch_size": {"values": [16, 32, 64]}
+    }
+}
+sweep_id = wandb.sweep(sweep_config)
+wandb.agent(sweep_id, train_function, count=50)
+```
+
+## Model Registry
+
+```python
+from mlflow.tracking import MlflowClient
+
+client = MlflowClient()
+
+# Register model
+model_uri = f"runs:/{run_id}/model"
+result = mlflow.register_model(model_uri, "production_model")
+
+# Transition stages
+client.transition_model_version_stage(
+    name="production_model",
+    version=result.version,
+    stage="Staging"
+)
+
+# Add description and tags
+client.update_model_version(
+    name="production_model",
+    version=result.version,
+    description="XGBoost model trained on Q4 data"
+)
+
+client.set_model_version_tag(
+    name="production_model",
+    version=result.version,
+    key="validation_status",
+    value="passed"
+)
+
+# Load production model
+model = mlflow.pyfunc.load_model("models:/production_model/Production")
+
+# Compare versions
+def compare_model_versions(model_name, version_a, version_b, test_data):
+    model_a = mlflow.pyfunc.load_model(f"models:/{model_name}/{version_a}")
+    model_b = mlflow.pyfunc.load_model(f"models:/{model_name}/{version_b}")
+
+    metrics_a = evaluate(model_a, test_data)
+    metrics_b = evaluate(model_b, test_data)
+
+    return {
+        "version_a": {"version": version_a, **metrics_a},
+        "version_b": {"version": version_b, **metrics_b}
+    }
+```
+
+## Pipeline Orchestration
+
+### Airflow DAG
+```python
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from datetime import datetime, timedelta
+
+default_args = {
+    'owner': 'ml-team',
+    'depends_on_past': False,
+    'email_on_failure': True,
+    'retries': 3,
+    'retry_delay': timedelta(minutes=5)
+}
+
+dag = DAG(
+    'ml_training_pipeline',
+    default_args=default_args,
+    schedule_interval='@daily',
+    start_date=datetime(2024, 1, 1),
+    catchup=False
+)
+
+# Tasks
+extract_data = PythonOperator(
+    task_id='extract_data',
+    python_callable=extract_training_data,
+    dag=dag
+)
+
+validate_data = PythonOperator(
+    task_id='validate_data',
+    python_callable=validate_data_quality,
+    dag=dag
+)
+
+train_model = PythonOperator(
+    task_id='train_model',
+    python_callable=train_and_log_model,
+    dag=dag
+)
+
+evaluate_model = PythonOperator(
+    task_id='evaluate_model',
+    python_callable=evaluate_model_performance,
+    dag=dag
+)
+
+deploy_model = PythonOperator(
+    task_id='deploy_model',
+    python_callable=deploy_to_production,
+    dag=dag
+)
+
+# Dependencies
+extract_data >> validate_data >> train_model >> evaluate_model >> deploy_model
+```
+
+### Kubeflow Pipeline
+```python
+from kfp import dsl
+from kfp.components import create_component_from_func
+
+@create_component_from_func
+def preprocess_data(input_path: str, output_path: str):
+    import pandas as pd
+    df = pd.read_csv(input_path)
+    # Preprocessing logic
+    df.to_parquet(output_path)
+
+@create_component_from_func
+def train_model(data_path: str, model_path: str, hyperparameters: dict):
+    import joblib
+    from sklearn.ensemble import RandomForestClassifier
+    # Training logic
+    model = RandomForestClassifier(**hyperparameters)
+    joblib.dump(model, model_path)
+
+@dsl.pipeline(
+    name='ML Training Pipeline',
+    description='End-to-end ML training pipeline'
+)
+def ml_pipeline(input_data: str, hyperparameters: dict):
+    preprocess_op = preprocess_data(input_data, '/tmp/processed.parquet')
+
+    train_op = train_model(
+        preprocess_op.output,
+        '/tmp/model.joblib',
+        hyperparameters
+    )
+
+    # Add GPU resources
+    train_op.set_gpu_limit(1)
+    train_op.set_memory_limit('8Gi')
+```
+
+## CI/CD for ML
+
+```yaml
+# .github/workflows/ml-pipeline.yml
+name: ML Pipeline
+
+on:
+  push:
+    paths:
+      - 'src/**'
+      - 'data/**'
+  schedule:
+    - cron: '0 0 * * 0'  # Weekly retraining
+
+jobs:
+  data-validation:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Validate data
+        run: |
+          python -m pytest tests/data_validation/
+          dvc pull
+          great_expectations checkpoint run data_quality
+
+  train:
+    needs: data-validation
+    runs-on: [self-hosted, gpu]
+    steps:
+      - uses: actions/checkout@v3
+      - name: Train model
+        run: |
+          python train.py --config configs/production.yaml
+          mlflow run . -P epochs=100
+
+  evaluate:
+    needs: train
+    runs-on: ubuntu-latest
+    steps:
+      - name: Evaluate model
+        run: |
+          python evaluate.py --model-version ${{ github.sha }}
+          python check_performance_regression.py
+
+  deploy:
+    needs: evaluate
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Deploy to staging
+        run: |
+          kubectl apply -f k8s/staging/
+          python smoke_test.py --env staging
+
+      - name: Deploy to production
+        run: |
+          kubectl apply -f k8s/production/
+          python smoke_test.py --env production
+```
+
+## Data Version Control
+
+```bash
+# Initialize DVC
+dvc init
+dvc remote add -d storage s3://my-bucket/dvc-storage
+
+# Track data files
+dvc add data/training.csv
+git add data/training.csv.dvc data/.gitignore
+git commit -m "Add training data"
+
+# Push data
+dvc push
+
+# Create pipeline
+dvc run -n preprocess \
+  -d src/preprocess.py -d data/raw.csv \
+  -o data/processed.csv \
+  python src/preprocess.py
+
+dvc run -n train \
+  -d src/train.py -d data/processed.csv \
+  -o models/model.pkl \
+  -M metrics.json \
+  python src/train.py
+
+# Reproduce pipeline
+dvc repro
+
+# Compare experiments
+dvc exp run --set-param train.lr=0.001
+dvc exp show
+dvc exp diff
+```
+
+## Commands
+- `/omgops:pipeline` - Pipeline management
+- `/omgops:registry` - Model registry
+- `/omgops:monitor` - System monitoring
+- `/omgml:status` - Project status
+
+## Best Practices
+
+1. Version everything (code, data, models)
+2. Automate training pipelines
+3. Implement quality gates
+4. Track all experiments
+5. Use feature stores for consistency