omgkit 2.20.0 → 2.21.0

package/plugin/commands/omgml/status.md

@@ -0,0 +1,82 @@
+---
+description: Display comprehensive ML project status including data, model, deployment, and pipeline health
+allowed-tools: Task, Read, Bash, Grep, Glob
+argument-hint: "[--detailed]"
+---
+
+# ML Project Status: $ARGUMENTS
+
+Show ML project status: **$ARGUMENTS**
+
+## Agent
+Uses **orchestrator-agent** for comprehensive status collection.
+
+## Parameters
+- **--detailed**: Show extended information for each component
+
+## Status Components
+
+### Data Status
+- Training set size and last update
+- Validation/Test set sizes
+- Data version (DVC)
+- Data quality metrics
+
+### Model Status
+- Current model version
+- Model architecture
+- Performance metrics (accuracy, F1, etc.)
+- Last training timestamp
+
+### Deployment Status
+- Production model version
+- Staging model version
+- Endpoint health
+- Request latency
+
+### Pipeline Status
+- Last pipeline run
+- Success/failure rate
+- Next scheduled run
+- Active experiments
+
+## Example Output
+```
+ML Project Status
+═══════════════════════════════════════════════════
+
+Project: customer-churn-prediction
+Type: Classification (Binary)
+Created: 2024-01-15
+
+DATA STATUS
+├── Training: 50,000 samples (updated: 2h ago)
+├── Validation: 10,000 samples
+├── Test: 10,000 samples
+└── Data Version: v1.2.3
+
+MODEL STATUS
+├── Current: XGBoost v2.1.0
+├── Accuracy: 94.2%
+├── F1-Score: 0.89
+└── Last trained: 1d ago
+
+DEPLOYMENT STATUS
+├── Production: v2.0.0 (deployed 3d ago)
+├── Staging: v2.1.0 (testing)
+└── Endpoint: https://api.example.com/predict
+
+PIPELINE STATUS
+├── Last run: 2h ago ✅
+├── Success rate: 98.5%
+└── Next scheduled: in 22h
+```
+
+## Progress
+- [ ] Data status collected
+- [ ] Model status collected
+- [ ] Deployment status collected
+- [ ] Pipeline status collected
+- [ ] Report generated
+
+Provide actionable insights for project health.

package/plugin/commands/omgops/drift.md

@@ -0,0 +1,87 @@
+---
+description: Detect and analyze data drift, label drift, and concept drift with statistical methods
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <drift_type> [--reference <ref>] [--current <current>]
+---
+
+# Drift Detection: $ARGUMENTS
+
+Detect drift: **$ARGUMENTS**
+
+## Agent
+Uses **monitoring-agent** for drift detection.
+
+## Parameters
+- **drift_type**: feature | label | concept | all
+- **reference**: Path to reference data
+- **current**: Path to current data
+
+## Drift Types
+
+### Feature Drift
+- Input distribution change
+- P(X) has changed
+- Methods: KS test, PSI, Wasserstein
+
+### Label Drift
+- Target distribution change
+- P(Y) has changed
+- Methods: Chi-square, JS divergence
+
+### Concept Drift
+- P(Y|X) relationship change
+- Model degradation
+- Methods: Performance monitoring, DDM, ADWIN
+
+## Code Template
+```python
+from omgkit.mlops import DriftDetector
+
+detector = DriftDetector()
+
+drift_report = detector.detect(
+    reference_data="data/reference/baseline.parquet",
+    current_data="data/production/current_week.parquet",
+    drift_types=["feature", "label", "concept"],
+    methods={
+        "feature": "ks_test",
+        "label": "chi_square",
+        "concept": "model_performance"
+    },
+    model_path="models/production/model.pt",
+    threshold=0.05
+)
+
+print(drift_report.summary())
+
+if drift_report.drift_detected:
+    detector.trigger_retraining()
+```
+
+## Statistical Methods
+- **KS Test**: Kolmogorov-Smirnov
+- **PSI**: Population Stability Index
+- **Wasserstein**: Earth Mover's Distance
+- **Chi-Square**: Categorical data
+- **JS Divergence**: Jensen-Shannon
+
+## Actions on Drift
+- Alert notification
+- Trigger retraining
+- Rollback model
+- Increase monitoring
+
+## Report Output
+- Drift scores per feature
+- Statistical significance
+- Trend visualization
+- Recommended actions
+
+## Progress
+- [ ] Data loaded
+- [ ] Drift computed
+- [ ] Analysis complete
+- [ ] Report generated
+- [ ] Actions triggered
+
+Proactively detect model degradation causes.

package/plugin/commands/omgops/monitor.md

@@ -0,0 +1,99 @@
+---
+description: Setup ML monitoring and alerting for data quality, drift detection, and model performance
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: "[--metrics <metrics>] [--alerts <alerts>]"
+---
+
+# ML Monitoring: $ARGUMENTS
+
+Setup monitoring: **$ARGUMENTS**
+
+## Agent
+Uses **monitoring-agent** for comprehensive ML monitoring.
+
+## Parameters
+- **metrics**: List of metrics to monitor
+- **alerts**: Alert configurations
+
+## Monitoring Types
+
+### Data Quality
+- Null rate monitoring
+- Schema violations
+- Value distributions
+- Anomaly detection
+
+### Data Drift
+- Feature drift
+- Label drift
+- Concept drift
+- Statistical tests
+
+### Model Performance
+- Accuracy degradation
+- Latency increase
+- Error rate
+- Throughput
+
+### System Metrics
+- Memory usage
+- CPU utilization
+- Request rate
+- Response time
+
+## Code Template
+```python
+from omgkit.mlops import MonitoringSetup
+
+monitor = MonitoringSetup()
+
+monitor.setup(
+    model_endpoint="https://api.example.com/predict",
+    metrics=[
+        {"type": "null_rate", "column": "user_id", "threshold": 0.01},
+        {"type": "feature_drift", "method": "ks_test", "threshold": 0.05},
+        {"type": "concept_drift", "window": "7d"},
+        {"type": "accuracy", "reference": 0.94, "threshold": 0.02},
+        {"type": "latency_p99", "threshold_ms": 100},
+        {"type": "memory_usage", "threshold_pct": 80},
+        {"type": "error_rate", "threshold": 0.01}
+    ],
+    alerts=[
+        {
+            "name": "accuracy_drop",
+            "condition": "accuracy < 0.90",
+            "channel": "slack",
+            "severity": "critical"
+        },
+        {
+            "name": "high_latency",
+            "condition": "latency_p99 > 200",
+            "channel": "pagerduty",
+            "severity": "warning"
+        }
+    ],
+    dashboard="grafana"
+)
+```
+
+## Alert Channels
+- Slack
+- PagerDuty
+- Email
+- Webhooks
+- SMS
+
+## Dashboard Features
+- Real-time metrics
+- Historical trends
+- Anomaly highlighting
+- Drill-down analysis
+
+## Progress
+- [ ] Metrics configured
+- [ ] Alerts defined
+- [ ] Dashboard created
+- [ ] Integration tested
+- [ ] Monitoring active
+
+Ensure production ML system health and reliability.

package/plugin/commands/omgops/pipeline.md

@@ -0,0 +1,102 @@
+---
+description: Create CI/CD pipeline for ML using GitHub Actions, GitLab CI, Jenkins, Kubeflow, or Airflow
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <platform> [--template <template>]
+---
+
+# ML Pipeline: $ARGUMENTS
+
+Create pipeline: **$ARGUMENTS**
+
+## Agent
+Uses **mlops-engineer-agent** for pipeline creation.
+
+## Parameters
+- **platform**: github_actions | gitlab_ci | jenkins | kubeflow | airflow
+- **template**: training | deployment | full (default: full)
+
+## Pipeline Types
+
+### Training Pipeline
+- Data validation
+- Feature engineering
+- Model training
+- Model evaluation
+- Model registration
+
+### Deployment Pipeline
+- Model validation
+- Canary deployment
+- Integration tests
+- Production promotion
+
+### Full Pipeline
+- Complete CI/CD
+- End-to-end automation
+- Monitoring integration
+
+## Code Template
+```python
+from omgkit.mlops import PipelineBuilder
+
+builder = PipelineBuilder(platform="github_actions")
+
+pipeline = builder.create_pipeline(
+    name="ml-training-pipeline",
+    stages=[
+        {
+            "name": "data_validation",
+            "script": "scripts/validate_data.py",
+            "triggers": ["data_push"]
+        },
+        {
+            "name": "training",
+            "script": "scripts/train.py",
+            "resources": {"gpu": True, "memory": "16Gi"},
+            "triggers": ["data_validation_success"]
+        },
+        {
+            "name": "evaluation",
+            "script": "scripts/evaluate.py",
+            "triggers": ["training_success"]
+        },
+        {
+            "name": "deployment",
+            "script": "scripts/deploy.py",
+            "triggers": ["evaluation_pass"],
+            "conditions": ["accuracy > 0.9"]
+        }
+    ]
+)
+
+pipeline.save(".github/workflows/ml-pipeline.yml")
+```
+
+## Platform Features
+
+### GitHub Actions
+- Matrix builds
+- Self-hosted runners
+- Secrets management
+- Artifact caching
+
+### Kubeflow
+- Kubernetes native
+- GPU scheduling
+- Experiment tracking
+- Pipeline versioning
+
+### Airflow
+- DAG scheduling
+- XCom data passing
+- Sensor triggers
+- Backfill support
+
+## Progress
+- [ ] Platform configured
+- [ ] Stages defined
+- [ ] Triggers set
+- [ ] Pipeline generated
+- [ ] Tests added
+
+Automate ML workflows with CI/CD best practices.

package/plugin/commands/omgops/registry.md

@@ -0,0 +1,109 @@
+---
+description: Model registry operations - register, list, promote, archive, and rollback model versions
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <action> [--model <model>] [--version <version>]
+---
+
+# Model Registry: $ARGUMENTS
+
+Registry operation: **$ARGUMENTS**
+
+## Agent
+Uses **mlops-engineer-agent** for model registry management.
+
+## Parameters
+- **action**: register | list | promote | archive | rollback
+- **model**: Model name
+- **version**: Model version
+
+## Registry Actions
+
+### Register
+- Add new model version
+- Store metadata
+- Link artifacts
+- Tag for tracking
+
+### List
+- Show all versions
+- Filter by stage
+- Display metrics
+- Show lineage
+
+### Promote
+- Move to next stage
+- Staging → Production
+- Update endpoints
+- Notify stakeholders
+
+### Archive
+- Deprecate old versions
+- Preserve artifacts
+- Update documentation
+
+### Rollback
+- Revert to previous version
+- Restore endpoint
+- Emergency recovery
+
+## Code Template
+```python
+from omgkit.mlops import ModelRegistry
+import mlflow
+
+registry = ModelRegistry(tracking_uri="http://mlflow.example.com")
+
+# Register new model
+registry.register(
+    model_path="models/best_model.pt",
+    model_name="churn_predictor",
+    tags={
+        "framework": "pytorch",
+        "task": "classification",
+        "team": "data-science"
+    },
+    metrics={
+        "accuracy": 0.94,
+        "f1": 0.89,
+        "latency_ms": 15
+    }
+)
+
+# Promote to production
+registry.promote(
+    model_name="churn_predictor",
+    version="2.1.0",
+    stage="Production"
+)
+
+# Rollback if needed
+registry.rollback(
+    model_name="churn_predictor",
+    to_version="2.0.0"
+)
+
+# List all versions
+versions = registry.list_versions("churn_predictor")
+```
+
+## Model Stages
+- **None**: Development
+- **Staging**: Testing
+- **Production**: Live
+- **Archived**: Deprecated
+
+## Metadata Stored
+- Training parameters
+- Performance metrics
+- Data version
+- Code commit
+- Environment specs
+
+## Progress
+- [ ] Action validated
+- [ ] Registry connected
+- [ ] Operation executed
+- [ ] Endpoints updated
+- [ ] Notifications sent
+
+Centralize model lifecycle management.

package/plugin/commands/omgops/retrain.md

@@ -0,0 +1,91 @@
+---
+description: Trigger model retraining with full, incremental, or transfer learning strategies
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <strategy> [--trigger <trigger>]
+---
+
+# Model Retraining: $ARGUMENTS
+
+Trigger retraining: **$ARGUMENTS**
+
+## Agent
+Uses **mlops-engineer-agent** for retraining orchestration.
+
+## Parameters
+- **strategy**: full | incremental | transfer (default: incremental)
+- **trigger**: manual | scheduled | drift_detected | performance_drop
+
+## Retraining Strategies
+
+### Full Retraining
+- Train from scratch
+- All historical data
+- New architecture possible
+- Use case: Major updates
+
+### Incremental Training
+- Update with new data
+- Stateful training
+- Faster convergence
+- Use case: Regular updates
+
+### Transfer Learning
+- Fine-tune from checkpoint
+- Domain adaptation
+- New task learning
+- Use case: Limited new data
+
+## Code Template
+```python
+from omgkit.mlops import RetrainingPipeline
+
+pipeline = RetrainingPipeline()
+
+pipeline.setup(
+    strategy="incremental",
+    triggers=[
+        {"type": "scheduled", "cron": "0 0 * * 0"},  # Weekly
+        {"type": "drift_detected", "threshold": 0.1},
+        {"type": "performance_drop", "metric": "accuracy", "threshold": 0.05}
+    ],
+    data_config={
+        "new_data_path": "data/production/new/",
+        "validation_data": "data/splits/val.parquet",
+        "min_new_samples": 1000
+    },
+    training_config={
+        "epochs": 10,
+        "learning_rate": 1e-4,
+        "early_stopping": True
+    },
+    validation_config={
+        "metrics": ["accuracy", "f1"],
+        "minimum_improvement": 0.01,
+        "fallback_on_regression": True
+    }
+)
+
+# Manual trigger
+pipeline.trigger(reason="manual_update")
+```
+
+## Safety Checks
+- Minimum improvement threshold
+- Regression detection
+- Automatic rollback
+- Shadow deployment
+
+## Triggers
+- **Scheduled**: Cron-based
+- **Drift**: Automatic on drift
+- **Performance**: Metrics degradation
+- **Data**: New data threshold
+
+## Progress
+- [ ] Trigger validated
+- [ ] Data prepared
+- [ ] Training started
+- [ ] Validation passed
+- [ ] Model deployed
+
+Maintain model freshness with automated retraining.

package/plugin/commands/omgoptim/distill.md

@@ -0,0 +1,90 @@
+---
+description: Knowledge distillation from teacher model to smaller student model for efficient deployment
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: "[--teacher <path>] [--student_config <config>]"
+---
+
+# Knowledge Distillation: $ARGUMENTS
+
+Distill knowledge: **$ARGUMENTS**
+
+## Agent
+Uses **performance-engineer-agent** for knowledge distillation.
+
+## Parameters
+- **teacher**: Path to teacher (large) model
+- **student_config**: Path to student model configuration
+
+## Distillation Types
+
+### Response-Based
+- Match soft labels
+- Temperature scaling
+- KL divergence loss
+
+### Feature-Based
+- Match intermediate layers
+- Attention transfer
+- Hidden state matching
+
+### Relation-Based
+- Match pairwise relations
+- Graph distillation
+- Contrastive learning
+
+## Code Template
+```python
+from omgkit.optimization import KnowledgeDistiller
+
+distiller = KnowledgeDistiller()
+
+# Define student model (smaller)
+student = distiller.create_student(
+    architecture="mlp",
+    hidden_dims=[64, 32],  # Smaller than teacher
+    output_dim=10
+)
+
+# Distill
+distilled_model = distiller.distill(
+    teacher_path="models/teacher_large.pt",
+    student=student,
+    train_data="data/splits/train.parquet",
+    temperature=4.0,
+    alpha=0.5,  # Weight for distillation loss
+    epochs=50
+)
+
+# Compare
+distiller.compare(
+    teacher="models/teacher_large.pt",
+    student=distilled_model,
+    test_data="data/splits/test.parquet"
+)
+```
+
+## Hyperparameters
+- **Temperature**: Higher = softer labels (typically 2-10)
+- **Alpha**: Balance between hard and soft labels
+- **Loss**: KL divergence + cross-entropy
+
+## Benefits
+- Smaller model size
+- Faster inference
+- Lower memory
+- Maintained accuracy
+
+## Comparison Output
+- Size comparison
+- Speed comparison
+- Accuracy comparison
+- Layer-by-layer analysis
+
+## Progress
+- [ ] Teacher loaded
+- [ ] Student created
+- [ ] Distillation training
+- [ ] Quality validated
+- [ ] Report generated
+
+Transfer knowledge from large to small model efficiently.