omgkit 2.20.0 → 2.21.1

package/plugin/commands/omgoptim/profile.md

@@ -0,0 +1,92 @@
+---
+description: Profile model performance including latency, memory usage, compute requirements, and bottlenecks
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <model_path> [--input_shape <shape>]
+---
+
+# Model Profiling: $ARGUMENTS
+
+Profile model: **$ARGUMENTS**
+
+## Agent
+Uses **performance-engineer-agent** for comprehensive profiling.
+
+## Parameters
+- **model_path**: Path to model
+- **input_shape**: Input tensor shape for inference
+
+## Profiling Dimensions
+
+### Latency
+- Cold start time
+- Warm inference time
+- P50/P90/P99 latencies
+- Batch size scaling
+
+### Memory
+- Peak memory usage
+- Memory per layer
+- Activation memory
+- Gradient memory (training)
+
+### Compute
+- FLOPs count
+- MACs (multiply-accumulate)
+- Parameter count
+- Layer-wise breakdown
+
+### Bottlenecks
+- Slowest layers
+- Memory-bound vs compute-bound
+- Data loading overhead
+- GPU utilization
+
+## Code Template
+```python
+from omgkit.optimization import ModelProfiler
+
+profiler = ModelProfiler()
+
+# Comprehensive profiling
+profile = profiler.profile(
+    model_path="models/best_model.pt",
+    input_shape=(1, 3, 224, 224),
+    device="cuda",
+    warmup_iterations=10,
+    profile_iterations=100
+)
+
+# Print summary
+print(f"Latency (P50): {profile.latency_p50:.2f}ms")
+print(f"Latency (P99): {profile.latency_p99:.2f}ms")
+print(f"Peak Memory: {profile.peak_memory_mb:.1f}MB")
+print(f"FLOPs: {profile.flops / 1e9:.2f}G")
+print(f"Parameters: {profile.params / 1e6:.2f}M")
+
+# Detailed breakdown
+profiler.layer_breakdown(profile)
+
+# Generate report
+profiler.report(profile, output="reports/profiling_report.html")
+```
+
+## Hardware Support
+- CPU profiling
+- CUDA profiling
+- TensorRT profiling
+- ONNX Runtime profiling
+
+## Optimization Recommendations
+- Identified bottlenecks
+- Suggested optimizations
+- Target hardware considerations
+- Trade-off analysis
+
+## Progress
+- [ ] Model loaded
+- [ ] Warmup complete
+- [ ] Profiling executed
+- [ ] Analysis complete
+- [ ] Report generated
+
+Identify performance bottlenecks and optimization opportunities.

package/plugin/commands/omgoptim/prune.md

@@ -0,0 +1,81 @@
+---
+description: Prune model weights using magnitude, structured, or lottery ticket methods to reduce redundancy
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <method> [--sparsity <sparsity>]
+---
+
+# Model Pruning: $ARGUMENTS
+
+Prune model: **$ARGUMENTS**
+
+## Agent
+Uses **performance-engineer-agent** for model pruning.
+
+## Parameters
+- **method**: magnitude | structured | lottery_ticket (default: magnitude)
+- **sparsity**: Target sparsity level (default: 0.5 = 50% removed)
+
+## Pruning Methods
+
+### Magnitude Pruning
+- Remove smallest weights
+- Unstructured sparsity
+- Most flexible
+- Requires sparse support
+
+### Structured Pruning
+- Remove entire channels/filters
+- Hardware-friendly
+- Immediate speedup
+- More accuracy loss
+
+### Lottery Ticket
+- Find sparse subnetwork
+- Train from scratch
+- Best theoretical results
+- Most compute-intensive
+
+## Code Template
+```python
+from omgkit.optimization import ModelPruner
+
+pruner = ModelPruner()
+
+# Iterative magnitude pruning
+pruned_model = pruner.prune(
+    model_path="models/best_model.pt",
+    method="magnitude",
+    target_sparsity=0.7,
+    iterative_steps=5,
+    finetune_epochs=10,
+    finetune_data="data/splits/train.parquet"
+)
+
+# Report
+pruner.report(
+    original_model="models/best_model.pt",
+    pruned_model=pruned_model,
+    output="reports/pruning_report.html"
+)
+```
+
+## Sparsity Levels
+- 50%: Safe, minimal accuracy loss
+- 70%: Moderate, some accuracy drop
+- 90%: Aggressive, significant drop
+- 95%+: Extreme, research only
+
+## Best Practices
+- Iterative pruning (gradual)
+- Fine-tune after pruning
+- Validate on holdout set
+- Consider structured for deployment
+
+## Progress
+- [ ] Model analyzed
+- [ ] Pruning applied
+- [ ] Fine-tuning complete
+- [ ] Quality validated
+- [ ] Report generated
+
+Remove redundant weights while maintaining performance.

package/plugin/commands/omgoptim/quantize.md

@@ -0,0 +1,83 @@
+---
+description: Quantize model using dynamic, static, or QAT methods to reduce size and improve inference speed
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <method> [--calibration_data <path>]
+---
+
+# Model Quantization: $ARGUMENTS
+
+Quantize model: **$ARGUMENTS**
+
+## Agent
+Uses **performance-engineer-agent** for model quantization.
+
+## Parameters
+- **method**: dynamic | static | qat (default: dynamic)
+- **calibration_data**: Path to calibration data (for static)
+
+## Quantization Methods
+
+### Dynamic Quantization
+- Quantize weights only
+- Activations at runtime
+- No calibration needed
+- Quick to apply
+
+### Static Quantization
+- Quantize weights and activations
+- Requires calibration data
+- Better performance
+- More accurate
+
+### QAT (Quantization-Aware Training)
+- Train with quantization simulation
+- Best accuracy preservation
+- Requires retraining
+- Most effort
+
+## Code Template
+```python
+from omgkit.optimization import ModelQuantizer
+
+quantizer = ModelQuantizer()
+
+# Static quantization
+quantized_model = quantizer.quantize(
+    model_path="models/best_model.pt",
+    method="static",
+    calibration_data="data/splits/val.parquet",
+    calibration_samples=1000,
+    dtype="int8"
+)
+
+# Evaluate quantized model
+metrics = quantizer.evaluate(
+    original_model="models/best_model.pt",
+    quantized_model=quantized_model,
+    test_data="data/splits/test.parquet"
+)
+
+print(f"Size reduction: {metrics['size_reduction']:.1%}")
+print(f"Speedup: {metrics['speedup']:.2f}x")
+print(f"Accuracy drop: {metrics['accuracy_drop']:.2%}")
+```
+
+## Output Formats
+- INT8: 4x smaller, ~2-4x faster
+- INT4: 8x smaller, experimental
+- FP16: 2x smaller, minimal accuracy loss
+
+## Comparison Metrics
+- Model size reduction
+- Inference speedup
+- Memory usage reduction
+- Accuracy degradation
+
+## Progress
+- [ ] Model loaded
+- [ ] Calibration complete
+- [ ] Quantization applied
+- [ ] Quality validated
+- [ ] Output saved
+
+Optimize model for production deployment.

package/plugin/commands/omgtrain/baseline.md

@@ -0,0 +1,78 @@
+---
+description: Train baseline models for comparison including logistic regression, random forest, and XGBoost
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <task_type> [--models <models>]
+---
+
+# Baseline Training: $ARGUMENTS
+
+Train baselines: **$ARGUMENTS**
+
+## Agent
+Uses **model-engineer-agent** for baseline model training.
+
+## Parameters
+- **task_type**: classification | regression | detection | segmentation | nlp
+- **models**: List of baseline models to train
+
+## Default Baselines by Task
+
+### Classification
+- Logistic Regression
+- Random Forest
+- XGBoost
+- LightGBM
+- Naive Bayes
+
+### Regression
+- Linear Regression
+- Random Forest Regressor
+- XGBoost Regressor
+- LightGBM Regressor
+
+### NLP
+- TF-IDF + SVM
+- FastText
+- DistilBERT
+
+## Code Template
+```python
+from omgkit.training import BaselineTrainer
+
+trainer = BaselineTrainer(task_type="classification")
+
+# Train multiple baselines
+results = trainer.train_baselines(
+    data_path="data/splits/",
+    models=["logistic_regression", "random_forest", "xgboost"],
+    target_column="label",
+    metrics=["accuracy", "f1", "roc_auc"]
+)
+
+# Compare and select best
+best_model = trainer.select_best(metric="f1")
+
+# Save report
+trainer.report(output="reports/baseline_comparison.html")
+```
+
+## Output
+- Trained models
+- Comparison metrics table
+- Best baseline selection
+- Confusion matrices
+- Feature importance (if applicable)
+
+## Comparison Metrics
+- Primary: Task-specific (accuracy, F1, RMSE)
+- Secondary: Training time, inference time
+- Tertiary: Model size, memory usage
+
+## Progress
+- [ ] Data loaded
+- [ ] Models trained
+- [ ] Metrics computed
+- [ ] Best selected
+- [ ] Report generated
+
+Establish performance baseline before complex models.

package/plugin/commands/omgtrain/compare.md

@@ -0,0 +1,99 @@
+---
+description: Compare multiple experiments and models across metrics, visualize differences, and select best
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <experiments> [--metric <metric>]
+---
+
+# Experiment Comparison: $ARGUMENTS
+
+Compare experiments: **$ARGUMENTS**
+
+## Agent
+Uses **model-engineer-agent** for experiment comparison.
+
+## Parameters
+- **experiments**: List of experiment names or run IDs
+- **metric**: Primary comparison metric
+
+## Comparison Dimensions
+
+### Metrics
+- Primary performance metric
+- Secondary metrics
+- Training metrics (loss curves)
+- Validation metrics
+
+### Resources
+- Training time
+- GPU memory usage
+- Inference latency
+- Model size
+
+### Parameters
+- Hyperparameter differences
+- Architecture variations
+- Data preprocessing
+
+## Code Template
+```python
+from omgkit.training import ExperimentComparer
+import mlflow
+
+comparer = ExperimentComparer(tracking_uri="http://mlflow.example.com")
+
+comparison = comparer.compare(
+    experiments=[
+        "churn_v1",
+        "churn_v2_deeper",
+        "churn_v3_ensemble"
+    ],
+    metrics=["accuracy", "f1", "roc_auc", "latency"],
+    primary_metric="f1"
+)
+
+# Visualize comparison
+comparer.plot_comparison(
+    comparison,
+    output="reports/experiment_comparison.html"
+)
+
+# Statistical significance test
+significance = comparer.significance_test(
+    experiment_a="churn_v2_deeper",
+    experiment_b="churn_v3_ensemble",
+    metric="f1"
+)
+
+# Select best model
+best = comparer.select_best(
+    comparison,
+    metric="f1",
+    constraints={"latency_ms": 50}
+)
+```
+
+## Visualizations
+- Parallel coordinates
+- Metric bar charts
+- Learning curve overlay
+- Hyperparameter importance
+
+## Statistical Tests
+- t-test for significance
+- Bootstrap confidence intervals
+- McNemar's test (classification)
+
+## Output
+- Comparison table
+- Winner recommendation
+- Trade-off analysis
+- Visual report
+
+## Progress
+- [ ] Experiments loaded
+- [ ] Metrics extracted
+- [ ] Comparison computed
+- [ ] Significance tested
+- [ ] Report generated
+
+Make data-driven model selection decisions.

package/plugin/commands/omgtrain/evaluate.md

@@ -0,0 +1,85 @@
+---
+description: Evaluate model comprehensively including performance, robustness, fairness, and efficiency metrics
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <model_path> [--test_data <path>]
+---
+
+# Model Evaluation: $ARGUMENTS
+
+Evaluate model: **$ARGUMENTS**
+
+## Agent
+Uses **model-engineer-agent** for comprehensive evaluation.
+
+## Parameters
+- **model_path**: Path to trained model
+- **test_data**: Path to test dataset
+
+## Evaluation Types
+
+### Performance Metrics
+- Accuracy, Precision, Recall, F1
+- ROC-AUC, PR-AUC
+- Confusion matrix
+- Classification report
+
+### Robustness Testing
+- Perturbation tests
+- Out-of-distribution detection
+- Adversarial evaluation
+- Noise sensitivity
+
+### Fairness Analysis
+- Demographic parity
+- Equal opportunity
+- Calibration by group
+- Bias detection
+
+### Efficiency Metrics
+- Inference latency
+- Memory usage
+- Throughput
+- Model size
+
+## Code Template
+```python
+from omgkit.training import ModelEvaluator
+
+evaluator = ModelEvaluator()
+
+results = evaluator.evaluate(
+    model_path="models/best_model.pt",
+    test_data="data/splits/test.parquet",
+    evaluation_types=[
+        "performance",
+        "robustness",
+        "fairness",
+        "efficiency"
+    ],
+    fairness_groups=["gender", "age_group"],
+    perturbation_levels=[0.01, 0.05, 0.1]
+)
+
+evaluator.report(
+    results=results,
+    output="reports/evaluation_report.html",
+    include_plots=True
+)
+```
+
+## Report Includes
+- Summary metrics
+- Per-class performance
+- Confusion matrix visualization
+- ROC/PR curves
+- Feature importance
+- Error analysis
+
+## Progress
+- [ ] Model loaded
+- [ ] Test data prepared
+- [ ] Metrics computed
+- [ ] Analysis complete
+- [ ] Report generated
+
+Comprehensive evaluation for production readiness.

package/plugin/commands/omgtrain/train.md

@@ -0,0 +1,81 @@
+---
+description: Train ML model with full pipeline including experiment tracking, checkpointing, and early stopping
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <model_type> [--config <config>] [--experiment <name>]
+---
+
+# Model Training: $ARGUMENTS
+
+Train model: **$ARGUMENTS**
+
+## Agent
+Uses **model-engineer-agent** for comprehensive model training.
+
+## Parameters
+- **model_type**: Model architecture or type
+- **config**: Path to training config
+- **experiment**: Experiment name for tracking
+
+## Features
+- Automatic experiment tracking (MLflow/W&B)
+- Checkpointing
+- Early stopping
+- Learning rate scheduling
+- Gradient accumulation
+- Mixed precision training
+- Distributed training support
+
+## Code Template
+```python
+from omgkit.training import Trainer
+import mlflow
+
+trainer = Trainer(
+    model_type="pytorch",
+    config_path="config/model_config.yaml",
+    experiment_name="churn_prediction_v2"
+)
+
+model = trainer.create_model(
+    architecture="mlp",
+    input_dim=100,
+    hidden_dims=[256, 128, 64],
+    output_dim=2,
+    dropout=0.3
+)
+
+with mlflow.start_run():
+    history = trainer.train(
+        model=model,
+        train_data="data/splits/train.parquet",
+        val_data="data/splits/val.parquet",
+        epochs=100,
+        batch_size=64,
+        learning_rate=1e-3,
+        early_stopping=True,
+        patience=10,
+        callbacks=[
+            trainer.callbacks.checkpoint("models/checkpoints/"),
+            trainer.callbacks.lr_scheduler("cosine"),
+            trainer.callbacks.tensorboard("logs/")
+        ]
+    )
+
+    mlflow.log_metrics(history.final_metrics)
+    mlflow.pytorch.log_model(model, "model")
+```
+
+## Training Options
+- Optimizers: Adam, SGD, AdamW
+- Schedulers: cosine, step, plateau
+- Loss functions: cross-entropy, focal, custom
+- Regularization: dropout, weight decay
+
+## Progress
+- [ ] Config loaded
+- [ ] Model created
+- [ ] Training started
+- [ ] Validation monitored
+- [ ] Model saved
+
+Train production-ready models with full reproducibility.

package/plugin/commands/omgtrain/tune.md

@@ -0,0 +1,89 @@
+---
+description: Hyperparameter tuning with Optuna using TPE, random, grid, or CMA-ES optimization
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: "[--n_trials <n>] [--method <method>]"
+---
+
+# Hyperparameter Tuning: $ARGUMENTS
+
+Tune hyperparameters: **$ARGUMENTS**
+
+## Agent
+Uses **model-engineer-agent** for hyperparameter optimization.
+
+## Parameters
+- **n_trials**: Number of optimization trials (default: 100)
+- **method**: tpe | random | grid | cmaes (default: tpe)
+
+## Search Methods
+
+### TPE (Tree-structured Parzen Estimator)
+- Bayesian optimization
+- Sample-efficient
+- Handles conditionals
+
+### Random Search
+- Simple baseline
+- Parallelizable
+- Good for high dimensions
+
+### Grid Search
+- Exhaustive search
+- Reproducible
+- Best for small spaces
+
+### CMA-ES
+- Evolution strategy
+- Good for continuous spaces
+- Noise-robust
+
+## Code Template
+```python
+from omgkit.training import HyperparameterTuner
+import optuna
+
+tuner = HyperparameterTuner(
+    study_name="churn_prediction_tuning",
+    direction="maximize",
+    sampler=optuna.samplers.TPESampler()
+)
+
+search_space = {
+    "learning_rate": {"type": "float", "low": 1e-5, "high": 1e-1, "log": True},
+    "hidden_dim": {"type": "categorical", "choices": [64, 128, 256, 512]},
+    "num_layers": {"type": "int", "low": 1, "high": 5},
+    "dropout": {"type": "float", "low": 0.0, "high": 0.5},
+    "batch_size": {"type": "categorical", "choices": [16, 32, 64, 128]}
+}
+
+best_params = tuner.tune(
+    train_fn=train_model,
+    search_space=search_space,
+    n_trials=100,
+    metric="val_f1",
+    pruner=optuna.pruners.MedianPruner()
+)
+
+tuner.report(output="reports/tuning_report.html")
+```
+
+## Pruning
+- Median pruner: Stop unpromising trials
+- Percentile pruner: More aggressive
+- Hyperband: Multi-fidelity
+
+## Output
+- Best hyperparameters
+- Optimization history
+- Parameter importance
+- Parallel coordinate plot
+- HTML report
+
+## Progress
+- [ ] Search space defined
+- [ ] Optimization started
+- [ ] Trials completed
+- [ ] Best params found
+- [ ] Report generated
+
+Find optimal hyperparameters efficiently.