omgkit 2.20.0 → 2.21.1

package/plugin/commands/omgdata/augment.md

@@ -0,0 +1,86 @@
+---
+description: Augment data to increase diversity and quantity for image, text, tabular, audio, and timeseries
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <data_type> [--techniques <techniques>] [--factor <factor>]
+---
+
+# Data Augmentation: $ARGUMENTS
+
+Augment data: **$ARGUMENTS**
+
+## Agent
+Uses **data-engineer-agent** for data augmentation.
+
+## Parameters
+- **data_type**: image | text | tabular | audio | timeseries
+- **techniques**: Comma-separated list of techniques
+- **factor**: Augmentation multiplier (default: 2.0)
+
+## Techniques by Type
+
+### Image
+- random_flip, random_rotation
+- random_crop, color_jitter
+- gaussian_noise, cutout
+- mixup, cutmix
+
+### Text
+- synonym_replacement
+- random_insertion/swap/deletion
+- back_translation
+- contextual_augmentation (BERT)
+
+### Tabular
+- SMOTE for imbalanced data
+- noise_injection
+- mixup
+- feature_permutation
+
+### Audio
+- time_stretch, pitch_shift
+- add_noise, time_shift
+- spectrogram augmentation
+
+### Timeseries
+- window_slicing
+- magnitude_warping
+- time_warping
+- jittering
+
+## Code Template
+```python
+from omgkit.data import DataAugmenter
+import albumentations as A
+
+augmenter = DataAugmenter(data_type="image")
+
+transform = augmenter.create_pipeline([
+    A.RandomRotate90(p=0.5),
+    A.Flip(p=0.5),
+    A.RandomBrightnessContrast(p=0.3),
+    A.GaussNoise(var_limit=(10, 50), p=0.2),
+    A.CoarseDropout(max_holes=8, max_height=32, max_width=32, p=0.3)
+])
+
+augmented_data = augmenter.augment(
+    data_path="data/processed/train_images/",
+    transform=transform,
+    factor=3.0,
+    output_path="data/augmented/train_images/"
+)
+```
+
+## Best Practices
+- Preserve label integrity
+- Match augmentation to task
+- Balance augmentation strength
+- Validate augmented samples
+
+## Progress
+- [ ] Data loaded
+- [ ] Techniques configured
+- [ ] Augmentation applied
+- [ ] Quality validated
+- [ ] Output saved
+
+Increase training data diversity while maintaining quality.

package/plugin/commands/omgdata/collect.md

@@ -0,0 +1,81 @@
+---
+description: Collect data from multiple sources including databases, APIs, files, and streaming systems
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <source_type> [--config <config_file>]
+---
+
+# Data Collection: $ARGUMENTS
+
+Collect data from: **$ARGUMENTS**
+
+## Agent
+Uses **data-engineer-agent** for data collection operations.
+
+## Parameters
+- **source_type**: database | api | files | streaming | scraping
+- **config**: Path to collection configuration file
+
+## Supported Sources
+
+### Database
+- PostgreSQL, MySQL, MongoDB
+- Connection pooling
+- Query optimization
+- Batch extraction
+
+### API
+- REST/GraphQL endpoints
+- Authentication handling
+- Rate limiting
+- Pagination support
+
+### Files
+- CSV, Parquet, JSON, Excel
+- S3, GCS, Azure Blob
+- Local filesystem
+- Compressed archives
+
+### Streaming
+- Kafka consumers
+- Kinesis streams
+- Real-time ingestion
+
+## Code Template
+```python
+from omgkit.data import DataCollector
+
+collector = DataCollector(config="config/data_config.yaml")
+
+data = collector.collect(
+    sources=[
+        {"type": "database", "connection": "postgres://..."},
+        {"type": "api", "endpoint": "https://api.example.com/data"},
+        {"type": "files", "path": "data/raw/*.csv"}
+    ],
+    validation_schema="schemas/raw_data_schema.json"
+)
+
+collector.save(
+    data=data,
+    output_path="data/raw/collected_data.parquet",
+    metadata=True
+)
+```
+
+## Actions
+1. Read collection configuration
+2. Connect to data sources
+3. Validate schemas during collection
+4. Apply data transformations
+5. Save to raw data directory
+6. Update data manifest
+
+## Progress
+- [ ] Config loaded
+- [ ] Sources connected
+- [ ] Data collected
+- [ ] Schema validated
+- [ ] Data saved
+- [ ] Manifest updated
+
+Track all data lineage for reproducibility.

package/plugin/commands/omgdata/label.md

@@ -0,0 +1,83 @@
+---
+description: Label data using manual annotation, weak supervision, active learning, or automated labeling
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <strategy> [--config <config>]
+---
+
+# Data Labeling: $ARGUMENTS
+
+Label data using: **$ARGUMENTS**
+
+## Agent
+Uses **data-engineer-agent** for data labeling workflows.
+
+## Parameters
+- **strategy**: manual | weak_supervision | active_learning | auto_label
+- **config**: Path to labeling configuration
+
+## Labeling Strategies
+
+### Manual
+- Export to Label Studio format
+- Human annotation interface
+- Quality assurance workflows
+- Inter-annotator agreement
+
+### Weak Supervision
+- Labeling functions with Snorkel
+- Programmatic labeling rules
+- Probabilistic label model
+- Noise-aware training
+
+### Active Learning
+- Uncertainty sampling
+- Query-by-committee
+- Prioritized labeling queue
+- Model-in-the-loop
+
+### Auto Label
+- Pretrained model inference
+- Confidence thresholding
+- Human review for low confidence
+- Continuous improvement
+
+## Code Template
+```python
+from omgkit.data import DataLabeler
+
+labeler = DataLabeler(strategy="weak_supervision")
+
+@labeler.labeling_function()
+def lf_keyword_spam(x):
+    spam_keywords = ["free", "win", "click here", "urgent"]
+    if any(kw in x.text.lower() for kw in spam_keywords):
+        return "SPAM"
+    return "ABSTAIN"
+
+@labeler.labeling_function()
+def lf_length(x):
+    if len(x.text) < 10:
+        return "SPAM"
+    return "ABSTAIN"
+
+labels = labeler.label(
+    data_path="data/raw/emails.parquet",
+    labeling_functions=[lf_keyword_spam, lf_length],
+    output_path="data/labeled/emails_labeled.parquet"
+)
+```
+
+## Metrics
+- Label coverage
+- Label accuracy (if ground truth available)
+- Annotator agreement
+- Model confidence distribution
+
+## Progress
+- [ ] Strategy configured
+- [ ] Labeling functions defined
+- [ ] Labels generated
+- [ ] Quality assessed
+- [ ] Output saved
+
+Maximize label quality with minimal manual effort.

package/plugin/commands/omgdata/split.md

@@ -0,0 +1,83 @@
+---
+description: Split data into train/validation/test sets with stratified, temporal, or group-based strategies
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: "[--strategy <strategy>] [--ratios <ratios>]"
+---
+
+# Data Split: $ARGUMENTS
+
+Split data: **$ARGUMENTS**
+
+## Agent
+Uses **data-engineer-agent** for data splitting.
+
+## Parameters
+- **strategy**: random | stratified | temporal | group (default: stratified)
+- **ratios**: Train/Val/Test ratios (default: "0.7,0.15,0.15")
+
+## Splitting Strategies
+
+### Random
+- Simple random sampling
+- Use case: IID data, no dependencies
+- Fastest approach
+
+### Stratified
+- Maintains class distribution
+- Use case: Classification with imbalanced classes
+- Ensures representative splits
+
+### Temporal
+- Split by time (no future leakage)
+- Use case: Time series, forecasting
+- Respects temporal order
+
+### Group
+- Keep groups together (e.g., same user)
+- Use case: User-level data
+- Prevents data leakage
+
+## Code Template
+```python
+from omgkit.data import DataSplitter
+
+splitter = DataSplitter()
+
+# Stratified split
+train, val, test = splitter.split(
+    data_path="data/processed/dataset.parquet",
+    strategy="stratified",
+    stratify_column="label",
+    ratios=[0.7, 0.15, 0.15],
+    random_state=42
+)
+
+# Temporal split (for time series)
+train, val, test = splitter.split(
+    data_path="data/processed/transactions.parquet",
+    strategy="temporal",
+    time_column="timestamp",
+    train_end="2023-10-01",
+    val_end="2023-11-01"
+)
+
+splitter.save_splits(
+    train=train, val=val, test=test,
+    output_dir="data/splits/"
+)
+```
+
+## Validation
+- No data leakage between splits
+- Class distribution preserved
+- Temporal order respected
+- Group integrity maintained
+
+## Progress
+- [ ] Strategy selected
+- [ ] Data loaded
+- [ ] Split executed
+- [ ] Validation passed
+- [ ] Splits saved
+
+Prevent data leakage for valid model evaluation.

package/plugin/commands/omgdata/validate.md

@@ -0,0 +1,76 @@
+---
+description: Validate data quality with schema checks, null analysis, range validation, and distribution analysis
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <data_path> [--schema <schema>] [--rules <rules>]
+---
+
+# Data Validation: $ARGUMENTS
+
+Validate data at: **$ARGUMENTS**
+
+## Agent
+Uses **data-engineer-agent** for comprehensive data validation.
+
+## Parameters
+- **data_path**: Path to data file or directory
+- **schema**: Path to schema definition (optional)
+- **rules**: Path to validation rules (optional)
+
+## Validation Types
+
+### Schema Validation
+- Column names match expected
+- Data types correct
+- Required columns present
+- Nullable constraints
+
+### Quality Validation
+- Null percentage thresholds
+- Unique constraints
+- Value range validation
+- Duplicate detection
+
+### Statistical Validation
+- Distribution drift detection
+- Outlier identification
+- Correlation analysis
+- Cardinality checks
+
+## Code Template
+```python
+from omgkit.data import DataValidator
+import great_expectations as ge
+
+validator = DataValidator()
+
+expectations = validator.create_expectations([
+    {"column": "user_id", "check": "not_null"},
+    {"column": "user_id", "check": "unique"},
+    {"column": "age", "check": "between", "min": 0, "max": 150},
+    {"column": "email", "check": "regex", "pattern": r"^[\w\.-]+@[\w\.-]+\.\w+$"},
+    {"column": "signup_date", "check": "date_format", "format": "%Y-%m-%d"}
+])
+
+results = validator.validate(
+    data_path="data/raw/users.parquet",
+    expectations=expectations
+)
+
+validator.generate_report(results, output="reports/validation_report.html")
+```
+
+## Output Report
+- Total checks executed
+- Passed/Failed/Warning counts
+- Per-column statistics
+- Recommended actions
+- Data quality score
+
+## Progress
+- [ ] Data loaded
+- [ ] Schema validated
+- [ ] Quality checks run
+- [ ] Statistics computed
+- [ ] Report generated
+
+Fail fast on critical violations, warn on soft issues.

package/plugin/commands/omgdata/version.md

@@ -0,0 +1,85 @@
+---
+description: Version data with DVC for reproducibility - commit, checkout, diff, and history operations
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <action> [--message <message>]
+---
+
+# Data Version: $ARGUMENTS
+
+Version data: **$ARGUMENTS**
+
+## Agent
+Uses **data-engineer-agent** for data versioning with DVC.
+
+## Parameters
+- **action**: commit | checkout | diff | history
+- **message**: Version commit message
+
+## Actions
+
+### Commit
+- Track changes with DVC
+- Create git commit with data hash
+- Push to remote storage
+- Tag version
+
+### Checkout
+- Restore specific data version
+- Download from remote storage
+- Verify data integrity
+
+### Diff
+- Compare data versions
+- Show added/removed files
+- Statistical differences
+
+### History
+- Show version history
+- List all data commits
+- Display metadata
+
+## Code Template
+```python
+from omgkit.data import DataVersioner
+
+versioner = DataVersioner()
+
+# Commit new version
+versioner.commit(
+    data_paths=["data/processed/"],
+    message="Added augmented training data v1.2",
+    tags=["training", "augmented"]
+)
+
+# Checkout specific version
+versioner.checkout(version="v1.1.0")
+
+# Compare versions
+diff = versioner.diff(v1="v1.0.0", v2="v1.2.0")
+print(diff)
+
+# Show history
+history = versioner.history()
+for version in history:
+    print(f"{version.tag}: {version.message} ({version.date})")
+```
+
+## Best Practices
+- Version data with each significant change
+- Include meaningful commit messages
+- Tag releases for reproducibility
+- Link data versions to model versions
+
+## Remote Storage
+- S3, GCS, Azure Blob
+- SSH/SFTP servers
+- Local network storage
+
+## Progress
+- [ ] Action validated
+- [ ] DVC operation executed
+- [ ] Remote synced
+- [ ] Git updated
+- [ ] Verification complete
+
+Enable full data reproducibility and lineage tracking.

package/plugin/commands/omgdeploy/ab.md

@@ -0,0 +1,94 @@
+---
+description: Setup A/B testing between model versions with traffic splitting and statistical analysis
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: "[--models <models>] [--traffic_split <split>]"
+---
+
+# A/B Testing: $ARGUMENTS
+
+Setup A/B test: **$ARGUMENTS**
+
+## Agent
+Uses **deployment-agent** for A/B testing configuration.
+
+## Parameters
+- **models**: List of model versions to test
+- **traffic_split**: Traffic distribution percentages
+
+## A/B Test Components
+
+### Traffic Splitting
+- Percentage-based routing
+- User-based stickiness
+- Geographic routing
+- Feature-based routing
+
+### Metrics Collection
+- Primary metrics (accuracy, latency)
+- Business metrics (conversion, revenue)
+- User experience metrics
+- System metrics
+
+### Statistical Analysis
+- Significance testing
+- Confidence intervals
+- Sample size calculation
+- Stopping rules
+
+## Code Template
+```python
+from omgkit.deployment import ABTester
+
+ab_tester = ABTester()
+
+experiment = ab_tester.create_experiment(
+    name="churn_model_v2_test",
+    models={
+        "control": "models/v1.0.0/model.pt",
+        "treatment": "models/v2.0.0/model.pt"
+    },
+    traffic_split={
+        "control": 0.8,
+        "treatment": 0.2
+    },
+    metrics=["accuracy", "latency", "conversion_rate"],
+    min_samples=10000,
+    significance_level=0.05
+)
+
+# Monitor experiment
+ab_tester.monitor(experiment)
+
+# Analyze results
+results = ab_tester.analyze(experiment)
+
+if results.is_significant and results.treatment_better:
+    ab_tester.promote("treatment")
+```
+
+## Experiment Types
+- A/B (two variants)
+- A/B/n (multiple variants)
+- Multi-armed bandit
+- Shadow mode
+
+## Safety Features
+- Automatic rollback
+- Circuit breaker
+- Error rate monitoring
+- Latency guardrails
+
+## Output Report
+- Statistical significance
+- Effect size
+- Confidence intervals
+- Recommendation
+
+## Progress
+- [ ] Experiment configured
+- [ ] Traffic routing active
+- [ ] Metrics collecting
+- [ ] Analysis running
+- [ ] Decision ready
+
+Make data-driven model rollout decisions.

package/plugin/commands/omgdeploy/cloud.md

@@ -0,0 +1,89 @@
+---
+description: Deploy model to cloud platforms with auto-scaling, monitoring, and production configurations
+allowed-tools: Task, Read, Write, Bash, Grep, Glob
+argument-hint: <provider> [--config <config>]
+---
+
+# Cloud Deployment: $ARGUMENTS
+
+Deploy to cloud: **$ARGUMENTS**
+
+## Agent
+Uses **deployment-agent** for cloud deployment.
+
+## Parameters
+- **provider**: aws | gcp | azure
+- **config**: Path to deployment configuration
+
+## Cloud Providers
+
+### AWS
+- SageMaker endpoints
+- Lambda (serverless)
+- ECS/EKS containers
+- Elastic Inference
+
+### GCP
+- Vertex AI endpoints
+- Cloud Run
+- GKE containers
+- TPU support
+
+### Azure
+- Azure ML endpoints
+- Azure Functions
+- AKS containers
+- Cognitive Services
+
+## Code Template
+```python
+from omgkit.deployment import CloudDeployer
+
+deployer = CloudDeployer(provider="aws")
+
+endpoint = deployer.deploy(
+    model_path="models/best_model.pt",
+    config={
+        "instance_type": "ml.m5.xlarge",
+        "instance_count": 2,
+        "autoscaling": {
+            "min_capacity": 1,
+            "max_capacity": 10,
+            "target_invocations": 1000
+        },
+        "monitoring": {
+            "data_capture": True,
+            "capture_percentage": 10
+        }
+    },
+    endpoint_name="churn-predictor-prod"
+)
+
+# Test endpoint
+response = deployer.invoke(
+    endpoint_name="churn-predictor-prod",
+    payload={"features": [1.0, 2.0, 3.0]}
+)
+```
+
+## Features
+- Auto-scaling policies
+- Blue-green deployment
+- A/B testing
+- Monitoring & logging
+- Cost optimization
+
+## Infrastructure as Code
+- Terraform templates
+- CloudFormation
+- Pulumi scripts
+- CDK support
+
+## Progress
+- [ ] Credentials validated
+- [ ] Resources provisioned
+- [ ] Model deployed
+- [ ] Health verified
+- [ ] Monitoring active
+
+Deploy production-grade ML endpoints to cloud.