swe-workflow-skills 0.1.0

package/skills/ml-experiment-tracking/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: ml-experiment-tracking
+description: "Design reproducible ML experiments — tracking, versioning, run comparison with MLflow, W&B, or DVC. Triggers: experiment tracking, MLflow, wandb, weights and biases, DVC, track experiments, compare models, hyperparameter, reproducibility, model registry, which model is better, experiment results, log metrics."
+model: sonnet
+allowed-tools: Read, Grep, Glob, Write, Edit, Bash
+---
+
+# ML Experiment Tracking
+
+Establish disciplined experiment tracking that makes ML development reproducible, comparable, and auditable. Without tracking, ML work degenerates into "which notebook had the good results?"
+
+## Why This Matters
+
+ML experiments differ from traditional software in critical ways:
+- **Non-deterministic**: Same code + same data can produce different results (random seeds, GPU nondeterminism)
+- **Multi-dimensional**: Performance depends on code, data, hyperparameters, and environment simultaneously
+- **Retrospective**: You often realize a past experiment was better only after running new ones
+
+Without systematic tracking, reproducing results becomes archaeology.
+
+## Workflow
+
+### Step 1: Set Up Tracking Infrastructure
+
+Before running experiments, establish:
+
+- **Experiment tracker**: MLflow, Weights & Biases, Neptune, or even a structured CSV/JSON approach
+- **Data versioning**: DVC, Delta Lake, or immutable dataset snapshots in object storage
+- **Code versioning**: Git (this should already exist)
+- **Environment capture**: Docker image hash, conda environment export, pip freeze
+
+Recommend MLflow for open-source simplicity, W&B for rich visualization needs. See [references/tracking-tools.md](references/tracking-tools.md) for setup patterns.
+
+### Step 2: Define What to Track
+
+For every experiment run, log:
+
+**Inputs:**
+- Dataset version or hash (not just "training_data.csv" — the specific version)
+- Git commit hash of the code
+- All hyperparameters (learning rate, batch size, epochs, model architecture choices)
+- Random seed
+- Environment (Python version, key library versions, GPU type)
+
+**Outputs:**
+- Primary metric (what you're optimizing: accuracy, F1, RMSE, etc.)
+- Secondary metrics (latency, model size, memory usage)
+- Training curves (loss per epoch)
+- Evaluation on held-out test set
+- Model artifacts (serialized model, weights)
+- Confusion matrix, ROC curves, or other diagnostic plots
+
+**Metadata:**
+- Experiment name and description (what hypothesis is being tested)
+- Run duration and compute cost
+- Tags for filtering (e.g., "baseline", "feature_engineering", "architecture_search")
+
+### Step 3: Design the Experiment
+
+Before running anything, define:
+
+1. **Hypothesis**: What do you expect to happen and why?
+2. **Baseline**: What's the current best result to beat?
+3. **Variable**: What exactly are you changing? (Change one thing at a time)
+4. **Metric**: What defines "better"? Set this before seeing results to avoid cherry-picking.
+5. **Stopping criteria**: When do you stop iterating? (Target metric reached, diminishing returns, time budget exhausted)
+
+Document this as the experiment description in your tracker.
+
+### Step 4: Run and Compare
+
+Run experiments with full tracking. After a batch of runs:
+
+1. **Compare metrics** across runs in a table or parallel coordinates plot
+2. **Identify the best run** by the pre-defined primary metric
+3. **Verify reproducibility** — can you re-run the best experiment and get similar results?
+4. **Check for data leakage** — is the test set truly held out? No information from test in training?
+5. **Assess overfitting** — compare train vs validation metrics
+
+### Step 5: Register the Best Model
+
+When a model is ready for staging or production:
+
+1. Register it in a model registry (MLflow Model Registry, or equivalent)
+2. Tag it with a stage: `staging`, `production`, `archived`
+3. Link it to the experiment run that produced it (full lineage)
+4. Document what makes this model better than the previous version
+
+This creates an auditable trail from "model in production" back to "exact data + code + params that created it."
+
+## Principles Applied
+
+- **DRY**: Track once, query many times. Don't reconstruct experiment details from memory or notebooks.
+- **YAGNI**: Start with simple tracking (parameters + metrics + artifacts). Add complexity (feature stores, automated comparisons) when the number of experiments justifies it.
+- **Reproducibility over speed**: A fast experiment you can't reproduce has zero long-term value.
+- **KISS**: One experiment changes one variable. Multi-variable changes make it impossible to attribute improvement.

package/skills/ml-experiment-tracking/evals/evals.json

@@ -0,0 +1,32 @@
+{
+  "skill_name": "ml-experiment-tracking",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "I'm training a text classification model using scikit-learn. I want to compare TF-IDF + Logistic Regression vs TF-IDF + Random Forest vs sentence-transformers + XGBoost. Set up experiment tracking so I can compare them properly.",
+      "expected_output": "Should set up MLflow or W&B tracking, define what to log for each run, ensure all three experiments are comparable, and provide the tracking code.",
+      "assertions": [
+        "Sets up experiment tracking infrastructure (MLflow or W&B)",
+        "Defines consistent metrics across all three approaches",
+        "Logs hyperparameters, dataset version, and git commit for each run",
+        "Logs model artifacts for later loading",
+        "Shows how to compare runs after completion",
+        "Emphasizes changing one variable at a time where possible",
+        "Includes reproducibility measures (random seed, environment capture)"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "I've been training models in Jupyter notebooks for 3 months and I have results scattered across 40+ notebooks. How do I get organized?",
+      "expected_output": "Should recommend setting up tracking infrastructure, then systematically re-running or cataloging key experiments. Should be practical about not re-running everything.",
+      "assertions": [
+        "Does NOT suggest re-running all 40 notebooks",
+        "Recommends setting up tracking for all future experiments",
+        "Suggests cataloging the best results from notebooks into the tracker manually",
+        "Recommends extracting reusable training code from notebooks into scripts",
+        "Suggests data versioning to prevent future dataset ambiguity",
+        "Provides a practical migration path, not an all-or-nothing approach"
+      ]
+    }
+  ]
+}

package/skills/ml-experiment-tracking/references/tracking-tools.md

@@ -0,0 +1,127 @@
+# Experiment Tracking Tools
+
+## Contents
+- MLflow setup and patterns
+- Weights & Biases patterns
+- DVC for data versioning
+- Comparison matrix
+
+## MLflow Setup
+
+### Basic tracking
+```python
+import mlflow
+
+mlflow.set_tracking_uri("http://localhost:5000")  # Or remote server
+mlflow.set_experiment("my-classification-project")
+
+with mlflow.start_run(run_name="baseline-logistic-regression"):
+    # Log parameters
+    mlflow.log_param("model_type", "logistic_regression")
+    mlflow.log_param("C", 1.0)
+    mlflow.log_param("max_iter", 100)
+    mlflow.log_param("dataset_version", "v2.3")
+    mlflow.log_param("git_commit", subprocess.check_output(["git", "rev-parse", "HEAD"]).strip())
+
+    # Train model
+    model = LogisticRegression(C=1.0, max_iter=100)
+    model.fit(X_train, y_train)
+
+    # Log metrics
+    y_pred = model.predict(X_test)
+    mlflow.log_metric("accuracy", accuracy_score(y_test, y_pred))
+    mlflow.log_metric("f1_score", f1_score(y_test, y_pred, average="weighted"))
+    mlflow.log_metric("precision", precision_score(y_test, y_pred, average="weighted"))
+
+    # Log model artifact
+    mlflow.sklearn.log_model(model, "model")
+
+    # Log additional artifacts
+    mlflow.log_artifact("confusion_matrix.png")
+```
+
+### Model Registry
+```python
+# Register a model from a run
+model_uri = f"runs:/{run_id}/model"
+mlflow.register_model(model_uri, "my-classifier")
+
+# Transition model stage
+client = mlflow.tracking.MlflowClient()
+client.transition_model_version_stage(
+    name="my-classifier",
+    version=3,
+    stage="Production"
+)
+```
+
+## Weights & Biases Patterns
+
+```python
+import wandb
+
+wandb.init(
+    project="my-classification-project",
+    name="baseline-logistic-regression",
+    config={
+        "model_type": "logistic_regression",
+        "C": 1.0,
+        "max_iter": 100,
+        "dataset_version": "v2.3",
+    }
+)
+
+# Training loop with automatic metric logging
+for epoch in range(config.epochs):
+    train_loss = train_one_epoch(model, train_loader)
+    val_loss, val_acc = evaluate(model, val_loader)
+    wandb.log({
+        "train_loss": train_loss,
+        "val_loss": val_loss,
+        "val_accuracy": val_acc,
+        "epoch": epoch
+    })
+
+# Log final artifacts
+wandb.log({"confusion_matrix": wandb.plot.confusion_matrix(
+    y_true=y_test, preds=y_pred, class_names=class_names
+)})
+
+wandb.finish()
+```
+
+## DVC for Data Versioning
+
+```bash
+# Initialize DVC in a git repo
+dvc init
+
+# Track a dataset
+dvc add data/training_data.csv
+git add data/training_data.csv.dvc data/.gitignore
+git commit -m "track training data v1"
+
+# Push data to remote storage
+dvc remote add -d myremote s3://my-bucket/dvc-store
+dvc push
+
+# Switch to a different data version
+git checkout v2.0  # Checkout the git tag
+dvc checkout        # Pull the matching data version
+```
+
+DVC tracks large files (datasets, model weights) alongside git without storing them in git. The `.dvc` file in git points to the actual data in remote storage.
+
+## Comparison Matrix
+
+| Feature | MLflow | W&B | Neptune | DVC |
+|---------|--------|-----|---------|-----|
+| Experiment tracking | ✓ | ✓ | ✓ | Partial |
+| Model registry | ✓ | ✓ | ✓ | ✗ |
+| Data versioning | ✗ | ✓ (Artifacts) | ✗ | ✓ |
+| Visualization | Basic | Rich | Rich | ✗ |
+| Self-hosted | ✓ | Enterprise | ✗ | ✓ |
+| Free tier | Open source | Free (limits) | Free (limits) | Open source |
+| Best for | Teams wanting open-source control | Rich visualization and collaboration | Team collaboration | Data and pipeline versioning |
+
+**Recommendation**: MLflow + DVC for open-source, self-hosted workflows. W&B for teams that value visualization and can use SaaS.

package/skills/ml-model-deployment/SKILL.md

@@ -0,0 +1,106 @@
+---
+name: ml-model-deployment
+description: "Deploy ML models to production — serving infrastructure, monitoring, drift detection, safe rollouts. Triggers: deploy model, serve model, model API, model monitoring, data drift, concept drift, model degradation, A/B test model, canary model, inference endpoint, prediction API, retrain trigger."
+model: sonnet
+allowed-tools: Read, Grep, Glob, Write, Edit, Bash
+---
+
+# ML Model Deployment
+
+Deploy models to production with serving infrastructure that is reliable, observable, and safely updatable. Deploying a model is not the finish line — it's the beginning of the operational lifecycle.
+
+## Key Difference from Software Deployment
+
+Software deployment is primarily concerned with "does the code work?" Model deployment adds: "does the model still work *well* on real-world data that changes over time?" Models degrade silently — they keep returning predictions, just increasingly wrong ones.
+
+## Workflow
+
+### Step 1: Package the Model
+
+Before deployment, the model must be a self-contained, versioned artifact:
+
+- **Serialize the model**: Use framework-native formats (joblib/pickle for sklearn, SavedModel for TF, TorchScript for PyTorch, ONNX for cross-framework)
+- **Pin dependencies**: Exact versions of libraries used during training
+- **Include preprocessing**: The serving pipeline must apply the same feature transformations as training. Feature skew (different preprocessing in training vs serving) is the #1 cause of silent model failure
+- **Version it**: Tag with model registry version, git commit, training data version, and training metrics
+
+### Step 2: Choose the Serving Pattern
+
+| Pattern | When to Use | Latency | Complexity |
+|---------|-------------|---------|------------|
+| **REST API** (Flask/FastAPI + model) | Low traffic, simple models | Medium (50-200ms) | Low |
+| **Dedicated serving** (TF Serving, Triton, Seldon) | High traffic, GPU inference | Low (5-50ms) | Medium |
+| **Batch prediction** (scheduled job) | High volume, latency-tolerant | N/A (scheduled) | Low |
+| **Streaming** (Kafka + model) | Real-time event processing | Low | High |
+| **Edge / embedded** | Offline, privacy-sensitive | Very low | High |
+
+For most teams starting out: REST API with FastAPI is the right default. Optimize for lower latency only when measured need justifies complexity.
+
+### Step 3: Build the Serving Application
+
+The serving app needs:
+
+```
+Request → Validate input → Preprocess → Predict → Postprocess → Response
+```
+
+Key requirements:
+- **Input validation**: Reject malformed requests before they reach the model
+- **Feature preprocessing**: Apply the same transformations used in training
+- **Error handling**: Model errors should return 500 with diagnostic info, not crash the service
+- **Logging**: Log every prediction request/response for monitoring and debugging
+- **Health check**: Endpoint that verifies the model is loaded and responsive
+
+See [references/serving-patterns.md](references/serving-patterns.md) for implementation templates.
+
+### Step 4: Deploy Safely
+
+Models should be deployed with the same caution as any production service — more, given their non-deterministic nature:
+
+**Shadow mode**: Deploy the new model alongside the current one. Both receive traffic, but only the current model's predictions are used. Compare predictions to evaluate the new model on real data.
+
+**Canary deployment**: Route a small percentage of traffic (5-10%) to the new model. Monitor metrics. Gradually increase if performance is acceptable.
+
+**A/B testing**: Split traffic between model versions to measure business impact (conversion rate, click-through, etc.), not just ML metrics.
+
+**Blue-green**: Deploy new model to a separate environment, switch all traffic at once after validation. Fastest rollback.
+
+### Step 5: Monitor in Production
+
+After deployment, monitor continuously. See [references/serving-patterns.md](references/serving-patterns.md) for implementation details.
+
+**Model performance monitoring:**
+- Track prediction accuracy against ground truth (when available)
+- Monitor prediction distribution (are predictions shifting?)
+- Track business KPIs that the model influences
+
+**Data drift detection:**
+- Compare input feature distributions against the training data baseline
+- Use statistical tests: Population Stability Index (PSI), Kolmogorov-Smirnov (KS) test, Jensen-Shannon divergence
+- Alert when drift exceeds thresholds
+
+**System monitoring:**
+- Prediction latency (P50, P95, P99)
+- Request throughput and error rates
+- Resource utilization (CPU, memory, GPU)
+- Model loading time
+
+**Retraining triggers:**
+- Accuracy drops below threshold
+- Data drift exceeds threshold
+- Scheduled retraining (weekly/monthly depending on domain)
+- New labeled data becomes available
+
+### Step 6: Plan Rollback
+
+Every model deployment needs a rollback plan:
+- Previous model version tagged and ready in the registry
+- One-command rollback through CI/CD or model registry stage transition
+- Automated rollback on monitoring alert (if confidence is high)
+
+## Principles Applied
+
+- **KISS**: Start with REST API + batch monitoring. Add real-time drift detection when scale justifies it.
+- **YAGNI**: Don't build an A/B testing framework for your first model. Shadow mode is enough.
+- **Defense in depth**: Validate inputs, monitor outputs, detect drift, and have rollback ready.
+- **Functional Independence**: Serving, monitoring, and retraining should be independent components, not monolithic.

package/skills/ml-model-deployment/evals/evals.json

@@ -0,0 +1,34 @@
+{
+  "skill_name": "ml-model-deployment",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "I have a scikit-learn fraud detection model that's performing well in our test environment. I need to deploy it as a real-time API that our payment service calls for every transaction. We process about 1000 transactions per second. What's the best approach?",
+      "expected_output": "Should design a serving architecture considering the 1000 TPS requirement, recommend appropriate serving pattern, address preprocessing consistency, and set up monitoring with drift detection.",
+      "assertions": [
+        "Considers the 1000 TPS requirement in architecture choice",
+        "Addresses feature preprocessing consistency (training-serving skew)",
+        "Recommends containerized serving with horizontal scaling",
+        "Sets up input validation for the prediction endpoint",
+        "Designs drift detection (fraud patterns change over time)",
+        "Plans a safe rollout strategy (shadow mode or canary)",
+        "Includes latency monitoring (fraud checks must be fast)",
+        "Plans for rollback"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Our recommendation model has been in production for 6 months and click-through rates are declining. The data science team says the model metrics look fine on their test set. What's going on and how do we fix it?",
+      "expected_output": "Should identify this as likely data drift or concept drift, recommend checking production data distributions against training baseline, and suggest monitoring and retraining infrastructure.",
+      "assertions": [
+        "Identifies data/concept drift as the likely cause",
+        "Explains why test set metrics can look fine while production degrades",
+        "Recommends comparing production feature distributions against training baseline",
+        "Suggests PSI or KS tests for drift detection",
+        "Recommends setting up continuous monitoring to prevent recurrence",
+        "Suggests a retraining pipeline with fresh data",
+        "Does NOT just say 'retrain the model' without diagnosing the cause"
+      ]
+    }
+  ]
+}

package/skills/ml-model-deployment/references/serving-patterns.md

@@ -0,0 +1,162 @@
+# Serving and Monitoring Patterns
+
+## Contents
+- FastAPI serving template
+- Drift detection patterns
+- Monitoring dashboard design
+- Retraining pipeline triggers
+
+## FastAPI Model Serving Template
+
+```python
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel, validator
+import mlflow
+import numpy as np
+import logging
+import time
+
+app = FastAPI(title="Model Serving API")
+logger = logging.getLogger(__name__)
+
+# Load model at startup (not per-request)
+model = None
+
+@app.on_event("startup")
+async def load_model():
+    global model
+    model = mlflow.pyfunc.load_model("models:/my-classifier/Production")
+    logger.info("Model loaded successfully")
+
+class PredictionRequest(BaseModel):
+    features: list[float]
+
+    @validator("features")
+    def validate_features(cls, v):
+        if len(v) != EXPECTED_FEATURE_COUNT:
+            raise ValueError(f"Expected {EXPECTED_FEATURE_COUNT} features, got {len(v)}")
+        return v
+
+class PredictionResponse(BaseModel):
+    prediction: int
+    probability: float
+    model_version: str
+    latency_ms: float
+
+@app.post("/predict", response_model=PredictionResponse)
+async def predict(request: PredictionRequest):
+    start = time.time()
+    try:
+        features = np.array(request.features).reshape(1, -1)
+        prediction = model.predict(features)
+        probability = model.predict_proba(features).max()
+
+        latency_ms = (time.time() - start) * 1000
+
+        # Log for monitoring
+        logger.info(f"prediction={prediction[0]} prob={probability:.4f} latency={latency_ms:.1f}ms")
+
+        return PredictionResponse(
+            prediction=int(prediction[0]),
+            probability=float(probability),
+            model_version=model.metadata.run_id,
+            latency_ms=latency_ms,
+        )
+    except Exception as e:
+        logger.error(f"Prediction failed: {e}")
+        raise HTTPException(status_code=500, detail="Prediction failed")
+
+@app.get("/health")
+async def health():
+    return {"status": "healthy", "model_loaded": model is not None}
+```
+
+## Drift Detection Patterns
+
+### Population Stability Index (PSI)
+
+PSI measures how much a feature distribution has shifted from a baseline.
+
+```python
+import numpy as np
+
+def calculate_psi(baseline, current, bins=10):
+    """Calculate Population Stability Index between two distributions."""
+    # Bin the data
+    breakpoints = np.quantile(baseline, np.linspace(0, 1, bins + 1))
+    breakpoints[0] = -np.inf
+    breakpoints[-1] = np.inf
+
+    baseline_counts = np.histogram(baseline, bins=breakpoints)[0] / len(baseline)
+    current_counts = np.histogram(current, bins=breakpoints)[0] / len(current)
+
+    # Avoid division by zero
+    baseline_counts = np.clip(baseline_counts, 1e-6, None)
+    current_counts = np.clip(current_counts, 1e-6, None)
+
+    psi = np.sum((current_counts - baseline_counts) * np.log(current_counts / baseline_counts))
+    return psi
+
+# Interpretation:
+# PSI < 0.1:  No significant drift
+# PSI 0.1-0.2: Moderate drift — investigate
+# PSI > 0.2:  Significant drift — retrain
+```
+
+### Monitoring with Evidently AI
+
+```python
+from evidently.report import Report
+from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
+
+# Compare current production data against training baseline
+report = Report(metrics=[
+    DataDriftPreset(),
+    TargetDriftPreset(),
+])
+
+report.run(reference_data=training_df, current_data=production_df)
+report.save_html("drift_report.html")
+
+# Programmatic access to results
+results = report.as_dict()
+drift_detected = results["metrics"][0]["result"]["dataset_drift"]
+```
+
+## Monitoring Dashboard Design
+
+A model monitoring dashboard should show:
+
+**Real-time panel:**
+- Prediction throughput (requests/sec)
+- Latency percentiles (P50, P95, P99)
+- Error rate
+- Prediction distribution (histogram updated hourly)
+
+**Drift panel (updated daily/hourly):**
+- PSI per feature over time (line chart)
+- Drift alert threshold line
+- Feature importance × drift magnitude (which drifted features matter most?)
+
+**Performance panel (updated as ground truth arrives):**
+- Primary metric (accuracy, F1, RMSE) over time
+- Comparison against baseline model
+- Performance by segment (if applicable)
+
+**Operational panel:**
+- Model version currently serving
+- Last retrain date
+- Resource utilization (CPU, memory, GPU)
+- Cost per prediction
+
+## Retraining Pipeline Triggers
+
+| Trigger | When to Use | Implementation |
+|---------|-------------|---------------|
+| **Scheduled** | Stable domains (weekly/monthly) | Cron job or orchestrator schedule |
+| **Performance-based** | When ground truth is available | Monitor metric, trigger when below threshold |
+| **Drift-based** | When data changes are detectable | Monitor PSI/KS, trigger when above threshold |
+| **Data-volume** | When new labeled data accumulates | Count new labels, trigger at threshold |
+| **Manual** | Regulatory or business-driven | Human approval in model registry |
+
+Start with scheduled + drift-based triggers. Add performance-based when ground truth labeling is established.