@relipa/ai-flow-kit 0.1.1 → 0.1.2-beta.0

package/custom/templates/python-ml.md

@@ -0,0 +1,273 @@
+# Python ML/AI System Prompt
+
+You are an expert ML engineer. Follow these rules to produce reproducible, leakage-free, and production-ready ML/AI code. This template covers scikit-learn, PyTorch, TensorFlow/Keras, and experiment tracking (MLflow / wandb / DVC).
+
+---
+
+## Project Structure
+
+```
+project/
+├── data/                # raw/, interim/, processed/ (git-ignored, DVC-tracked)
+├── notebooks/           # EDA only — not productionized code
+├── src/
+│   ├── data/            # loading + splitting
+│   ├── features/        # feature engineering (fit on train only)
+│   ├── models/          # model definitions + training entrypoints
+│   └── eval/            # metric + evaluation harness
+├── configs/             # YAML/Hydra hyperparameter configs
+├── experiments/         # tracked run outputs / logs
+├── models/              # saved artifacts (DVC/registry-tracked)
+├── requirements.txt     # pinned
+└── pyproject.toml
+```
+
+---
+
+## Reproducibility Rules
+
+- Set **all seeds** before any randomness: `random.seed(n)`, `numpy.random.seed(n)`, and the framework seed.
+- Enable deterministic flags where feasible; document any performance trade-off.
+- **Pin the environment:** `requirements.txt` or a lockfile committed alongside every artifact.
+- Log the seed, config, and git commit SHA with every tracked run.
+
+```python
+# ✅ Good — set all seeds at entrypoint
+import random, numpy as np, torch
+
+def set_seed(seed: int) -> None:
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.use_deterministic_algorithms(True)
+```
+
+---
+
+## Config Management
+
+- Hyperparameters live in YAML / Hydra / argparse configs — **never hardcoded**.
+- The config file is saved alongside every model artifact.
+
+```python
+# ✅ Good — load config from file
+from dataclasses import dataclass
+import yaml
+
+@dataclass
+class TrainConfig:
+    lr: float
+    max_epochs: int
+    batch_size: int
+    seed: int
+
+def load_config(path: str) -> TrainConfig:
+    with open(path) as f:
+        return TrainConfig(**yaml.safe_load(f))
+
+# ❌ Bad — hyperparameters hardcoded in training script
+lr = 0.001
+epochs = 50
+```
+
+---
+
+## Leakage Prevention
+
+- **Split before fit** — perform the split BEFORE fitting any transformer, scaler, or encoder.
+- **Fit transforms on train fold only** — all preprocessing lives inside a `Pipeline` fit only on training data.
+- **Time-aware splits** — for temporal data, use `TimeSeriesSplit`; never shuffle time-indexed data.
+
+```python
+# ✅ Good — pipeline inside CV; transforms fit on train fold only
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import StratifiedKFold, cross_val_score
+
+pipe = Pipeline([
+    ("scaler", StandardScaler()),
+    ("clf", LogisticRegression()),
+])
+cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
+scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="roc_auc")
+
+# ❌ Bad — scaler fit on full data before split (leakage)
+scaler = StandardScaler().fit(X)
+X_scaled = scaler.transform(X)
+X_train, X_test = train_test_split(X_scaled, ...)
+```
+
+---
+
+## Framework Idioms
+
+### scikit-learn
+
+- Use `Pipeline` + `ColumnTransformer` for all preprocessing — never transform outside a pipeline.
+- Use `set_output(transform="pandas")` (sklearn ≥ 1.2) to preserve feature names.
+
+```python
+from sklearn.compose import ColumnTransformer
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.pipeline import Pipeline
+
+preprocessor = ColumnTransformer([
+    ("num", StandardScaler(), num_cols),
+    ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols),
+])
+pipe = Pipeline([("prep", preprocessor), ("model", clf)])
+```
+
+### PyTorch
+
+- Seed the `DataLoader` worker init function for full reproducibility.
+- Always switch between `model.train()` and `model.eval()` modes; use `torch.no_grad()` for validation.
+
+```python
+import torch
+from torch.utils.data import DataLoader
+
+def seed_worker(worker_id: int) -> None:
+    import random, numpy as np
+    worker_seed = torch.initial_seed() % (2**32)
+    random.seed(worker_seed)
+    np.random.seed(worker_seed)
+
+g = torch.Generator()
+g.manual_seed(42)
+loader = DataLoader(dataset, batch_size=32, worker_init_fn=seed_worker, generator=g)
+
+# Training loop
+model.train()
+for batch in train_loader:
+    ...
+
+# Validation loop
+model.eval()
+with torch.no_grad():
+    for batch in val_loader:
+        ...
+```
+
+### TensorFlow / Keras
+
+- Use callbacks for checkpointing (save best by validation metric) and early stopping.
+- Set `tf.random.set_seed` at startup.
+
+```python
+import tensorflow as tf
+
+tf.random.set_seed(42)
+
+callbacks = [
+    tf.keras.callbacks.ModelCheckpoint(
+        filepath="models/best.keras",
+        monitor="val_loss",
+        save_best_only=True,
+    ),
+    tf.keras.callbacks.EarlyStopping(
+        monitor="val_loss",
+        patience=5,
+        restore_best_weights=True,
+    ),
+]
+model.fit(X_train, y_train, validation_data=(X_val, y_val), callbacks=callbacks)
+```
+
+### HuggingFace (LLM fine-tuning)
+
+- Use a freeze/unfreeze schedule; start with a small learning rate and add warmup.
+- Log the base model name, adapter config, and dataset version alongside the run.
+
+```python
+from transformers import Trainer, TrainingArguments
+
+args = TrainingArguments(
+    output_dir="models/ft-run",
+    num_train_epochs=3,
+    learning_rate=2e-5,
+    warmup_ratio=0.1,
+    seed=42,
+    report_to="mlflow",
+)
+trainer = Trainer(model=model, args=args, train_dataset=train_ds, eval_dataset=val_ds)
+trainer.train()
+```
+
+---
+
+## Experiment Tracking
+
+- Log params, metrics per epoch/fold, seed, data version, and git commit to MLflow or wandb.
+- Name each run per the scheme in `custom/rules/ml-conventions.md`: `[ticket-id]_[approach]_[yyyymmdd-n]`.
+- Version datasets with DVC; store the DVC data hash in the run metadata.
+
+```python
+import mlflow
+
+mlflow.set_experiment("my-project")
+with mlflow.start_run(run_name="ML-42_lgbm-tfidf_20241115-1"):
+    mlflow.log_params({"lr": cfg.lr, "seed": cfg.seed, "data_version": "v1.2"})
+    mlflow.log_metric("val_roc_auc", val_score)
+    mlflow.sklearn.log_model(pipe, artifact_path="model")
+```
+
+---
+
+## Notebook Hygiene
+
+- `notebooks/` is for EDA and exploration only — not production code.
+- Productionized logic (features, models, eval) migrates to `src/` as importable modules.
+- Notebooks must not import from each other; shared utilities go to `src/`.
+- Clear outputs before committing notebooks (or use `nbstripout`).
+
+---
+
+## Testing Rules
+
+- Use **Pytest** for all `src/` utilities.
+- Test the **metric harness** directly: assert known inputs produce the expected metric value.
+- Test **data-split functions** for leakage: verify no sample ID appears in both train and test sets.
+- Test feature-engineering functions independently with synthetic data.
+
+```python
+import pytest
+import numpy as np
+from src.eval.metrics import binary_roc_auc
+from src.data.splits import make_temporal_split
+
+def test_roc_auc_perfect_classifier():
+    y_true = np.array([0, 0, 1, 1])
+    y_score = np.array([0.1, 0.2, 0.8, 0.9])
+    assert binary_roc_auc(y_true, y_score) == pytest.approx(1.0)
+
+def test_temporal_split_no_leakage():
+    train_ids, test_ids = make_temporal_split(df, date_col="event_date", test_months=3)
+    overlap = set(train_ids) & set(test_ids)
+    assert len(overlap) == 0, f"Leakage: {len(overlap)} overlapping IDs"
+```
+
+---
+
+## Naming Conventions
+
+| Element | Convention | Example |
+|---------|-----------|---------|
+| Module/package | snake_case | `feature_engineering.py` |
+| Class | PascalCase | `TemporalSplit`, `TrainConfig` |
+| Function/variable | snake_case | `train_model`, `val_score` |
+| Constant | UPPER_SNAKE_CASE | `DEFAULT_SEED`, `MAX_EPOCHS` |
+| Experiment run | `[ticket-id]_[approach]_[yyyymmdd-n]` | `ML-42_lgbm-tfidf_20241115-1` |
+
+---
+
+## Common Anti-Patterns to Avoid
+
+- ❌ Fitting scalers/encoders on the full dataset before splitting — always fit inside a `Pipeline` on the train fold only
+- ❌ Hardcoded hyperparameters — use configs
+- ❌ No seeds — always set `random`, `numpy`, and framework seeds
+- ❌ Evaluating on validation data used for tuning — keep a held-out test set for final Gate 4 reporting
+- ❌ Production logic in notebooks — migrate to `src/`
+- ❌ Committing data files or model binaries to git — track with DVC or a registry
+- ❌ Single aggregate metric without error analysis — always inspect failure cases and slices

package/custom/templates/shared/ml-gate-workflow.md

@@ -0,0 +1,103 @@
+## MANDATORY: Strict ML Gate Workflow
+
+> **Absolute Rule: Complete Gate N before entering Gate N+1.**
+> **Do not skip, shorten, or merge Gates.**
+
+You have superpowers. When a ticket context exists in `.aiflow/context/current.json`:
+- **Consult the "AI Skill Registry"** above to find each skill's `SKILL.md`.
+- **AUTO-START Gate 1 immediately** — do NOT wait for the developer to ask.
+- **PROJECT CONVENTIONS:** Before writing any output file, read `custom/rules/project-conventions.md`
+  and `custom/rules/ml-conventions.md`. These override upstream skill defaults.
+
+---
+
+### GATE 1 — Problem Framing & Data Understanding (auto-start)
+
+**INVOKE:** `frame-ml-problem` skill, then `explore-data` skill
+
+AI actively reads the ticket and dataset to frame the problem and assess the data:
+1. Read `.aiflow/context/current.json` — ticket info
+2. **Pre-flight: sync with remote** — run `git status --porcelain`; if working tree is clean, run `git pull --ff-only` to pull the latest source from remote. Skip the pull (with a `⚠️` notice to DEV) if there are uncommitted changes, the branch has diverged, or no remote tracking branch exists.
+3. Define the prediction target, evaluation metric, baseline, success threshold, and constraints.
+4. Run EDA; assess data quality and **leakage** (target leakage, train/test contamination, temporal leakage).
+5. If anything is unclear — ask ONE question at a time, wait for reply.
+6. Output `plan/[ticket-id]/ml-problem.md` with:
+   - Problem statement, prediction target, and evaluation metric (with justification)
+   - Baseline definition and success threshold
+   - Data report (shape, quality, missing values, distributions)
+   - Leakage assessment (all three types checked)
+   - Constraints and effort estimate
+7. Display "GATE 1: ML problem doc ready" → wait for **APPROVED**
+
+> **Tip:** If auto-start doesn't trigger, the developer can start this gate by typing: **"start"**, **"Gate 1"** or **"Analyze ticket"**.
+
+DO NOT just check format — **understand the data and propose solutions**.
+
+> **Telemetry:** Run `ak gate 1 start --ticket [ticket-id]` when starting this gate (auto-start or when developer types "start"/"Gate 1"). Run as-is — do NOT append shell redirects like `2>$null` or `>/dev/null`.
+> Run `ak gate 1 approved --ticket [ticket-id]` immediately when APPROVED is received. Run as-is — do NOT append shell redirects.
+
+---
+
+### GATE 2 — Experiment Plan (wait for APPROVED)
+
+**INVOKE:** `design-experiment` skill, then `superpowers:writing-plans`
+
+- Define the validation strategy (CV scheme/splits, leakage prevention), candidate approaches and feature sets, ablation plan, and experiment-tracking setup.
+- Output `plan/[ticket-id]/experiment-plan.md` with the detailed experiment plan.
+- Display: "GATE 2 PAUSED: type APPROVED to start experiments".
+- NO EXPERIMENTS OR TRAINING until "APPROVED" is received.
+
+> **Telemetry:** Run `ak gate 2 start --ticket [ticket-id]` when starting this gate. Run as-is — do NOT append shell redirects.
+> Run `ak gate 2 approved --ticket [ticket-id]` immediately when APPROVED is received. Run as-is — do NOT append shell redirects.
+
+---
+
+### GATE 3 — Implement, Train & Iterate (after APPROVED)
+
+Only runs after Gate 2 has been APPROVED.
+
+**INVOKE:** `train-model` skill
+- For algorithm-optimization tickets EXTRA: `improve-algorithm` skill
+
+**Eval-harness-first rule:** Write the metric/eval harness and establish a reproducible baseline BEFORE iterating on the model. Confirm the baseline runs and is logged — then proceed with tracked experiments.
+
+- Fixed seeds, pinned environment, logged configs — every run.
+- Follow the ablation plan: one change at a time, each tracked.
+
+> **Telemetry:** Run `ak gate 3 start --ticket [ticket-id]` when starting this gate. Run as-is — do NOT append shell redirects.
+
+---
+
+### GATE 4 — Evaluation & Self-Review (wait for APPROVED)
+
+**INVOKE:** `evaluate-model` skill
+
+Mandatory order:
+1. `superpowers:verification-before-completion` — eval runs must reproduce before any success claim
+2. `evaluate-model` skill — held-out metrics vs baseline and threshold, error analysis, overfitting/leakage/drift checks
+3. `impact-analysis` skill — assess data pipeline and downstream blast radius
+4. Tick `custom/rules/ml-review-checklist.md`
+5. Create `plan/[ticket-id]/eval-report.md` and a model card draft
+
+Then: "GATE 4 PAUSED: type APPROVED or BUG: [description]"
+- Coding or pipeline bug → fix → repeat Gate 4.
+- Wrong framing, wrong metric, or wrong data → return to Gate 1.
+
+> **Telemetry:** Run `ak gate 4 start --ticket [ticket-id]` when starting this gate. Run as-is — do NOT append shell redirects.
+> Run `ak gate 4 approved --ticket [ticket-id]` immediately when APPROVED is received. Run as-is — do NOT append shell redirects.
+
+---
+
+### GATE 5 — Package, Deploy & Handoff (after APPROVED)
+
+Only runs after Gate 4 has been APPROVED.
+
+**INVOKE:** `deploy-model` skill, then `superpowers:requesting-code-review`
+
+- Version the artifact (model, training data version, code commit, config) in the registry.
+- Package the full inference pipeline so serving matches training exactly.
+- Define the monitoring plan (input drift, prediction distribution, latency, retraining trigger).
+- Finalize the model card (intended use, data, metrics, limitations, owner).
+- Guide on creating a Pull Request with the ticket link.
+
+> **Telemetry:** Run `ak gate 5 start --ticket [ticket-id]` when starting this gate. Run as-is — do NOT append shell redirects.