@relipa/ai-flow-kit 0.1.1 → 0.1.2

package/custom/rules/ml-review-checklist.md

@@ -0,0 +1,14 @@
+# ML Review Checklist
+
+Before completing Gate 4 and creating a Pull Request, verify:
+
+- [ ] Evaluation metric matches the problem type and business goal defined in Gate 1.
+- [ ] Model beats the baseline by at least the success threshold defined in Gate 1.
+- [ ] No data leakage — all three types re-checked: target leakage, train/test contamination, and temporal leakage.
+- [ ] Result is reproducible: re-run with the logged seed, pinned environment, and saved config produces the same metric.
+- [ ] No overfitting: train / validation / test metric gap is acceptable and documented.
+- [ ] Error analysis performed: worst slices, classes, or failure cases identified and documented.
+- [ ] Robustness assessed: behavior under noise, shifted inputs, or edge cases verified; fairness across sensitive slices checked where applicable.
+- [ ] All experiments tracked: params, metrics, data version, and git commit SHA logged in MLflow / wandb for every run (including failures).
+- [ ] Model card present: intended use, training data, evaluation metrics, known limitations, and owner documented.
+- [ ] Impact on data pipeline assessed via `impact-analysis` skill and implications documented.

package/custom/skills/deploy-model/SKILL.md

@@ -0,0 +1,90 @@
+---
+name: deploy-model
+description: Package and ship a model — versioned artifact, serving API, model card, and a monitoring plan for production.
+keywords: deploy, serving, mlops, model card, monitoring, versioning, registry
+---
+
+# Deploy Model
+
+> Pairs with `superpowers:requesting-code-review` at Gate 5 — once the model is packaged and the monitoring plan is written, open the PR with the ticket link for peer review before any production rollout.
+
+---
+
+## When to use this skill
+
+- Gate 5 of the ML workflow, after the evaluation has been APPROVED
+- Any time a trained model artifact needs to be promoted from experiment to production
+- When a new model version replaces an existing one in a registry or serving endpoint
+
+---
+
+## Steps
+
+### 1. Version the artifact
+
+Register the model in the artifact store or model registry (MLflow Model Registry, Weights & Biases Artifacts, or equivalent). The registered entry must include:
+
+- Model version number
+- Training data version (DVC hash, dataset URI, or snapshot timestamp)
+- Code commit SHA that produced the artifact
+- Config file used for training (hyperparameters, seed)
+
+Without these four identifiers, the artifact cannot be reproduced or audited. Do not promote an unversioned artifact to production.
+
+### 2. Package the full inference pipeline
+
+Save the complete end-to-end inference pipeline — preprocessing transforms plus the model — as a single serializable object (sklearn `Pipeline`, MLflow `pyfunc` model, ONNX graph, TorchScript, TF SavedModel, etc.). The packaging must guarantee **train/serve parity**: the identical preprocessing that was applied during training must run at inference time, with no manual re-implementation in the serving layer.
+
+Pin the serving environment to the same dependency versions used during training. Document any framework-specific serialization caveats (e.g., custom PyTorch modules, sklearn custom transformers).
+
+### 3. Expose a serving API
+
+Build or configure an inference endpoint with the following properties:
+
+- **Input validation:** reject malformed or out-of-range inputs with a clear error before they reach the model
+- **Batch and timeout handling:** define the maximum batch size and inference timeout; return a graceful error on timeout
+- **Health check endpoint:** a lightweight `/health` or equivalent that confirms the model is loaded and responsive — required for orchestrators and load balancers
+- **Versioned endpoint path:** include the model version in the route so multiple versions can coexist during a canary or A/B rollout
+
+Example minimal stack: FastAPI + uvicorn with a Pydantic input schema for validation.
+
+### 4. Define the monitoring plan
+
+Specify what will be observed in production and what actions each signal triggers. The plan must cover:
+
+- **Input drift:** statistical test (e.g., KS test, PSI) on feature distributions vs the training baseline; alert threshold
+- **Prediction distribution drift:** shift in output scores or class probabilities; alert threshold
+- **Latency:** p50 / p95 / p99 response times; SLA breach alert
+- **Error rate:** HTTP 5xx and model exception rate; alert threshold
+- **Retraining trigger:** the specific condition (drift score, metric degradation, elapsed time, data volume) that initiates a retraining pipeline run
+
+Document the monitoring plan in the model card and link it to the observability tooling (Prometheus, Grafana, Arize, Evidently, etc.).
+
+### 5. Finalize the model card
+
+Update the model card drafted at Gate 4 with:
+
+- Intended use and out-of-scope uses
+- Training data description (source, version, date range, class balance)
+- Evaluation metrics (held-out results, baseline comparison, per-slice breakdown)
+- Known limitations and failure modes identified in error analysis
+- Monitoring and retraining plan summary
+- Owner name and contact; approval date
+
+The model card is the primary audit document for the deployed model.
+
+### 6. Create the PR
+
+Invoke `superpowers:requesting-code-review`. Open a PR that includes: serving code, model card, monitoring plan, and any pipeline changes. Link the ticket. Reference the eval report and the registered artifact version in the PR description.
+
+---
+
+## Completion Checklist
+
+- [ ] Artifact registered with data version, code commit, and config
+- [ ] Full inference pipeline packaged — preprocessing + model, train/serve parity guaranteed
+- [ ] Serving environment pinned to training-time dependency versions
+- [ ] Inference API includes input validation, batch/timeout handling, and health check
+- [ ] Monitoring plan defined: input drift, prediction drift, latency, error rate, retraining trigger
+- [ ] Model card finalized: intended use, data, metrics, limitations, monitoring summary, owner
+- [ ] PR created with ticket link, eval report reference, and artifact version noted

package/custom/skills/design-experiment/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: design-experiment
+description: Design a leakage-safe validation strategy and experiment plan with ablations and tracking before training begins.
+keywords: experiment, validation, cross-validation, ablation, tracking, splits
+---
+
+# Design Experiment
+
+## When to use this skill
+
+- Gate 2, after the problem framing document (`ml-problem.md`) has been APPROVED
+- Any time a new round of experiments is planned after a framing change
+- Before writing any model training code — the plan must exist first
+
+---
+
+## Steps
+
+### 1. Choose the validation strategy
+
+Select the CV scheme that matches the data structure and justify the choice explicitly:
+- **Stratified k-fold:** use when the target is imbalanced; preserves class proportions in every fold
+- **Group k-fold / GroupShuffleSplit:** use when rows belong to named entities (users, patients, stores) and the task requires generalising to unseen entities — random splits would let the same entity appear in train and validation
+- **TimeSeriesSplit:** use for temporal data; always respect chronological order; never shuffle before splitting
+
+State the number of folds (or holdout fraction) and why that choice is appropriate given the dataset size and variance of the metric.
+
+### 2. Define the leakage-safe split protocol
+
+Specify the exact sequence of operations to prevent any information from the validation or test set contaminating the training fold:
+- **Split before fit:** the train/validation boundary is established before any preprocessing step is fitted
+- **Pipeline inside CV:** all transformations (scaling, encoding, imputation, feature selection) are wrapped in a `Pipeline` (sklearn) or equivalent so they are fit on each training fold and applied to the validation fold — never fit on the combined data
+- **Test set quarantine:** the held-out test set is touched only at Gate 4 evaluation; it is never used to tune hyperparameters or select features
+
+### 3. Enumerate candidate approaches and feature sets
+
+List experiments in ascending order of complexity:
+1. **Baseline** (from `frame-ml-problem`): the simplest reference (majority class, median, or current rule)
+2. **Simple model** (e.g. logistic regression, linear regression, decision tree) with a minimal feature set
+3. **Intermediate model** (e.g. gradient-boosted trees) with engineered features
+4. **Complex model** (e.g. neural network, fine-tuned transformer) if warranted by the problem
+
+For each approach, list the feature set explicitly — which raw columns, which engineered features. Separating feature sets from model families makes it possible to attribute performance differences cleanly.
+
+### 4. Define the ablation plan
+
+Describe what each experiment isolates. The rule is **one change at a time**:
+- Vary the model family while holding the feature set constant
+- Add a feature group while holding the model constant
+- Vary a key hyperparameter while holding everything else fixed
+- Document the hypothesis for each ablation (what do you expect and why)
+
+A plan with no ablations is not acceptable — it produces no actionable insight.
+
+### 5. Set up experiment tracking
+
+Define what every tracked run must log before any experiment is executed:
+- **Parameters:** all hyperparameters, including their defaults; the feature set identifier; the validation scheme
+- **Metrics:** primary metric (from Gate 1) per fold + mean ± std; secondary metrics if relevant
+- **Reproducibility metadata:** random seed, data version (hash or DVC tag), git commit SHA
+- Agree on the tracking tool (MLflow, wandb, or simple CSV) and the run-naming convention from `ml-conventions.md` (`[ticket-id]_[approach]_[yyyymmdd-n]`)
+
+### 6. Define the stopping and decision rule
+
+State, before any experiment runs, how you will decide which approach moves forward:
+- Which metric on which split is the decision criterion
+- What constitutes a meaningful improvement over the baseline (link to the success threshold from Gate 1)
+- The budget: maximum number of experiments or wall-clock time before a decision is forced
+
+An open-ended experiment loop with no stopping rule is out of scope for a gated workflow.
+
+### 7. Output the experiment plan
+
+Write all of the above to `plan/[ticket-id]/experiment-plan.md`. The document must be self-contained: a developer who did not attend the framing discussion should be able to reproduce the full experiment sequence from the plan alone.
+
+---
+
+## Completion Checklist
+
+- [ ] Validation scheme chosen (stratified k-fold / group k-fold / time-series split) and justified
+- [ ] Leakage-safe split protocol defined (split before fit; transforms inside pipeline; test set quarantined)
+- [ ] Candidate approaches listed in ascending complexity order
+- [ ] Feature sets enumerated per approach
+- [ ] Ablation plan written (one change at a time; hypothesis stated for each ablation)
+- [ ] Experiment tracking configured (params, metrics, seed, data version, git commit)
+- [ ] Run-naming convention agreed and follows `ml-conventions.md`
+- [ ] Stopping and decision rule stated (metric, threshold, experiment budget)
+- [ ] Full plan written to `plan/[ticket-id]/experiment-plan.md`

package/custom/skills/evaluate-model/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: evaluate-model
+description: Rigorous model evaluation — held-out metrics vs baseline, error analysis, and overfitting/leakage/drift detection with reproducibility verification.
+keywords: evaluation, error analysis, overfitting, leakage, robustness, model card
+---
+
+# Evaluate Model
+
+> Run `superpowers:verification-before-completion` first — evaluation runs must reproduce cleanly before any success claim is made. If results cannot be reproduced from the logged seed and config, the evaluation is not valid.
+
+---
+
+## When to use this skill
+
+- Gate 4 of the ML workflow, after training and iteration are complete
+- Any time a claim about model quality needs to be verified before sharing with stakeholders
+- When preparing the eval report and model card for review
+
+---
+
+## Steps
+
+### 1. Evaluate on the held-out test set
+
+Run inference on the test set — the split that was never touched during training or hyperparameter tuning. Report the primary metric agreed in Gate 1. Where feasible, include a confidence interval (bootstrap CI, or Wilson interval for classification). Report secondary metrics if relevant to the use case.
+
+### 2. Compare to baseline and success threshold
+
+State pass or fail explicitly:
+
+- Did the model beat the baseline (majority class / heuristic / current production model)?
+- Did it meet the success threshold defined in Gate 1?
+
+A result that beats baseline but falls short of the threshold is a fail — report it honestly and return to Gate 3 or Gate 1 as appropriate.
+
+### 3. Error analysis
+
+Identify where and how the model fails:
+
+- Break down the metric by slice (class, demographic, time period, data source, input length, etc.)
+- For classification: examine the confusion matrix; find the most common error patterns
+- Select representative failure cases and inspect them manually — understand the failure mode before claiming the model is production-ready
+
+### 4. Overfitting check
+
+Compare train, validation, and test metrics side by side. Plot learning curves (metric vs training set size or epoch). Characterize the gap:
+
+- Large train/val gap → overfitting; the model did not generalize
+- Large train/test gap after acceptable val → distribution shift or leakage in val
+
+A large gap must be resolved or explained before the evaluation is APPROVED.
+
+### 5. Leakage re-check
+
+Re-run all three leakage checks from `explore-data` on the final pipeline:
+
+1. **Target leakage** — does any feature embed information derived from or computed after the target?
+2. **Train/test contamination** — were any transforms fitted on data that includes test rows? Are there duplicate IDs across splits?
+3. **Temporal leakage** — for time-ordered data, does any feature use information that would not have been available at prediction time?
+
+Document the result of each check. A positive finding is a Gate 1 regression, not a Gate 4 footnote.
+
+### 6. Robustness assessment
+
+Test behavior under distribution shifts and edge cases relevant to the production use case:
+
+- Inject noise or missing values; check metric degradation
+- Test on known difficult slices (rare classes, short inputs, boundary values)
+- If the model affects decisions about people, check for disparate performance across sensitive groups and flag for review
+
+### 7. Verify reproducibility
+
+Re-run the final eval using only the logged seed, the pinned config, and the registered data version. Confirm the reported number matches to within numerical noise. If it does not reproduce, stop — do not proceed to Gate 5 until the discrepancy is resolved.
+
+### 8. Write the eval report and model card
+
+Output `plan/[ticket-id]/eval-report.md` containing: the metric table (baseline, threshold, achieved), error analysis findings, overfitting gap, leakage check results, and robustness notes.
+
+Draft a model card covering: intended use, training data description, evaluation metrics and their context, known limitations, and owner/contact.
+
+---
+
+## Completion Checklist
+
+- [ ] Held-out test metric reported with confidence interval where feasible
+- [ ] Explicit pass/fail against baseline and success threshold stated
+- [ ] Error analysis performed — worst slices, confusion patterns, representative failure cases
+- [ ] Train / validation / test gap inspected; learning curves reviewed
+- [ ] All three leakage types re-checked and documented (target, contamination, temporal)
+- [ ] Robustness assessed under noise, edge cases, and sensitive slices
+- [ ] Result reproduced from logged seed and config — numbers match
+- [ ] `plan/[ticket-id]/eval-report.md` written
+- [ ] Model card drafted (intended use, data, metrics, limitations, owner)

package/custom/skills/explore-data/SKILL.md

@@ -0,0 +1,87 @@
+---
+name: explore-data
+description: Systematic EDA and data-quality assessment with explicit data-leakage detection before modeling.
+keywords: eda, data, leakage, data quality, exploration, dataset
+---
+
+# Explore Data
+
+> This skill borrows the investigative mindset of `superpowers:systematic-debugging` — form
+> hypotheses about the data, verify each one with evidence, and document what you find.
+> Do not skip a check because the data "looks clean".
+
+---
+
+## When to use this skill
+
+- Gate 1, immediately after `frame-ml-problem` has defined the target and metric
+- Any time a new dataset enters the project (new data source, updated extract, external dataset)
+- Before re-training when data distribution changes are suspected
+
+---
+
+## Steps
+
+### 1. Profile the dataset
+
+Establish the basic facts before any analysis:
+- **Shape:** row count, column count; confirm against expected data volume from the ticket
+- **dtypes:** numeric, categorical, datetime, free text, binary — flag any type mismatches (e.g. numeric column stored as string)
+- **Missing values:** per-column missing rate; distinguish structural missing (feature does not apply) from random missing (collection gap)
+- **Duplicates:** exact duplicate rows; near-duplicate rows (same entity, slightly different values); duplicate IDs that should be unique
+- **Target distribution:** class balance for classification (report majority-class baseline accuracy); value range, skew, and outliers for regression
+
+### 2. Univariate and bivariate analysis
+
+- **Univariate:** distribution shape, outliers, value ranges, cardinality of categoricals
+- **Bivariate:** feature–target correlations or mutual information; flag suspiciously high correlations that may indicate leakage (see Step 3)
+- **Temporal trends:** if a time column exists, plot target rate and key features over time to detect distribution shift
+
+### 3. Leakage detection (mandatory)
+
+Run all three checks explicitly and document the result of each — do not skip any.
+
+**(a) Target leakage** — features derived from or computed after the target event:
+- Identify any column that could not be known at prediction time (e.g. a "days to resolution" column when predicting whether a ticket will be resolved)
+- Drop or exclude any such column before training
+- Flag columns with suspiciously high predictive power; investigate their origin
+
+**(b) Train/test contamination** — information bleeding between splits:
+- Check for duplicate or near-duplicate rows that may span the train and test sets
+- Verify that entity IDs (user IDs, order IDs) do not appear in both splits if the task requires generalising to unseen entities
+- Confirm that scalers, encoders, and imputers will be fit on the training fold only — fitting on the full dataset before splitting is a hard error
+
+**(c) Temporal leakage** — using future data to predict the past:
+- If the data has a time dimension, verify that all features for a given row reflect information available strictly before the prediction timestamp
+- Check for lag features or aggregation windows that inadvertently include the target period
+- Time-series splits must respect chronological order; random splits are wrong for temporal data
+
+### 4. Data-quality flags
+
+Document any of the following that are present:
+- **Inconsistent units:** mixed currencies, mixed date formats, mixed length units
+- **Label noise:** suspiciously contradictory labels for near-identical features
+- **Sampling bias:** is the historical sample representative of the production population? Check for selection effects (e.g. data collected only from active users)
+- **Train / serving distribution drift:** if serving statistics are available, compare them to training distributions on key features; flag deviations above a sensible threshold
+
+### 5. Record findings in the Gate 1 data report
+
+Write the EDA results into the `ml-problem.md` file created by `frame-ml-problem`:
+- Data shape, quality issues, and missing-value strategy
+- Leakage assessment — one paragraph per leakage type stating what was checked and what was found
+- Cleaning actions required before training (drop columns, impute, clip outliers, re-encode)
+
+---
+
+## Completion Checklist
+
+- [ ] Dataset shape, dtypes, and missing-value rates profiled
+- [ ] Duplicate rows and duplicate IDs checked
+- [ ] Target distribution inspected (class balance or value distribution)
+- [ ] Univariate and bivariate analysis completed; suspiciously high correlations investigated
+- [ ] **Target leakage** checked and documented (no future-derived features in the feature set)
+- [ ] **Train/test contamination** checked and documented (no ID overlap, no full-data fitting)
+- [ ] **Temporal leakage** checked and documented (chronological integrity verified if applicable)
+- [ ] Data-quality issues listed (units, label noise, sampling bias, drift)
+- [ ] Required cleaning steps noted
+- [ ] All findings written into `plan/[ticket-id]/ml-problem.md` data report section

package/custom/skills/frame-ml-problem/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: frame-ml-problem
+description: Frame an ML problem before any modeling — define the prediction target, the correct evaluation metric, a baseline, and the success threshold.
+keywords: ml, problem framing, metric, baseline, target, success criteria
+---
+
+# Frame ML Problem
+
+> This skill **inherits and extends** `superpowers:brainstorming` for collaborative Q&A.
+> First, invoke `superpowers:brainstorming` to explore intent and ambiguities with the developer,
+> then apply the structured framing steps below.
+
+---
+
+## When to use this skill
+
+- Gate 1 of the ML workflow — always the first step before any modeling
+- Any new ML ticket where the prediction target or success criterion is not yet explicit
+- Whenever the evaluation metric has not been pinned down or justified against the business goal
+
+---
+
+## Steps
+
+### 1. Read the ticket context
+
+Read `.aiflow/context/current.json`:
+- `description` — what the stakeholder wants the model to do
+- `comments` — additional context, constraints, or prior attempts
+- `context.files` — related notebooks, data samples, or prior model artifacts
+
+### 2. State the prediction target precisely
+
+Define what is being predicted, at what granularity (row level, user level, session level, etc.), and crucially **when the input features are available** relative to the prediction moment. Guard against accidentally incorporating future information — any feature computed from or derived after the target event is a candidate for leakage.
+
+### 3. Classify the problem type
+
+Explicitly label the task:
+- **Classification** (binary or multi-class): predict a category
+- **Regression**: predict a continuous value
+- **Ranking / Recommendation**: order items by relevance
+- **Generation / Seq-to-seq**: produce text, code, or structured output
+- **Clustering / Anomaly detection**: unsupervised grouping or outlier detection
+
+State the type before choosing any model family or metric.
+
+### 4. Choose the evaluation metric and justify it
+
+Select the primary metric and tie it explicitly to the business goal:
+- Imbalanced classification: prefer F1, PR-AUC, or recall at fixed precision over plain accuracy
+- Regression: choose MAE (robust to outliers) vs RMSE (penalises large errors) deliberately
+- Ranking: use MAP, NDCG, or MRR depending on position sensitivity
+- Note the difference between the **offline proxy metric** (what can be measured from historical data) and the **online KPI** (what the business ultimately cares about); acknowledge any gap
+
+**State the metric before any modeling begins. It must not change after training.**
+
+### 5. Define a baseline and success threshold
+
+- **Baseline:** the simplest defensible reference — majority-class prediction, median value, current rule-based system, or the model already in production
+- **Success threshold:** the quantitative improvement over baseline that is meaningful to the business (e.g. "+3 pp F1 on the held-out set" or "<10 ms p99 latency with no accuracy regression")
+- A threshold of "as high as possible" is not acceptable — make it concrete
+
+### 6. List constraints
+
+Document non-negotiable constraints that shape model choice and architecture:
+- **Latency / throughput:** real-time vs batch; maximum inference time
+- **Memory / hardware:** CPU-only, GPU budget, edge deployment
+- **Interpretability:** regulatory or stakeholder requirements for explanations
+- **Data volume:** row count, feature count, labelling cost
+- **Compliance:** PII handling, data residency, auditability requirements
+
+### 7. Ask one clarifying question at a time when ambiguous
+
+If any of the above cannot be answered from the ticket, ask exactly **one question**, wait for a reply, and continue. Do not bundle multiple questions.
+
+---
+
+## Completion Checklist
+
+- [ ] Ticket context read (description + comments + related files)
+- [ ] Prediction target defined with the exact availability time of input features
+- [ ] Problem type classified (classification / regression / ranking / generation / clustering)
+- [ ] Evaluation metric chosen and justified against the business goal
+- [ ] Offline proxy metric vs online KPI gap acknowledged
+- [ ] Baseline defined (majority class, heuristic, or current production model)
+- [ ] Success threshold quantified (not "as high as possible")
+- [ ] Constraints listed (latency, memory, interpretability, data volume, compliance)
+- [ ] Findings written to `plan/[ticket-id]/ml-problem.md`

package/custom/skills/improve-algorithm/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: improve-algorithm
+description: Systematically improve an existing model or algorithm — profile, hypothesize, ablate, and measure accuracy/latency/memory. No guessing.
+keywords: optimization, algorithm, performance, latency, accuracy, ablation, profiling
+---
+
+# Improve Algorithm
+
+> This skill inherits the discipline of `superpowers:systematic-debugging` — measure before changing, change one thing at a time, measure again. Guessing is not a method; every change must be motivated by evidence from profiling or error analysis.
+
+---
+
+## When to use this skill
+
+- Gate 3, for tickets specifically about improving an existing model or algorithm (accuracy, latency, or memory targets)
+- Any time a model already exists and the goal is to make it better rather than build from scratch
+- When performance regressions are reported in production
+
+---
+
+## Steps
+
+### 1. Establish the current baseline
+
+Run the existing system through the eval harness and record the exact number — metric, split, and seed. This is the floor that every proposed change must be measured against. If no eval harness exists yet, invoke `train-model` first to establish one.
+
+### 2. Profile before optimizing
+
+Do not form a hypothesis before you know where the bottleneck is.
+
+- **Accuracy bottleneck:** run error analysis by slice and by class. Plot learning curves to distinguish bias (underfitting — train error high) from variance (overfitting — large train/val gap). Identify the failure pattern first.
+- **Latency/memory bottleneck:** profile the hot path end-to-end. Tools: `cProfile` / `line_profiler` for Python; `torch.profiler` or `torch.autograd.profiler` for PyTorch; `tf.profiler` for TF. Measure throughput (samples/sec) and peak memory before and after every change.
+
+Never skip this step. Optimizing the wrong part wastes time and can introduce regressions.
+
+### 3. Form ONE hypothesis
+
+Based on the profiling/error-analysis evidence, form exactly one hypothesis tied to the identified bottleneck. Examples:
+
+- "Error analysis shows the model fails on short-text inputs → hypothesis: adding n-gram features reduces error on that slice"
+- "Profiler shows 60% of latency in the preprocessing step → hypothesis: vectorizing the transform halves latency"
+- "Learning curves show high variance → hypothesis: stronger L2 regularization closes the train/val gap"
+
+State the hypothesis explicitly before writing any code.
+
+### 4. Run a single ablation
+
+Implement only the change implied by the hypothesis. Run the eval harness with the same seed, same split, same data version. Track the run in MLflow/wandb with a descriptive name and note the hypothesis in the run description. Compare to the baseline number directly.
+
+### 5. Decide: keep or revert
+
+Keep the change only if it:
+
+- Beats the baseline meaningfully (respects the success threshold from Gate 1)
+- Respects all constraints: accuracy/latency/memory trade-off stated explicitly
+- Does not introduce a new regression in a different slice or metric
+
+Revert if it does not satisfy the above. Log the failure reason; failed experiments are also information.
+
+### 6. Repeat and document
+
+Repeat steps 3–5 for the next bottleneck. Log every attempt — including reversions — so the improvement story is fully reproducible. The final state must be explainable: "we tried X (failed because Y), then tried Z (improved metric by W%)."
+
+---
+
+## Completion Checklist
+
+- [ ] Baseline number recorded on the eval harness before any change
+- [ ] Bottleneck identified by profiling (latency/memory) or error analysis (accuracy) — not guessed
+- [ ] Each change is a single isolated ablation motivated by a stated hypothesis
+- [ ] Every attempt (including failures) measured on the same harness and tracked with params/metrics
+- [ ] Trade-offs between accuracy, latency, and memory stated explicitly for kept changes
+- [ ] Kept changes beat the baseline by the success threshold
+- [ ] Full improvement log available (hypothesis → result → decision for each attempt)

package/custom/skills/train-model/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: train-model
+description: Reproducible model training — eval-harness-first, fixed seeds, pinned environment, logged configs, checkpointing, experiment tracking.
+keywords: training, reproducibility, seed, checkpoint, mlflow, wandb, fine-tune
+---
+
+# Train Model
+
+> **Eval-harness-first is the ML analog of `superpowers:test-driven-development`** — build the metric/eval harness and a reproducible baseline BEFORE writing any model improvement code. If the harness does not run clean, stop and fix it first.
+
+---
+
+## When to use this skill
+
+- Gate 3 of the ML workflow, after the experiment plan has been APPROVED
+- Any time a new model or training run is initiated
+- Fine-tuning or transfer-learning tasks where reproducibility is critical
+
+---
+
+## Steps
+
+### 1. Eval-harness-first
+
+Implement the metric function and the validation loop as defined in the approved experiment plan. Run it against the baseline (majority class, heuristic, or current production model). Confirm you get a reproducible number before touching any model code. This number is the target to beat.
+
+### 2. Enforce reproducibility
+
+Set all random seeds at the top of every training script:
+
+- `random.seed(seed)` and `numpy.random.seed(seed)`
+- Framework seed: `torch.manual_seed(seed)` + `torch.cuda.manual_seed_all(seed)` for PyTorch; `tf.random.set_seed(seed)` for TF/Keras
+- Enable deterministic ops where feasible (`torch.use_deterministic_algorithms(True)`)
+
+Pin the environment: `requirements.txt` or a lockfile (`pip freeze > requirements.txt`). No floating version ranges in the pinned file.
+
+### 3. Config over hardcoding
+
+Place all hyperparameters (learning rate, batch size, number of layers, regularization strength, etc.) in a YAML, Hydra config, or argparse definition — never as literals in the training script. This ensures any run can be exactly reproduced from its config file alone.
+
+### 4. Experiment tracking
+
+Log the following to MLflow or wandb for every run:
+
+- All hyperparameters (from config)
+- Metric per epoch / per fold
+- Seed, data version (DVC hash or path), and git commit SHA
+- Run name following the naming convention in `custom/rules/ml-conventions.md`: `[ticket-id]_[approach]_[yyyymmdd-n]`
+
+Do not leave runs unnamed or untracked.
+
+### 5. Checkpointing
+
+Save the best checkpoint by the primary validation metric (not the final epoch). Store the config file alongside the artifact so the checkpoint is self-contained. Use the model registry (MLflow Model Registry or equivalent) when available.
+
+### 6. Framework-specific considerations
+
+- **scikit-learn:** wrap preprocessing + model in a single `Pipeline` so transforms are fitted on training data only — never on the full dataset.
+- **PyTorch:** seed `DataLoader` workers (`worker_init_fn`), toggle `model.train()` before training and `model.eval()` before validation/inference. Use `torch.no_grad()` for inference.
+- **TF/Keras:** use callbacks for `ModelCheckpoint` (save best) and `EarlyStopping`; set `tf.config.experimental.enable_op_determinism()` if needed.
+- **Fine-tuning:** apply a freeze/unfreeze schedule; use LR warmup before unfreezing. Log the freeze schedule as a config parameter.
+
+### 7. Iterate per the ablation plan
+
+Follow the ablation plan from the experiment design — change exactly one factor per run, track it, compare to the established baseline on the same harness. Do not run ad-hoc experiments that are not tracked or not comparable.
+
+---
+
+## Completion Checklist
+
+- [ ] Eval harness implemented and confirmed running on baseline before any model change
+- [ ] Reproducible baseline number recorded
+- [ ] All random seeds set (language, numpy, framework, CUDA)
+- [ ] Environment pinned (requirements.txt or lockfile committed)
+- [ ] All hyperparameters in a config file — no literals in training code
+- [ ] Every run tracked with params, metrics, seed, data version, and git commit
+- [ ] Best checkpoint saved with its config alongside the artifact
+- [ ] Experiments follow the ablation plan (one change at a time)