@relipa/ai-flow-kit 0.1.1-beta.0 → 0.1.2-beta.0

package/custom/skills/deploy-model/SKILL.md

@@ -0,0 +1,90 @@
+---
+name: deploy-model
+description: Package and ship a model — versioned artifact, serving API, model card, and a monitoring plan for production.
+keywords: deploy, serving, mlops, model card, monitoring, versioning, registry
+---
+
+# Deploy Model
+
+> Pairs with `superpowers:requesting-code-review` at Gate 5 — once the model is packaged and the monitoring plan is written, open the PR with the ticket link for peer review before any production rollout.
+
+---
+
+## When to use this skill
+
+- Gate 5 of the ML workflow, after the evaluation has been APPROVED
+- Any time a trained model artifact needs to be promoted from experiment to production
+- When a new model version replaces an existing one in a registry or serving endpoint
+
+---
+
+## Steps
+
+### 1. Version the artifact
+
+Register the model in the artifact store or model registry (MLflow Model Registry, Weights & Biases Artifacts, or equivalent). The registered entry must include:
+
+- Model version number
+- Training data version (DVC hash, dataset URI, or snapshot timestamp)
+- Code commit SHA that produced the artifact
+- Config file used for training (hyperparameters, seed)
+
+Without these four identifiers, the artifact cannot be reproduced or audited. Do not promote an unversioned artifact to production.
+
+### 2. Package the full inference pipeline
+
+Save the complete end-to-end inference pipeline — preprocessing transforms plus the model — as a single serializable object (sklearn `Pipeline`, MLflow `pyfunc` model, ONNX graph, TorchScript, TF SavedModel, etc.). The packaging must guarantee **train/serve parity**: the identical preprocessing that was applied during training must run at inference time, with no manual re-implementation in the serving layer.
+
+Pin the serving environment to the same dependency versions used during training. Document any framework-specific serialization caveats (e.g., custom PyTorch modules, sklearn custom transformers).
+
+### 3. Expose a serving API
+
+Build or configure an inference endpoint with the following properties:
+
+- **Input validation:** reject malformed or out-of-range inputs with a clear error before they reach the model
+- **Batch and timeout handling:** define the maximum batch size and inference timeout; return a graceful error on timeout
+- **Health check endpoint:** a lightweight `/health` or equivalent that confirms the model is loaded and responsive — required for orchestrators and load balancers
+- **Versioned endpoint path:** include the model version in the route so multiple versions can coexist during a canary or A/B rollout
+
+Example minimal stack: FastAPI + uvicorn with a Pydantic input schema for validation.
+
+### 4. Define the monitoring plan
+
+Specify what will be observed in production and what actions each signal triggers. The plan must cover:
+
+- **Input drift:** statistical test (e.g., KS test, PSI) on feature distributions vs the training baseline; alert threshold
+- **Prediction distribution drift:** shift in output scores or class probabilities; alert threshold
+- **Latency:** p50 / p95 / p99 response times; SLA breach alert
+- **Error rate:** HTTP 5xx and model exception rate; alert threshold
+- **Retraining trigger:** the specific condition (drift score, metric degradation, elapsed time, data volume) that initiates a retraining pipeline run
+
+Document the monitoring plan in the model card and link it to the observability tooling (Prometheus, Grafana, Arize, Evidently, etc.).
+
+### 5. Finalize the model card
+
+Update the model card drafted at Gate 4 with:
+
+- Intended use and out-of-scope uses
+- Training data description (source, version, date range, class balance)
+- Evaluation metrics (held-out results, baseline comparison, per-slice breakdown)
+- Known limitations and failure modes identified in error analysis
+- Monitoring and retraining plan summary
+- Owner name and contact; approval date
+
+The model card is the primary audit document for the deployed model.
+
+### 6. Create the PR
+
+Invoke `superpowers:requesting-code-review`. Open a PR that includes: serving code, model card, monitoring plan, and any pipeline changes. Link the ticket. Reference the eval report and the registered artifact version in the PR description.
+
+---
+
+## Completion Checklist
+
+- [ ] Artifact registered with data version, code commit, and config
+- [ ] Full inference pipeline packaged — preprocessing + model, train/serve parity guaranteed
+- [ ] Serving environment pinned to training-time dependency versions
+- [ ] Inference API includes input validation, batch/timeout handling, and health check
+- [ ] Monitoring plan defined: input drift, prediction drift, latency, error rate, retraining trigger
+- [ ] Model card finalized: intended use, data, metrics, limitations, monitoring summary, owner
+- [ ] PR created with ticket link, eval report reference, and artifact version noted

package/custom/skills/design-experiment/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: design-experiment
+description: Design a leakage-safe validation strategy and experiment plan with ablations and tracking before training begins.
+keywords: experiment, validation, cross-validation, ablation, tracking, splits
+---
+
+# Design Experiment
+
+## When to use this skill
+
+- Gate 2, after the problem framing document (`ml-problem.md`) has been APPROVED
+- Any time a new round of experiments is planned after a framing change
+- Before writing any model training code — the plan must exist first
+
+---
+
+## Steps
+
+### 1. Choose the validation strategy
+
+Select the CV scheme that matches the data structure and justify the choice explicitly:
+- **Stratified k-fold:** use when the target is imbalanced; preserves class proportions in every fold
+- **Group k-fold / GroupShuffleSplit:** use when rows belong to named entities (users, patients, stores) and the task requires generalising to unseen entities — random splits would let the same entity appear in train and validation
+- **TimeSeriesSplit:** use for temporal data; always respect chronological order; never shuffle before splitting
+
+State the number of folds (or holdout fraction) and why that choice is appropriate given the dataset size and variance of the metric.
+
+### 2. Define the leakage-safe split protocol
+
+Specify the exact sequence of operations to prevent any information from the validation or test set contaminating the training fold:
+- **Split before fit:** the train/validation boundary is established before any preprocessing step is fitted
+- **Pipeline inside CV:** all transformations (scaling, encoding, imputation, feature selection) are wrapped in a `Pipeline` (sklearn) or equivalent so they are fit on each training fold and applied to the validation fold — never fit on the combined data
+- **Test set quarantine:** the held-out test set is touched only at Gate 4 evaluation; it is never used to tune hyperparameters or select features
+
+### 3. Enumerate candidate approaches and feature sets
+
+List experiments in ascending order of complexity:
+1. **Baseline** (from `frame-ml-problem`): the simplest reference (majority class, median, or current rule)
+2. **Simple model** (e.g. logistic regression, linear regression, decision tree) with a minimal feature set
+3. **Intermediate model** (e.g. gradient-boosted trees) with engineered features
+4. **Complex model** (e.g. neural network, fine-tuned transformer) if warranted by the problem
+
+For each approach, list the feature set explicitly — which raw columns, which engineered features. Separating feature sets from model families makes it possible to attribute performance differences cleanly.
+
+### 4. Define the ablation plan
+
+Describe what each experiment isolates. The rule is **one change at a time**:
+- Vary the model family while holding the feature set constant
+- Add a feature group while holding the model constant
+- Vary a key hyperparameter while holding everything else fixed
+- Document the hypothesis for each ablation (what do you expect and why)
+
+A plan with no ablations is not acceptable — it produces no actionable insight.
+
+### 5. Set up experiment tracking
+
+Define what every tracked run must log before any experiment is executed:
+- **Parameters:** all hyperparameters, including their defaults; the feature set identifier; the validation scheme
+- **Metrics:** primary metric (from Gate 1) per fold + mean ± std; secondary metrics if relevant
+- **Reproducibility metadata:** random seed, data version (hash or DVC tag), git commit SHA
+- Agree on the tracking tool (MLflow, wandb, or simple CSV) and the run-naming convention from `ml-conventions.md` (`[ticket-id]_[approach]_[yyyymmdd-n]`)
+
+### 6. Define the stopping and decision rule
+
+State, before any experiment runs, how you will decide which approach moves forward:
+- Which metric on which split is the decision criterion
+- What constitutes a meaningful improvement over the baseline (link to the success threshold from Gate 1)
+- The budget: maximum number of experiments or wall-clock time before a decision is forced
+
+An open-ended experiment loop with no stopping rule is out of scope for a gated workflow.
+
+### 7. Output the experiment plan
+
+Write all of the above to `plan/[ticket-id]/experiment-plan.md`. The document must be self-contained: a developer who did not attend the framing discussion should be able to reproduce the full experiment sequence from the plan alone.
+
+---
+
+## Completion Checklist
+
+- [ ] Validation scheme chosen (stratified k-fold / group k-fold / time-series split) and justified
+- [ ] Leakage-safe split protocol defined (split before fit; transforms inside pipeline; test set quarantined)
+- [ ] Candidate approaches listed in ascending complexity order
+- [ ] Feature sets enumerated per approach
+- [ ] Ablation plan written (one change at a time; hypothesis stated for each ablation)
+- [ ] Experiment tracking configured (params, metrics, seed, data version, git commit)
+- [ ] Run-naming convention agreed and follows `ml-conventions.md`
+- [ ] Stopping and decision rule stated (metric, threshold, experiment budget)
+- [ ] Full plan written to `plan/[ticket-id]/experiment-plan.md`

package/custom/skills/evaluate-model/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: evaluate-model
+description: Rigorous model evaluation — held-out metrics vs baseline, error analysis, and overfitting/leakage/drift detection with reproducibility verification.
+keywords: evaluation, error analysis, overfitting, leakage, robustness, model card
+---
+
+# Evaluate Model
+
+> Run `superpowers:verification-before-completion` first — evaluation runs must reproduce cleanly before any success claim is made. If results cannot be reproduced from the logged seed and config, the evaluation is not valid.
+
+---
+
+## When to use this skill
+
+- Gate 4 of the ML workflow, after training and iteration are complete
+- Any time a claim about model quality needs to be verified before sharing with stakeholders
+- When preparing the eval report and model card for review
+
+---
+
+## Steps
+
+### 1. Evaluate on the held-out test set
+
+Run inference on the test set — the split that was never touched during training or hyperparameter tuning. Report the primary metric agreed in Gate 1. Where feasible, include a confidence interval (bootstrap CI, or Wilson interval for classification). Report secondary metrics if relevant to the use case.
+
+### 2. Compare to baseline and success threshold
+
+State pass or fail explicitly:
+
+- Did the model beat the baseline (majority class / heuristic / current production model)?
+- Did it meet the success threshold defined in Gate 1?
+
+A result that beats baseline but falls short of the threshold is a fail — report it honestly and return to Gate 3 or Gate 1 as appropriate.
+
+### 3. Error analysis
+
+Identify where and how the model fails:
+
+- Break down the metric by slice (class, demographic, time period, data source, input length, etc.)
+- For classification: examine the confusion matrix; find the most common error patterns
+- Select representative failure cases and inspect them manually — understand the failure mode before claiming the model is production-ready
+
+### 4. Overfitting check
+
+Compare train, validation, and test metrics side by side. Plot learning curves (metric vs training set size or epoch). Characterize the gap:
+
+- Large train/val gap → overfitting; the model did not generalize
+- Large train/test gap after acceptable val → distribution shift or leakage in val
+
+A large gap must be resolved or explained before the evaluation is APPROVED.
+
+### 5. Leakage re-check
+
+Re-run all three leakage checks from `explore-data` on the final pipeline:
+
+1. **Target leakage** — does any feature embed information derived from or computed after the target?
+2. **Train/test contamination** — were any transforms fitted on data that includes test rows? Are there duplicate IDs across splits?
+3. **Temporal leakage** — for time-ordered data, does any feature use information that would not have been available at prediction time?
+
+Document the result of each check. A positive finding is a Gate 1 regression, not a Gate 4 footnote.
+
+### 6. Robustness assessment
+
+Test behavior under distribution shifts and edge cases relevant to the production use case:
+
+- Inject noise or missing values; check metric degradation
+- Test on known difficult slices (rare classes, short inputs, boundary values)
+- If the model affects decisions about people, check for disparate performance across sensitive groups and flag for review
+
+### 7. Verify reproducibility
+
+Re-run the final eval using only the logged seed, the pinned config, and the registered data version. Confirm the reported number matches to within numerical noise. If it does not reproduce, stop — do not proceed to Gate 5 until the discrepancy is resolved.
+
+### 8. Write the eval report and model card
+
+Output `plan/[ticket-id]/eval-report.md` containing: the metric table (baseline, threshold, achieved), error analysis findings, overfitting gap, leakage check results, and robustness notes.
+
+Draft a model card covering: intended use, training data description, evaluation metrics and their context, known limitations, and owner/contact.
+
+---
+
+## Completion Checklist
+
+- [ ] Held-out test metric reported with confidence interval where feasible
+- [ ] Explicit pass/fail against baseline and success threshold stated
+- [ ] Error analysis performed — worst slices, confusion patterns, representative failure cases
+- [ ] Train / validation / test gap inspected; learning curves reviewed
+- [ ] All three leakage types re-checked and documented (target, contamination, temporal)
+- [ ] Robustness assessed under noise, edge cases, and sensitive slices
+- [ ] Result reproduced from logged seed and config — numbers match
+- [ ] `plan/[ticket-id]/eval-report.md` written
+- [ ] Model card drafted (intended use, data, metrics, limitations, owner)

package/custom/skills/explore-data/SKILL.md

@@ -0,0 +1,87 @@
+---
+name: explore-data
+description: Systematic EDA and data-quality assessment with explicit data-leakage detection before modeling.
+keywords: eda, data, leakage, data quality, exploration, dataset
+---
+
+# Explore Data
+
+> This skill borrows the investigative mindset of `superpowers:systematic-debugging` — form
+> hypotheses about the data, verify each one with evidence, and document what you find.
+> Do not skip a check because the data "looks clean".
+
+---
+
+## When to use this skill
+
+- Gate 1, immediately after `frame-ml-problem` has defined the target and metric
+- Any time a new dataset enters the project (new data source, updated extract, external dataset)
+- Before re-training when data distribution changes are suspected
+
+---
+
+## Steps
+
+### 1. Profile the dataset
+
+Establish the basic facts before any analysis:
+- **Shape:** row count, column count; confirm against expected data volume from the ticket
+- **dtypes:** numeric, categorical, datetime, free text, binary — flag any type mismatches (e.g. numeric column stored as string)
+- **Missing values:** per-column missing rate; distinguish structural missing (feature does not apply) from random missing (collection gap)
+- **Duplicates:** exact duplicate rows; near-duplicate rows (same entity, slightly different values); duplicate IDs that should be unique
+- **Target distribution:** class balance for classification (report majority-class baseline accuracy); value range, skew, and outliers for regression
+
+### 2. Univariate and bivariate analysis
+
+- **Univariate:** distribution shape, outliers, value ranges, cardinality of categoricals
+- **Bivariate:** feature–target correlations or mutual information; flag suspiciously high correlations that may indicate leakage (see Step 3)
+- **Temporal trends:** if a time column exists, plot target rate and key features over time to detect distribution shift
+
+### 3. Leakage detection (mandatory)
+
+Run all three checks explicitly and document the result of each — do not skip any.
+
+**(a) Target leakage** — features derived from or computed after the target event:
+- Identify any column that could not be known at prediction time (e.g. a "days to resolution" column when predicting whether a ticket will be resolved)
+- Drop or exclude any such column before training
+- Flag columns with suspiciously high predictive power; investigate their origin
+
+**(b) Train/test contamination** — information bleeding between splits:
+- Check for duplicate or near-duplicate rows that may span the train and test sets
+- Verify that entity IDs (user IDs, order IDs) do not appear in both splits if the task requires generalising to unseen entities
+- Confirm that scalers, encoders, and imputers will be fit on the training fold only — fitting on the full dataset before splitting is a hard error
+
+**(c) Temporal leakage** — using future data to predict the past:
+- If the data has a time dimension, verify that all features for a given row reflect information available strictly before the prediction timestamp
+- Check for lag features or aggregation windows that inadvertently include the target period
+- Time-series splits must respect chronological order; random splits are wrong for temporal data
+
+### 4. Data-quality flags
+
+Document any of the following that are present:
+- **Inconsistent units:** mixed currencies, mixed date formats, mixed length units
+- **Label noise:** suspiciously contradictory labels for near-identical features
+- **Sampling bias:** is the historical sample representative of the production population? Check for selection effects (e.g. data collected only from active users)
+- **Train / serving distribution drift:** if serving statistics are available, compare them to training distributions on key features; flag deviations above a sensible threshold
+
+### 5. Record findings in the Gate 1 data report
+
+Write the EDA results into the `ml-problem.md` file created by `frame-ml-problem`:
+- Data shape, quality issues, and missing-value strategy
+- Leakage assessment — one paragraph per leakage type stating what was checked and what was found
+- Cleaning actions required before training (drop columns, impute, clip outliers, re-encode)
+
+---
+
+## Completion Checklist
+
+- [ ] Dataset shape, dtypes, and missing-value rates profiled
+- [ ] Duplicate rows and duplicate IDs checked
+- [ ] Target distribution inspected (class balance or value distribution)
+- [ ] Univariate and bivariate analysis completed; suspiciously high correlations investigated
+- [ ] **Target leakage** checked and documented (no future-derived features in the feature set)
+- [ ] **Train/test contamination** checked and documented (no ID overlap, no full-data fitting)
+- [ ] **Temporal leakage** checked and documented (chronological integrity verified if applicable)
+- [ ] Data-quality issues listed (units, label noise, sampling bias, drift)
+- [ ] Required cleaning steps noted
+- [ ] All findings written into `plan/[ticket-id]/ml-problem.md` data report section