ctx-cc 3.5.0 → 4.1.0

package/skills/ctx-ml-experiment/SKILL.md

@@ -0,0 +1,334 @@
+---
+name: ctx-ml-experiment
+description: |
+  WHEN: User wants to run ML experiments, track hypotheses, compare models, analyze results, tune hyperparameters, or manage the experiment loop (hypothesize → design → implement → run → analyze → iterate).
+  WHEN NOT: Software development, UI design, non-ML coding tasks, infrastructure work unrelated to model training or evaluation.
+---
+
+# CTX ML Experiment — Hypothesis-Driven Experiment Lifecycle
+
+You manage the full ML experiment lifecycle: from hypothesis formation through result analysis and iteration. Every experiment is persisted, comparable, and traceable.
+
+## Core Principle
+
+No ad-hoc runs. Every model change starts with a hypothesis, proceeds through a defined design, and concludes with recorded results. The experiment log is the ground truth.
+
+## Experiment Lifecycle
+
+```
+hypothesize → design → implement → run → analyze → iterate
+     ↑                                        │
+     └────────── learn from results ──────────┘
+```
+
+Never skip phases. A run without a hypothesis is noise, not science.
+
+## Directory Structure
+
+Bootstrap `.ctx/ml/` if it does not exist:
+
+```
+.ctx/ml/
+├── experiments/
+│   ├── EXP-001/
+│   │   ├── HYPOTHESIS.md       # What we believe and why
+│   │   ├── DESIGN.md           # How to test it
+│   │   ├── config.yaml         # Reproducible run config
+│   │   ├── RESULTS.md          # Actual outcomes vs expected
+│   │   └── artifacts/          # Model checkpoints, plots, logs
+│   └── EXPERIMENT-LOG.md       # Running table of all experiments
+├── analysis/
+│   ├── EDA-<dataset>.md        # Exploratory data analysis
+│   └── plots/
+├── features/
+│   ├── feature-registry.yaml   # All features, transforms, lineage
+│   └── transforms/             # Reusable transform definitions
+├── models/
+│   ├── registry.yaml           # Model versions, metrics, lineage
+│   └── configs/                # Canonical model configs
+└── ML-STATUS.md                # Current ML project status
+```
+
+## Experiment ID Convention
+
+IDs are sequential: `EXP-001`, `EXP-002`, etc. Read the current max from `EXPERIMENT-LOG.md` and increment. Never reuse IDs.
+
+## HYPOTHESIS.md Format
+
+```markdown
+# EXP-{id}: {Short Hypothesis Title}
+
+**Created**: {ISO date}
+**Author**: {agent or human}
+**Status**: draft | running | concluded
+
+## Hypothesis
+
+{One sentence: "We believe that X will improve Y by Z because of W."}
+
+## Rationale
+
+- {Observation or data point that motivates this}
+- {Prior experiment result that informs this}
+- {Domain knowledge or literature support}
+
+## Expected Outcome
+
+- Primary metric: {metric} improves from {baseline} to {target}
+- Guard rail: {metric} does not degrade below {threshold}
+
+## Null Hypothesis
+
+No meaningful difference between {treatment} and {control} on {metric}.
+
+## Risk
+
+{What would make this hypothesis wrong? What could go wrong?}
+```
+
+## DESIGN.md Format
+
+```markdown
+# EXP-{id}: Experiment Design
+
+## Setup
+
+| Property | Value |
+|----------|-------|
+| Baseline | {model/config used as control} |
+| Treatment | {what changes} |
+| Dataset | {train/val/test splits, sizes} |
+| Random seed | {seed value} |
+
+## Changes from Baseline
+
+- {File or config}: {change description}
+
+## Metrics
+
+| Metric | Direction | Threshold | Notes |
+|--------|-----------|-----------|-------|
+| {primary} | maximize | {value} | promotion gate |
+| {guard} | minimize | {value} | regression gate |
+
+## Evaluation Protocol
+
+1. {Step 1}
+2. {Step 2}
+3. {Step 3}
+
+## Acceptance Criteria
+
+- [ ] Primary metric meets threshold
+- [ ] Guard rail metrics not violated
+- [ ] Training is stable (no loss explosion, no NaN)
+- [ ] Inference latency within budget
+```
+
+## config.yaml Format
+
+```yaml
+experiment_id: EXP-001
+hypothesis: "XGBoost with depth=6 outperforms depth=4 baseline"
+created: "2026-03-25"
+
+data:
+  train: data/train_v2.parquet
+  val: data/val_v2.parquet
+  test: data/test_v2.parquet
+  seed: 42
+
+model:
+  type: xgboost
+  params:
+    max_depth: 6
+    n_estimators: 300
+    learning_rate: 0.05
+    subsample: 0.8
+    colsample_bytree: 0.8
+
+evaluation:
+  primary_metric: auc
+  guard_metrics: [precision, recall]
+  cv_folds: 5
+
+artifacts:
+  model_path: artifacts/model.pkl
+  plots: artifacts/plots/
+  logs: artifacts/train.log
+```
+
+## RESULTS.md Format
+
+```markdown
+# EXP-{id}: Results
+
+**Concluded**: {ISO date}
+**Status**: accepted | rejected | inconclusive
+
+## Outcome
+
+| Metric | Baseline | Result | Delta | Pass? |
+|--------|----------|--------|-------|-------|
+| {primary} | {value} | {value} | {+/-} | yes/no |
+| {guard} | {value} | {value} | {+/-} | yes/no |
+
+## Verdict
+
+{accepted | rejected | inconclusive} — {One sentence reason}
+
+## Key Findings
+
+- {Finding 1}
+- {Finding 2}
+
+## What This Tells Us
+
+{2-3 sentences on what we learned, not just the metric deltas.}
+
+## Next Experiment
+
+{What should EXP-{n+1} test, given these results?}
+```
+
+## EXPERIMENT-LOG.md Format
+
+```markdown
+# ML Experiment Log
+
+| ID | Hypothesis | Model | Primary Metric | Result | Status |
+|----|-----------|-------|---------------|--------|--------|
+| EXP-001 | XGBoost > baseline | XGBoost depth=4 | AUC 0.85 target | AUC 0.82 | rejected |
+| EXP-002 | Feature X improves AUC | XGBoost+feat_x | AUC 0.87 target | AUC 0.87 | accepted |
+| EXP-003 | HPO on accepted config | XGBoost+feat_x tuned | AUC 0.90 target | running | running |
+```
+
+Always append — never delete rows. Mark rejected experiments as `rejected`, not removed.
+
+## Model Registry Format (models/registry.yaml)
+
+```yaml
+models:
+  risk-classifier:
+    current: v3
+    versions:
+      v1:
+        metrics: { auc: 0.82, precision: 0.79 }
+        experiment: EXP-001
+        date: "2026-01-15"
+        status: retired
+        artifacts: .ctx/ml/experiments/EXP-001/artifacts/
+      v2:
+        metrics: { auc: 0.87, precision: 0.84 }
+        experiment: EXP-002
+        date: "2026-02-01"
+        status: retired
+        artifacts: .ctx/ml/experiments/EXP-002/artifacts/
+      v3:
+        metrics: { auc: 0.91, precision: 0.88 }
+        experiment: EXP-005
+        date: "2026-03-10"
+        status: production
+        artifacts: .ctx/ml/experiments/EXP-005/artifacts/
+    promotion_criteria:
+      primary: "auc >= current + 0.02"
+      guard: "precision regression <= 0.01"
+      stability: "training loss converges within 50 epochs"
+```
+
+## Feature Registry Format (features/feature-registry.yaml)
+
+```yaml
+features:
+  age_z_score:
+    type: numeric
+    source: demographics
+    transform: "(age - mean) / std"
+    version: 1
+    created: "2026-01-15"
+    used_by: [risk-classifier, bio-age-model]
+    validated: true
+    notes: "mean=45.2, std=12.1 from train_v1"
+
+  cholesterol_ratio:
+    type: numeric
+    source: labs
+    transform: "hdl / ldl"
+    version: 2
+    created: "2026-02-10"
+    used_by: [risk-classifier]
+    validated: true
+    notes: "v2 clips outliers at 99th percentile"
+```
+
+## ML-STATUS.md Format
+
+```markdown
+# ML Project Status
+
+**Updated**: {ISO date}
+**Active Experiment**: EXP-{n} — {hypothesis title}
+
+## Current Focus
+
+{1-2 sentences on what we are trying to solve right now}
+
+## Recent Results
+
+| EXP | Outcome | Key Learning |
+|-----|---------|--------------|
+| EXP-{n-1} | accepted | {learning} |
+| EXP-{n-2} | rejected | {learning} |
+
+## Blocking Issues
+
+- {Issue if any, else "none"}
+
+## Next Experiments Queued
+
+1. {EXP-{n+1}} — {hypothesis}
+2. {EXP-{n+2}} — {hypothesis}
+```
+
+## Workflow Rules
+
+1. Every experiment gets its own numbered directory. No experiments in ad-hoc files.
+2. `config.yaml` must be committed before running. Results are not reproducible without it.
+3. Record results even for rejected experiments. Negative results have value.
+4. Update `EXPERIMENT-LOG.md` after every concluded experiment — keep it current.
+5. Update `ML-STATUS.md` whenever the active experiment changes.
+6. Never promote a model to registry without a concluded RESULTS.md with accepted status.
+7. Feature registry is append-only with versioning. Old feature versions stay in the file.
+
+## Agent Spawn Patterns
+
+When orchestrating experiment work, spawn agents appropriate to the phase:
+
+```
+Phase: hypothesize → spawn ctx-ml-scientist
+  prompt: "Form a hypothesis for EXP-{n}. Context: {prior results}. Write HYPOTHESIS.md."
+
+Phase: design → spawn ctx-ml-scientist
+  prompt: "Design the experiment for EXP-{n}. Write DESIGN.md and config.yaml."
+
+Phase: implement → spawn ctx-ml-engineer
+  prompt: "Implement training script for EXP-{n} using config.yaml."
+
+Phase: analyze → spawn ctx-ml-analyst
+  prompt: "Analyze results from EXP-{n} run. Write RESULTS.md. Update EXPERIMENT-LOG.md."
+
+Phase: review → spawn ctx-ml-reviewer
+  prompt: "Review EXP-{n} results. Recommend: accept, reject, or run follow-up."
+```
+
+## Iteration Decision Tree
+
+After `analyze`:
+
+```
+Result meets primary metric AND guards hold?
+  YES → Accept. Update model registry. Queue follow-up if headroom remains.
+  NO (primary miss) → Reject. Document learnings. Form new hypothesis.
+  NO (guard violated) → Inconclusive. Address regression before retrying.
+  Unclear → Run validation on holdout set before deciding.
+```