ctx-cc 3.5.0 → 4.0.0

package/commands/ml-status.md

@@ -0,0 +1,197 @@
+---
+name: ctx:ml-status
+description: Show ML project status — experiments, models, features, drift alerts. Read-only dashboard. No agents spawned.
+---
+
+<objective>
+Display a concise ML project dashboard. Read-only. Shows active experiment, recent experiment history, production model versions, feature count, and any drift alerts. No writes, no agents.
+</objective>
+
+<usage>
+```bash
+/ctx:ml-status          # Full dashboard
+/ctx:ml-status models   # Model registry only
+/ctx:ml-status exp      # Experiment log only
+/ctx:ml-status drift    # Drift alerts only
+```
+</usage>
+
+<process>
+
+## Step 1: Check ML Directory Exists
+
+If `.ctx/ml/` does not exist:
+```
+[ML Status] No ML project found.
+
+Run /ctx:experiment new "<hypothesis>" to start.
+```
+Exit.
+
+## Step 2: Parse Filter Argument
+
+| Arg | Show |
+|-----|------|
+| none | Full dashboard |
+| `models` | Model registry section only |
+| `exp` | Experiment log section only |
+| `drift` | Drift alerts section only |
+
+## Step 3: Read Files
+
+Read the following files (skip silently if missing):
+
+| File | Purpose |
+|------|---------|
+| `.ctx/ml/ML-STATUS.md` | Active experiment, current focus, blockers |
+| `.ctx/ml/EXPERIMENT-LOG.md` | Experiment history table |
+| `.ctx/ml/models/registry.yaml` | Model versions and metrics |
+| `.ctx/ml/features/feature-registry.yaml` | Feature count and inventory |
+| `.ctx/ml/experiments/*/artifacts/drift_alerts.json` | Any saved drift alerts |
+
+## Step 4: Render Dashboard
+
+### Full Dashboard Output
+
+```
+[ML Status] {project_dir}
+Updated: {ML-STATUS.md updated date}
+
+━━━ Active Experiment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+  {experiment_id} — {hypothesis title}
+  Status: {draft | running | concluded}
+  Phase:  {hypothesize | design | train | review | done}
+
+  Current Focus:
+    {from ML-STATUS.md Current Focus section}
+
+━━━ Recent Experiments ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+  {last 5 rows from EXPERIMENT-LOG.md, formatted as table}
+
+  ID        Status        Primary Metric    Result
+  EXP-{n}   running       AUC >= 0.90       —
+  EXP-{n-1} accepted      AUC >= 0.88       AUC 0.91
+  EXP-{n-2} rejected      AUC >= 0.88       AUC 0.86
+  EXP-{n-3} accepted      AUC >= 0.85       AUC 0.87
+
+━━━ Production Models ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+  {for each model in registry.yaml}
+  {model_name}: v{current} — {primary metric}: {value}
+    Promoted: {date}   Experiment: {experiment_id}
+
+━━━ Features ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+  {count} features registered
+  {count} features validated
+  {list feature names used by production models, comma-separated}
+
+━━━ Drift Alerts ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+  {if no drift_alerts.json found}
+  No drift alerts.
+
+  {if drift alerts found}
+  {count} feature(s) drifted in latest check:
+    {feature}: KS={ks_stat}, p={pvalue} [{severity}]
+
+━━━ Blockers ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+  {from ML-STATUS.md Blocking Issues section, or "none"}
+
+━━━ Next Steps ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+  {context-aware suggestions:}
+
+  If active experiment is draft:
+    Run /ctx:train to start training.
+
+  If active experiment is running:
+    Training in progress. Run /ctx:train again to check.
+
+  If active experiment is concluded (accepted):
+    Run /ctx:experiment new "<next hypothesis>" to iterate.
+
+  If active experiment is concluded (rejected):
+    Run /ctx:experiment new "<revised hypothesis>" based on learnings.
+
+  If no active experiment:
+    Run /ctx:experiment new "<hypothesis>" to start.
+
+  If drift alerts present:
+    Run /ctx:experiment new "retrain on updated data distribution" to address drift.
+```
+
+### Models-Only Output
+
+```
+[ML Status] Production Models
+
+  {model_name}
+    Current: v{n}
+    {primary metric}: {value}
+    Promoted: {date}
+    Experiment: {experiment_id}
+    Promotion criteria: {from registry.yaml}
+
+    Version history:
+      v{n}:   {primary metric}: {value} — production
+      v{n-1}: {primary metric}: {value} — retired
+      v{n-2}: {primary metric}: {value} — retired
+```
+
+### Experiments-Only Output
+
+```
+[ML Status] Experiment Log
+
+  ID        Hypothesis (truncated 60 chars)      Model         Metric    Result    Status
+  ─────────────────────────────────────────────────────────────────────────────────────────
+  EXP-{n}   {hypothesis}                         {model}       {metric}  {result}  {status}
+  ...
+
+Total: {count} experiments ({accepted} accepted, {rejected} rejected, {running} running)
+```
+
+### Drift-Only Output
+
+```
+[ML Status] Drift Alerts
+
+  Source: .ctx/ml/experiments/{latest_exp}/artifacts/drift_alerts.json
+  Checked: {file modified date}
+
+  {if no alerts}
+  No drift detected.
+
+  {if alerts}
+  Feature           KS Stat   p-value    Severity
+  ──────────────────────────────────────────────
+  {feature}         {stat}    {pvalue}   {high|medium}
+
+  Recommendation: Run /ctx:experiment new "retrain on updated distribution"
+```
+
+## Step 5: Handle Missing Files Gracefully
+
+If a file is missing, show that section as empty with a hint:
+
+```
+━━━ Production Models ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+  No models registered yet.
+  Models appear here after a /ctx:train run passes review.
+```
+
+Never error on missing files — display what is available.
+
+</process>
+
+<guardrails>
+- This command is read-only. It never writes files, spawns agents, or modifies state.
+- Do not parse YAML strictly — if registry.yaml is malformed, show a warning and continue.
+- Drift alert files are found by globbing .ctx/ml/experiments/*/artifacts/drift_alerts.json — show the most recently modified one.
+- Truncate long hypothesis strings to 60 characters in table views.
+- If ML-STATUS.md does not exist but EXPERIMENT-LOG.md does, derive status from the log.
+</guardrails>

package/commands/monitor.md

@@ -4,7 +4,7 @@ description: Self-healing deployments - connect to error tracking (Sentry/LogRoc
 ---
 
 <objective>
-CTX 3.3 Self-Healing Deployments - Monitor production errors and automatically create fix stories or even auto-fix with PR creation.
+CTX 3.5 Self-Healing Deployments - Monitor production errors and automatically create fix stories or even auto-fix with PR creation.
 </objective>
 
 <usage>

package/commands/train.md

@@ -0,0 +1,266 @@
+---
+name: ctx:train
+description: ML model training workflow — feature engineering, training, HPO, evaluation, and registry promotion. Uses Digital Twin patterns from ctx-ml-pipeline skill.
+args: experiment_id (optional — defaults to active experiment from ML-STATUS.md)
+---
+
+<objective>
+Orchestrate a full ML training run for a designed experiment. Loads config, spawns engineer to build/run the pipeline, spawns reviewer to validate results, and promotes to model registry if promotion criteria are met.
+
+This command assumes the experiment already has HYPOTHESIS.md, DESIGN.md, and config.yaml. Run /ctx:experiment new first if not.
+</objective>
+
+<usage>
+```bash
+/ctx:train              # Train for active experiment (from ML-STATUS.md)
+/ctx:train EXP-003      # Train for specific experiment
+/ctx:train --dry-run    # Validate config without running training
+```
+</usage>
+
+<process>
+
+## Step 1: Parse Arguments
+
+- No args → read active experiment from `.ctx/ml/ML-STATUS.md`
+- `EXP-{n}` → use that experiment ID
+- `--dry-run` → validate config and design, report issues, do not train
+
+If no active experiment and no ID given:
+```
+[Train] No active experiment found.
+Run /ctx:experiment new "<hypothesis>" to create one.
+```
+
+## Step 2: Validate Prerequisites
+
+Check these files exist before proceeding:
+
+```bash
+.ctx/ml/experiments/{id}/HYPOTHESIS.md    → must exist
+.ctx/ml/experiments/{id}/DESIGN.md        → must exist
+.ctx/ml/experiments/{id}/config.yaml      → must exist
+```
+
+If any are missing:
+```
+[Train] Cannot run — missing required files for {experiment_id}:
+  - {missing file}
+
+Run /ctx:experiment new to create them.
+```
+
+Also check:
+- `config.yaml` has a `data.train` path and it exists on disk
+- `config.yaml` has a `model.type` set
+- `config.yaml` has a `seed` set (reject if missing — non-reproducible)
+
+## Step 3: Dry Run (if --dry-run)
+
+Read DESIGN.md acceptance criteria and config.yaml. Report:
+
+```
+[Train] Dry run for {experiment_id}
+
+Config:
+  Model: {model.type}
+  Params: {key params}
+  Data: {data paths}
+  Seed: {seed}
+
+Design checks:
+  Primary metric: {metric} — {target}
+  Guard rails: {metrics}
+  Acceptance criteria: {count} items
+
+Issues: {none | list any problems}
+
+Ready to train: {yes | no}
+```
+
+Exit after dry run. Do not spawn training agent.
+
+## Step 4: Update Experiment Status
+
+Update `.ctx/ml/EXPERIMENT-LOG.md` — change row status from `draft` to `running`.
+Update `.ctx/ml/ML-STATUS.md` — set active experiment and status to running.
+
+## Step 5: Spawn ctx-ml-engineer for Training
+
+```
+Agent({
+  subagent_type: "ctx-ml-engineer",
+  prompt: |
+    Run ML training pipeline for {experiment_id}.
+
+    Read these files:
+    - .ctx/ml/experiments/{experiment_id}/config.yaml
+    - .ctx/ml/experiments/{experiment_id}/DESIGN.md
+    - .ctx/ml/features/feature-registry.yaml
+
+    Execute the full pipeline:
+
+    1. DATA VALIDATION
+       - Load data from config.yaml data paths
+       - Apply Pandera schema validation (fail hard on violations)
+       - Log row counts and class distribution
+
+    2. FEATURE PIPELINE
+       - Build transform pipeline from feature-registry.yaml
+       - Fit on training data only
+       - Transform train, val, test
+       - Save fitted pipeline to artifacts/pipeline.pkl
+
+    3. TRAINING
+       - Train model from config.yaml model params
+       - Use early stopping (patience=20)
+       - Log training curve to artifacts/train.log
+
+    4. HPO (if config.yaml has hpo: true)
+       - Run Optuna with n_trials from config or default 100
+       - Save study to artifacts/hpo_study.pkl
+       - Retrain with best params
+
+    5. EVALUATION
+       - Compute primary metric and all guard rail metrics
+       - Compute calibration error
+       - Generate ROC curve, calibration curve, feature importance plots
+       - Save metrics to artifacts/metrics.json
+       - Save plots to artifacts/plots/
+
+    6. CONFORMAL WRAPPER
+       - Fit MAPIE on calibration split at alpha=0.1
+       - Save to artifacts/mapie.pkl
+
+    7. INFERENCE SMOKE TEST
+       - Load model + pipeline + mapie
+       - Run 5 predictions with full envelope
+       - Verify envelope structure is correct
+
+    Write artifacts/metrics.json with all metrics.
+    Write all artifacts per ctx-ml-pipeline skill reproducibility requirements.
+
+    Do NOT write RESULTS.md — the reviewer will do that.
+    Do NOT update the model registry — that happens after review passes.
+})
+```
+
+## Step 6: Validate Training Artifacts
+
+After engineer completes, verify these files exist:
+
+```
+.ctx/ml/experiments/{id}/artifacts/
+├── model.pkl         → required
+├── pipeline.pkl      → required
+├── mapie.pkl         → required
+├── config.yaml       → required (copy of run config)
+├── metrics.json      → required
+├── train.log         → required
+└── plots/            → required (at least roc_curve.png)
+```
+
+If any required artifact is missing, report which and do not proceed to review.
+
+## Step 7: Spawn ctx-ml-reviewer for Evaluation
+
+```
+Agent({
+  subagent_type: "ctx-ml-reviewer",
+  prompt: |
+    Review training results for {experiment_id}.
+
+    Read:
+    - .ctx/ml/experiments/{experiment_id}/HYPOTHESIS.md
+    - .ctx/ml/experiments/{experiment_id}/DESIGN.md
+    - .ctx/ml/experiments/{experiment_id}/artifacts/metrics.json
+    - .ctx/ml/experiments/{experiment_id}/artifacts/train.log
+    - .ctx/ml/models/registry.yaml (current production model metrics for comparison)
+
+    Review checklist:
+    - [ ] Primary metric meets DESIGN.md acceptance threshold
+    - [ ] Guard rail metrics not violated
+    - [ ] Training loss converged (check train.log — no NaN, no divergence)
+    - [ ] Calibration error < 0.05
+    - [ ] Primary metric improves on production model by promotion criteria
+
+    Write .ctx/ml/experiments/{experiment_id}/RESULTS.md with:
+    - Metrics table (baseline vs result vs delta)
+    - Verdict: accepted | rejected | inconclusive
+    - Key findings
+    - Next experiment recommendation
+
+    Return verdict as final line: VERDICT: accepted | rejected | inconclusive
+})
+```
+
+## Step 8: Handle Verdict
+
+Read reviewer verdict.
+
+### Verdict: accepted
+
+1. Determine next version: read current version from `models/registry.yaml`, increment.
+2. Update `models/registry.yaml`:
+   - Add new version entry with metrics, experiment ID, date, status: production
+   - Set previous production version status to: retired
+   - Set `current` to new version
+3. Update `EXPERIMENT-LOG.md` row status to: `accepted`
+4. Update `ML-STATUS.md` with outcome
+
+Output:
+```
+[Train] EXP-{n} accepted — model promoted to {name} v{version}
+
+Metrics:
+  {primary}: {value} (was {baseline}, +{delta})
+  {guard}: {value} (was {baseline})
+
+Artifacts: .ctx/ml/experiments/{experiment_id}/artifacts/
+Registry:  .ctx/ml/models/registry.yaml
+
+Run /ctx:experiment new "<next hypothesis>" to continue.
+```
+
+### Verdict: rejected
+
+1. Update `EXPERIMENT-LOG.md` row status to: `rejected`
+2. Update `ML-STATUS.md` with outcome and reviewer's next recommendation
+
+Output:
+```
+[Train] EXP-{n} rejected — {reason from RESULTS.md}
+
+Primary metric: {value} (target was {target})
+
+Key findings:
+  {findings from RESULTS.md}
+
+Next recommendation: {from reviewer}
+
+Run /ctx:experiment new "<next hypothesis>" to iterate.
+```
+
+### Verdict: inconclusive
+
+1. Update `EXPERIMENT-LOG.md` row status to: `inconclusive`
+2. Report what is blocking a clear verdict
+
+Output:
+```
+[Train] EXP-{n} inconclusive — {reason}
+
+Blocking issue: {issue}
+
+Recommended action: {action}
+```
+
+</process>
+
+<guardrails>
+- Never promote to registry without an accepted verdict from ctx-ml-reviewer.
+- Never proceed to review if required training artifacts are missing.
+- Seed is mandatory in config.yaml — non-reproducible runs are rejected.
+- Dry run never touches EXPERIMENT-LOG.md or ML-STATUS.md.
+- If training fails mid-run, update EXPERIMENT-LOG.md status to "failed" before exiting.
+</guardrails>