claude-turing 2.2.0 → 2.3.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "2.2.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 30 commands, 2 specialized agents, experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "2.3.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 33 commands, 2 specialized agents, deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -341,6 +341,9 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:report` | Generate research report |
 | `/turing:poster` | Generate research poster |
 | `/turing:preflight` | Pre-release validation checks |
+| `/turing:diff <a> <b>` | Deep experiment comparison — config diffs, metric significance, per-class regressions, curve divergence |
+| `/turing:watch [--analyze]` | Live training monitor — loss spikes, NaN detection, overfitting, plateau alerts |
+| `/turing:regress [--tolerance]` | Performance regression gate — verify metrics haven't degraded after changes |
 
 And for fully hands-off operation:
 
@@ -525,11 +528,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-30 commands, 2 agents, 9 config files, 49 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), 727 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+33 commands, 2 agents, 10 config files, 52 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              29 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration)
+├── commands/              32 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/diff.md

@@ -0,0 +1,48 @@
+---
+name: diff
+description: Deep experiment comparison — config diffs, metric significance, per-class regressions, training curve divergence, feature importance shifts.
+disable-model-invocation: true
+argument-hint: "<exp-a> <exp-b> [--code]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Deep diagnostic comparison of two experiments. Goes beyond "which metric is higher" to show where, when, and why two experiments diverge.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First two arguments are experiment IDs (required), e.g. `exp-042 exp-053`
+   - `--code` includes git diff of train.py between the two experiments' commits
+   - `--json` outputs raw JSON instead of markdown
+
+3. **Run deep comparison:**
+   ```bash
+   python scripts/experiment_diff.py $ARGUMENTS
+   ```
+
+4. **Report results — the diff includes:**
+   - **Config diff:** which hyperparameters changed, with magnitude (e.g., `max_depth: 6 → 8 (+33%)`)
+   - **Metric diff:** all metrics with deltas and statistical significance (if seed studies exist)
+   - **Per-class diff:** which classes improved/regressed — flags regressions hidden by aggregate improvement
+   - **Training curve divergence:** the epoch where the two experiments' loss/metric curves separate
+   - **Feature importance shifts:** which features gained/lost importance
+   - **Code diff (--code):** git diff of train.py between the two commits
+
+5. **Saved output:** report written to `experiments/diffs/<exp-a>-vs-<exp-b>.yaml`
+
+6. **If experiment ID not found:** list available experiment IDs from `experiments/log.jsonl`
+
+7. **If no training pipeline exists:** suggest `/turing:init` first.
+
+## Examples
+
+```
+/turing:diff exp-042 exp-053                # Full diagnostic comparison
+/turing:diff exp-042 exp-053 --code         # Include train.py code changes
+/turing:diff exp-001 exp-010 --json         # Raw JSON output
+```

package/commands/regress.md

@@ -0,0 +1,53 @@
+---
+name: regress
+description: Performance regression gate — re-run best experiment after code/dependency changes and verify metrics haven't degraded.
+disable-model-invocation: true
+argument-hint: "[--tolerance 0.01] [--against exp-id] [--quick]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+CI for your model. After any change to code, dependencies, or data, verify metrics haven't silently regressed.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--tolerance 0.01` sets the relative tolerance (default 1%)
+   - `--against exp-042` checks against a specific experiment (default: best)
+   - `--quick` runs 1 seed instead of 3 for fast checks
+   - `--runs 5` sets number of regression runs (default 3)
+   - `--json` outputs raw JSON
+
+3. **Run regression gate:**
+   ```bash
+   python scripts/regression_gate.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **PASS:** all metrics within tolerance — no regression
+   - **WARNING:** some metrics degraded within 2x tolerance — investigate
+   - **FAIL:** REGRESSION DETECTED — at least one metric degraded beyond tolerance
+   - Shows per-metric comparison with deltas and relative differences
+   - Shows environment diff if library versions changed (may explain regression)
+
+5. **Saved output:** report written to `experiments/regressions/check-YYYY-MM-DD.yaml`
+
+6. **If no experiments exist:** suggest running `/turing:train` first.
+
+7. **On FAIL verdict:** suggest investigating with:
+   - `/turing:diff <baseline> <latest>` to see what changed
+   - `pip freeze` comparison to identify library version changes
+   - `git diff` to review code changes
+
+## Examples
+
+```
+/turing:regress                              # Default: check best, 1% tolerance, 3 runs
+/turing:regress --quick                      # Fast check: 1 run
+/turing:regress --against exp-042            # Check specific experiment
+/turing:regress --tolerance 0.005 --runs 5   # Strict: 0.5% tolerance, 5 runs
+```

package/commands/turing.md

@@ -39,6 +39,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "fork", "branch", "try both", "parallel experiments", "A or B" | `/turing:fork` | Orchestrate |
 | "profile", "profiling", "bottleneck", "slow training", "why is it slow", "timing" | `/turing:profile` | Check |
 | "checkpoint", "checkpoints", "prune checkpoints", "disk space", "resume training" | `/turing:checkpoint` | Check |
+| "diff", "deep compare", "what changed", "why did it diverge", "experiment diff" | `/turing:diff` | Analyze |
+| "watch", "monitor", "live training", "loss spike", "is it overfitting", "training progress" | `/turing:watch` | Monitor |
+| "regress", "regression", "did metrics degrade", "check for regression", "CI gate", "stability check" | `/turing:regress` | Validate |
 
 ## Sub-commands
 
@@ -74,6 +77,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:fork <exp-id> --branches` | Experiment branching: run parallel tracks, report winner | (inline) |
 | `/turing:profile [exp-id]` | Computational profiling: timing, memory, throughput, bottleneck detection | (inline) |
 | `/turing:checkpoint <action>` | Smart checkpoint management: list, prune (Pareto), average, resume, stats | (inline) |
+| `/turing:diff <exp-a> <exp-b>` | Deep experiment comparison: config diff, metric significance, per-class regressions, curve divergence | (inline) |
+| `/turing:watch [--analyze]` | Live training monitor with early-warning alerts (loss spike, NaN, overfitting, plateau) | (inline) |
+| `/turing:regress [--tolerance]` | Performance regression gate: re-run best experiment, verify metrics haven't degraded | (inline) |
 
 ## Proactive Detection
 

package/commands/watch.md

@@ -0,0 +1,60 @@
+---
+name: watch
+description: Live training monitor with early-warning alerts for loss spikes, NaN, overfitting, and metric plateaus.
+disable-model-invocation: true
+argument-hint: "[--alerts] [--interval 10] [--analyze run.log]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Stream metrics during training with early-warning alerts. Catches problems mid-run instead of at the end.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--analyze run.log` — post-hoc analysis of a completed log (non-blocking)
+   - `--alerts` — show only alert lines, suppress normal output
+   - `--interval 10` — check interval in seconds (default: 10)
+   - `--alerts-config config/watch_alerts.yaml` — custom alert rules
+   - `--json` — raw JSON output (for `--analyze` mode)
+
+3. **For post-hoc analysis:**
+   ```bash
+   python scripts/training_monitor.py --analyze run.log
+   ```
+
+4. **For live monitoring (inform user):**
+   Live monitoring requires a running training process. Suggest the user run in a separate terminal:
+   ```bash
+   python scripts/training_monitor.py --log run.log --interval 10
+   ```
+
+5. **Alert types:**
+   - **Loss spike:** loss > 3x rolling mean (configurable multiplier)
+   - **NaN detected:** any metric is NaN — CRITICAL, suggests pausing
+   - **Overfitting onset:** train/val gap widening for 3+ consecutive epochs
+   - **Plateau:** metric improvement < 0.001 for 5+ consecutive epochs
+
+6. **Dashboard line format:**
+   ```
+   Epoch 23/100 | loss: 0.342 ↓ | acc: 0.865 ↑ | gap: 0.018 | ⚠ plateau
+   ```
+
+7. **Alert config:** rules are in `config/watch_alerts.yaml` — users can customize thresholds.
+
+8. **Saved output:** analysis report written to `experiments/monitors/analysis-*.yaml`
+
+9. **If no training log exists:** suggest running `/turing:train` first.
+
+## Examples
+
+```
+/turing:watch --analyze run.log           # Analyze completed training
+/turing:watch --analyze run.log --json    # JSON output for scripting
+/turing:watch --alerts                    # Live: show only alerts
+/turing:watch --interval 5               # Live: check every 5 seconds
+```

package/config/watch_alerts.yaml

@@ -0,0 +1,36 @@
+# Training monitor alert rules for /turing:watch
+#
+# Each alert has:
+#   condition:   alert type (loss_spike, nan_detected, overfitting, plateau)
+#   severity:    info | warning | critical
+#   action:      optional action on trigger (e.g., "pause")
+#   message:     alert message template with {epoch}, {value}, {mean}, etc.
+#
+# Customize thresholds to match your training dynamics.
+
+alerts:
+  loss_spike:
+    condition: loss_spike
+    multiplier: 3.0            # Trigger if loss > N * rolling_mean
+    severity: warning
+    message: "Loss spike at epoch {epoch}: {value} vs rolling mean {mean:.4f}"
+
+  nan_detected:
+    condition: nan_detected
+    severity: critical
+    action: pause              # Suggest pausing training on NaN
+    message: "NaN detected in {metric} at epoch {epoch}"
+
+  overfitting_onset:
+    condition: overfitting
+    gap_ratio: 0.5             # train_loss / val_loss ratio threshold
+    consecutive: 3             # N consecutive epochs of widening gap
+    severity: warning
+    message: "Overfitting detected — train/val gap widening since epoch {onset}"
+
+  plateau:
+    condition: plateau
+    min_improvement: 0.001     # Minimum metric change per epoch
+    consecutive: 5             # N consecutive flat epochs
+    severity: info
+    message: "Metric plateaued — consider early stopping or learning rate reduction"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "2.2.0",
+  "version": "2.3.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -25,6 +25,7 @@ const SUB_COMMANDS = [
   "report", "mode", "preflight", "card", "seed", "reproduce",
   "diagnose", "ablate", "frontier", "profile", "checkpoint", "export",
   "lit", "paper", "queue", "retry", "fork",
+  "diff", "watch", "regress",
 ];
 
 export async function install(opts = {}) {
@@ -79,6 +80,7 @@ export async function install(opts = {}) {
     "experiment_archetypes.yaml", "novelty_aliases.yaml",
     "relationships.toml", "state.toml", "task_taxonomy.yaml",
     "failure_modes.yaml",
+    "watch_alerts.yaml",
   ];
   for (const file of CONFIG_FILES) {
     await copyFile(

package/src/verify.js

@@ -44,6 +44,9 @@ const EXPECTED_COMMANDS = [
   "queue/SKILL.md",
   "retry/SKILL.md",
   "fork/SKILL.md",
+  "diff/SKILL.md",
+  "watch/SKILL.md",
+  "regress/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];
@@ -53,6 +56,7 @@ const EXPECTED_CONFIG = [
   "experiment_archetypes.yaml", "novelty_aliases.yaml",
   "relationships.toml", "state.toml", "task_taxonomy.yaml",
   "failure_modes.yaml",
+  "watch_alerts.yaml",
 ];
 
 async function fileExists(path) {