claude-turing 3.1.0 → 3.3.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "3.1.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 44 commands, 2 specialized agents, pre-training intelligence (sanity checks + baseline generation + leakage detection), meta-intelligence (cross-project knowledge transfer + methodology audit), scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "3.3.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 49 commands, 2 specialized agents, feature & training intelligence (feature selection + curriculum optimization), model debugging (xray + sensitivity + calibration), pre-training intelligence (sanity checks + baseline generation + leakage detection), meta-intelligence (cross-project knowledge transfer + methodology audit), scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -355,6 +355,11 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:sanity [--quick]` | Pre-training sanity checks — initial loss, single-batch overfit, gradient flow, output validation |
 | `/turing:baseline [--methods]` | Automatic baseline generation — random, majority/mean, linear, k-NN |
 | `/turing:leak [--deep]` | Targeted leakage detection — single-feature tests, correlation, train/test overlap |
+| `/turing:xray [exp-id]` | Internal model diagnostics — gradient flow, dead neurons, weight distributions, tree analysis |
+| `/turing:sensitivity [exp-id]` | Hyperparameter sensitivity — rank parameters by impact, detect non-monotonic responses |
+| `/turing:calibrate [exp-id]` | Probability calibration — ECE/MCE, reliability diagrams, Platt/isotonic/temperature scaling |
+| `/turing:feature [--method]` | Automated feature selection — multi-method consensus ranking, redundancy, interactions |
+| `/turing:curriculum [exp-id]` | Training curriculum optimization — difficulty scoring, strategy comparison, mislabeled sample detection |
 
 And for fully hands-off operation:
 
@@ -539,11 +544,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-44 commands, 2 agents, 10 config files, 63 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), pre-training intelligence (sanity + baseline + leak), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+49 commands, 2 agents, 10 config files, 68 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), pre-training intelligence (sanity + baseline + leak), model debugging (xray + sensitivity + calibrate), feature & training intelligence (feature + curriculum), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              43 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency + meta-intelligence + pre-training intelligence)
+├── commands/              48 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency + meta-intelligence + pre-training intelligence + model debugging + feature & training intelligence)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/calibrate.md

@@ -0,0 +1,47 @@
+---
+name: calibrate
+description: Probability calibration — measure ECE, plot reliability diagrams, apply Platt scaling or isotonic regression.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--method platt|isotonic|temperature|auto]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Make model probabilities trustworthy. Does 80% confidence actually mean 80% correct?
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - Optional experiment ID
+   - `--method platt|isotonic|temperature|auto` — calibration method (default: auto)
+   - `--json` — raw JSON output
+
+3. **Run calibration:**
+   ```bash
+   python scripts/calibration.py $ARGUMENTS
+   ```
+
+4. **Report includes:**
+   - ECE/MCE before calibration
+   - Reliability diagram (predicted vs actual per bin)
+   - Calibration method comparison table
+   - Verdict: ALREADY CALIBRATED / IMPROVED / NO IMPROVEMENT
+
+5. **Methods:**
+   - **Platt:** logistic regression on logits
+   - **Isotonic:** non-parametric (more flexible, needs more data)
+   - **Temperature:** single scalar T parameter
+   - **Auto:** tries all, picks lowest ECE
+
+6. **Saved output:** report in `experiments/calibration/<exp-id>-calibration.yaml`
+
+## Examples
+
+```
+/turing:calibrate exp-042                  # Auto-select best method
+/turing:calibrate exp-042 --method platt   # Platt scaling only
+```

package/commands/curriculum.md

@@ -0,0 +1,43 @@
+---
+name: curriculum
+description: Training curriculum optimization — order data by difficulty, compare easy-to-hard vs hard-to-easy vs self-paced strategies.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--strategies easy-to-hard,random]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Does the order your model sees data matter? Find out systematically.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - Optional experiment ID
+   - `--strategies "easy_to_hard,hard_to_easy,self_paced,random"` — strategies to test
+   - `--json` — raw JSON output
+
+3. **Run curriculum analysis:**
+   ```bash
+   python scripts/curriculum_optimizer.py $ARGUMENTS
+   ```
+
+4. **Strategies tested:**
+   - **Random:** standard shuffling (control)
+   - **Easy-to-hard:** classic curriculum learning
+   - **Hard-to-easy:** anti-curriculum
+   - **Self-paced:** start easy, gradually include harder samples
+
+5. **Report includes:** strategy comparison table with metric, convergence epoch, and speedup vs random; impossible sample detection (likely mislabeled)
+
+6. **Saved output:** report in `experiments/curriculum/<exp-id>-curriculum.yaml`
+
+## Examples
+
+```
+/turing:curriculum exp-042                      # All strategies
+/turing:curriculum --strategies easy_to_hard,random  # Specific strategies
+```

package/commands/feature.md

@@ -0,0 +1,42 @@
+---
+name: feature
+description: Automated feature selection — multi-method importance consensus, redundancy detection, and interaction feature generation.
+disable-model-invocation: true
+argument-hint: "[--method all|importance] [--top-k 20]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Systematically evaluate which features matter and which are noise.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--method all|importance|selection|generation` — analysis type (default: all)
+   - `--top-k 20` — number of top features to consider
+   - `--json` — raw JSON output
+
+3. **Run feature analysis:**
+   ```bash
+   python scripts/feature_intelligence.py $ARGUMENTS
+   ```
+
+4. **Report includes:**
+   - Consensus ranking: features ranked by number of methods placing them in top-K
+   - Per-method ranks: mutual information, L1, tree-based
+   - Redundant pairs: features with |r| > 0.95
+   - Candidate interaction features from top consensus set
+   - Drop recommendation for zero-consensus features
+
+5. **Saved output:** report in `experiments/features/features-*.yaml`
+
+## Examples
+
+```
+/turing:feature                      # Full analysis
+/turing:feature --top-k 10           # Top-10 consensus
+```

package/commands/sensitivity.md

@@ -0,0 +1,41 @@
+---
+name: sensitivity
+description: Hyperparameter sensitivity analysis — rank parameters by impact, identify which matter and which are noise.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--params learning_rate,max_depth]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Which hyperparameters actually matter? Stop wasting time on the ones that don't.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - Optional experiment ID
+   - `--params "learning_rate,max_depth"` — specific parameters to analyze
+   - `--json` — raw JSON output
+
+3. **Run sensitivity analysis:**
+   ```bash
+   python scripts/sensitivity_analysis.py $ARGUMENTS
+   ```
+
+4. **Report includes:**
+   - Per-parameter sensitivity ranking: HIGH / MED / LOW / NONE
+   - Metric range for each parameter sweep
+   - Monotonicity detection (is there a sweet spot?)
+   - Recommendations: focus tuning on X, stop tuning Y
+
+5. **Saved output:** report in `experiments/sensitivity/<exp-id>-sensitivity.yaml`
+
+## Examples
+
+```
+/turing:sensitivity exp-042                           # All tunable params
+/turing:sensitivity --params "learning_rate,max_depth" # Specific params
+```

package/commands/turing.md

@@ -53,6 +53,11 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "sanity", "sanity check", "pre-training", "is it broken", "before training", "quick check" | `/turing:sanity` | Check |
 | "baseline", "baselines", "trivial baseline", "majority class", "is it better than random" | `/turing:baseline` | Analyze |
 | "leak", "leakage", "data leakage scan", "suspicious feature", "train test overlap" | `/turing:leak` | Validate |
+| "xray", "model internals", "dead neurons", "gradient flow", "weight distribution", "inside the model" | `/turing:xray` | Analyze |
+| "sensitivity", "which params matter", "hyperparameter importance", "parameter ranking" | `/turing:sensitivity` | Analyze |
+| "calibrate", "calibration", "ECE", "reliability diagram", "overconfident", "probability calibration" | `/turing:calibrate` | Analyze |
+| "feature", "features", "feature selection", "feature importance", "which features matter", "redundant features" | `/turing:feature` | Analyze |
+| "curriculum", "training order", "easy to hard", "data ordering", "curriculum learning" | `/turing:curriculum` | Optimize |
 
 ## Sub-commands
 
@@ -102,6 +107,11 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:sanity [--quick]` | Pre-training sanity checks: initial loss, overfit test, gradient flow, output validation | (inline) |
 | `/turing:baseline [--methods]` | Automatic baseline generation: random, majority/mean, linear, k-NN | (inline) |
 | `/turing:leak [--deep]` | Targeted leakage detection: single-feature tests, correlation, train/test overlap | (inline) |
+| `/turing:xray [exp-id]` | Internal model diagnostics: gradient flow, dead neurons, weight distributions, tree analysis | (inline) |
+| `/turing:sensitivity [exp-id]` | Hyperparameter sensitivity analysis: rank parameters by impact, detect non-monotonic responses | (inline) |
+| `/turing:calibrate [exp-id]` | Probability calibration: ECE/MCE, reliability diagrams, Platt/isotonic/temperature scaling | (inline) |
+| `/turing:feature [--method]` | Automated feature selection: multi-method consensus ranking, redundancy, interaction generation | (inline) |
+| `/turing:curriculum [exp-id]` | Training curriculum optimization: difficulty scoring, strategy comparison, impossible sample detection | (inline) |
 
 ## Proactive Detection
 

package/commands/xray.md

@@ -0,0 +1,43 @@
+---
+name: xray
+description: Internal model diagnostics — gradient flow, dead neurons, activation stats, weight distributions, tree depth analysis.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--layer encoder.layer.2] [--compare exp-a exp-b]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+See inside the model. When it underperforms, the fix depends on *why*.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - Optional experiment ID
+   - `--layer "name"` — focus on specific layer
+   - `--compare exp-a exp-b` — side-by-side diagnostics
+   - `--json` — raw JSON output
+
+3. **Run model diagnostics:**
+   ```bash
+   python scripts/model_xray.py $ARGUMENTS
+   ```
+
+4. **Diagnostics by model type:**
+   - **Neural networks:** gradient magnitudes, activation stats, dead neuron %, weight distributions, gradient-to-weight ratio
+   - **Tree models:** depth utilization, leaf purity, feature split dominance
+   - **scikit-learn:** coefficient magnitudes, feature importance concentration
+
+5. **Issues detected:** dead gradients, vanishing/exploding gradients, dead neurons, sparse weights, feature dominance, overfitting risk
+
+6. **Saved output:** report in `experiments/xrays/<exp-id>-xray.yaml`
+
+## Examples
+
+```
+/turing:xray exp-042              # Full diagnostics
+/turing:xray                      # Best experiment
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "3.1.0",
+  "version": "3.3.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -30,6 +30,8 @@ const SUB_COMMANDS = [
   "scale", "budget", "distill",
   "transfer", "audit",
   "sanity", "baseline", "leak",
+  "xray", "sensitivity", "calibrate",
+  "feature", "curriculum",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -58,6 +58,11 @@ const EXPECTED_COMMANDS = [
   "sanity/SKILL.md",
   "baseline/SKILL.md",
   "leak/SKILL.md",
+  "xray/SKILL.md",
+  "sensitivity/SKILL.md",
+  "calibrate/SKILL.md",
+  "feature/SKILL.md",
+  "curriculum/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];