claude-turing 1.3.0 → 1.5.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "1.3.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 19 commands, 2 specialized agents, statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "1.5.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 24 commands, 2 specialized agents, performance profiling, smart Pareto-based checkpoint management, experiment intelligence (error analysis, ablation studies, Pareto frontiers), statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -325,6 +325,11 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:validate [--auto]` | Check metric stability — auto-configure multi-run if noisy |
 | `/turing:seed [N] [--quick]` | Multi-seed study — mean/std/CI, flag seed-sensitive results |
 | `/turing:reproduce <exp-id>` | Reproducibility verification — re-run and check tolerance |
+| `/turing:diagnose [exp-id]` | Error analysis — failure modes, confused pairs, feature-range bias |
+| `/turing:ablate [--components]` | Ablation study — remove components, measure impact, flag dead weight |
+| `/turing:frontier [--metrics]` | Pareto frontier — multi-objective tradeoff visualization |
+| `/turing:profile [exp-id]` | Computational profiling — timing, memory, throughput, bottleneck detection |
+| `/turing:checkpoint <action>` | Smart checkpoint management — list, prune (Pareto), average, resume, stats |
 | `/turing:card` | Generate a model card — performance, limitations, intended use, artifact contract |
 | `/turing:logbook` | Generate HTML experiment logbook |
 | `/turing:report` | Generate research report |
@@ -514,11 +519,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-19 commands, 2 agents, 8 config files, 34 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor (seed studies + reproducibility), 407 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+24 commands, 2 agents, 8 config files, 39 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling + smart checkpoints, 542 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              18 skill files (core + taste-leverage + reporting + exploration + statistical rigor)
+├── commands/              23 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/ablate.md

@@ -0,0 +1,47 @@
+---
+name: ablate
+description: Run systematic ablation study — remove components one at a time, measure impact, produce publication-ready table with dead-weight flagging.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--components \"X,Y\"] [--seeds 3] [--latex]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Run a systematic ablation study to measure the contribution of each model component.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--components "dropout,feature_X,regularization"` specifies components to ablate
+   - `--seeds 3` runs each ablation 3 times for statistical robustness (uses seed runner)
+   - `--latex` outputs a LaTeX-formatted table instead of markdown
+
+3. **Run ablation study:**
+   ```bash
+   python scripts/ablation_study.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - Show the ablation table: Configuration | Metric | Δ from Full | % Change
+   - Rank by impact (largest Δ first)
+   - Flag **dead-weight** components (removing them improves the metric)
+   - If `--latex`, output ready for copy-paste into a paper
+
+5. **Saved output:** results written to `experiments/ablations/exp-NNN-ablation.yaml`
+
+6. **If no ablatable components detected:** suggest using `--components` explicitly.
+
+## Examples
+
+```
+/turing:ablate                                    # Auto-detect components
+/turing:ablate exp-042                            # Specific experiment
+/turing:ablate --components "dropout,subsample"   # Specific components
+/turing:ablate --seeds 3                          # Multi-seed for robustness
+/turing:ablate --latex                            # LaTeX table output
+```

package/commands/checkpoint.md

@@ -0,0 +1,47 @@
+---
+name: checkpoint
+description: Smart checkpoint management — list, prune (Pareto-based), average top-K, resume from any point, disk usage stats.
+disable-model-invocation: true
+argument-hint: "<list|prune|average|resume|stats> [exp-id] [--top 3] [--dry-run]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Manage model checkpoints intelligently using Pareto dominance.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First word is the action: `list`, `prune`, `average`, `resume`, `stats`
+   - `resume` requires an experiment ID as second argument
+   - `--top 3` sets the number of checkpoints for averaging
+   - `--dry-run` previews pruning without deleting
+
+3. **Run checkpoint manager:**
+   ```bash
+   python scripts/checkpoint_manager.py $ARGUMENTS
+   ```
+
+4. **Report results by action:**
+   - **list:** Table of all checkpoints with metrics, size, and Pareto status
+   - **prune:** Removes dominated checkpoints, reports space saved
+   - **average:** Lists top-K checkpoints for weight averaging
+   - **resume:** Locates checkpoint for a specific experiment
+   - **stats:** Disk usage summary by total, average, and model type
+
+5. **Saved output:** report written to `experiments/checkpoints/checkpoint-report.yaml`
+
+## Examples
+
+```
+/turing:checkpoint list              # Show all checkpoints
+/turing:checkpoint stats             # Disk usage summary
+/turing:checkpoint prune --dry-run   # Preview what would be pruned
+/turing:checkpoint prune             # Remove dominated checkpoints
+/turing:checkpoint average --top 5   # Top 5 for averaging
+/turing:checkpoint resume exp-042    # Resume from checkpoint
+```

package/commands/diagnose.md

@@ -0,0 +1,52 @@
+---
+name: diagnose
+description: Error analysis — cluster failure cases, identify systematic failure modes, and suggest targeted fixes with auto-queued hypotheses.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--auto-queue] [--top 5]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Analyze where and why the model fails, beyond aggregate metrics.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Generate predictions if needed:**
+   Check if `experiments/predictions/exp-NNN-preds.yaml` exists. If not, run:
+   ```bash
+   python train.py --predict-only --output experiments/predictions/
+   ```
+   The predictions file must contain `y_true`, `y_pred`, `task_type`, and optionally `features`.
+
+3. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--auto-queue` auto-queues hypotheses from failure modes into `hypotheses.yaml`
+   - `--top 5` limits to top N failure modes (default 5)
+
+4. **Run error analysis:**
+   ```bash
+   python scripts/diagnose_errors.py $ARGUMENTS
+   ```
+
+5. **Report results:**
+   - **Classification:** confusion matrix, most-confused pairs, per-class P/R/F1, low-recall classes
+   - **Regression:** residual stats, P90/P95 errors, feature-range bias, systematic bias
+   - **Failure modes:** ranked by impact, with suggested fixes
+   - **Auto-hypotheses:** if `--auto-queue`, shows queued hypotheses targeting weaknesses
+
+6. **Saved output:** report written to `experiments/diagnoses/exp-NNN-diagnosis.yaml`
+
+7. **If no predictions file exists:** instruct user to run the model on validation set first.
+
+## Examples
+
+```
+/turing:diagnose                    # Analyze best experiment
+/turing:diagnose exp-042            # Specific experiment
+/turing:diagnose --auto-queue       # Queue fix hypotheses
+/turing:diagnose --top 10           # Top 10 failure modes
+```

package/commands/frontier.md

@@ -0,0 +1,45 @@
+---
+name: frontier
+description: Visualize Pareto frontier across multiple objectives — answers "which model is actually best?" when there are tradeoffs.
+disable-model-invocation: true
+argument-hint: "[--metrics \"accuracy,train_seconds,n_params\"] [--ascii]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Visualize the Pareto frontier across multiple objectives from experiment history.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--metrics "accuracy,train_seconds,n_params"` specifies metrics to analyze
+   - Without `--metrics`, uses primary metric + train_seconds from config
+   - `--ascii` generates an ASCII scatter plot (2D projection)
+
+3. **Run Pareto analysis:**
+   ```bash
+   python scripts/pareto_frontier.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **Pareto-optimal experiments:** table with all metrics and what each is best at
+   - **Dominated experiments:** with their nearest Pareto neighbor
+   - **ASCII scatter plot** (if `--ascii`): 2D projection with * for Pareto, · for dominated
+   - Summary: "N Pareto-optimal of M experiments across K metrics"
+
+5. **Saved output:** results written to `experiments/frontiers/frontier-YYYY-MM-DD.yaml`
+
+6. **If no experiments have all requested metrics:** suggest which metrics are available.
+
+## Examples
+
+```
+/turing:frontier                                              # Default: metric vs time
+/turing:frontier --metrics "accuracy,train_seconds"           # 2D frontier
+/turing:frontier --metrics "accuracy,train_seconds,n_params"  # 3D frontier
+/turing:frontier --ascii                                      # With scatter plot
+```

package/commands/profile.md

@@ -0,0 +1,43 @@
+---
+name: profile
+description: Profile a training run — timing breakdown, memory usage, throughput, bottleneck detection with actionable recommendations.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--seed 42]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Profile a training run to identify performance bottlenecks.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--seed 42` sets the random seed for the profiling run
+
+3. **Run profiling:**
+   ```bash
+   python scripts/profile_training.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **Timing:** total time, training time, overhead breakdown
+   - **Memory:** peak RSS, Python peak, GPU peak (if applicable)
+   - **Throughput:** samples/sec
+   - **Bottleneck:** identified bottleneck type and severity
+   - **Recommendations:** actionable fixes for the detected bottleneck
+
+5. **Saved output:** results written to `experiments/profiles/exp-NNN-profile.yaml`
+
+6. **If no training pipeline exists:** suggest `/turing:init` first.
+
+## Examples
+
+```
+/turing:profile              # Profile best experiment config
+/turing:profile exp-042      # Profile specific experiment
+```

package/commands/turing.md

@@ -28,6 +28,11 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "mode", "explore", "exploit", "replicate", "strategy" | `/turing:mode` | Strategy |
 | "preflight", "resources", "VRAM", "memory", "can I run", "OOM", "GPU" | `/turing:preflight` | Check |
 | "card", "model card", "document model", "model documentation" | `/turing:card` | Document |
+| "diagnose", "error analysis", "failure modes", "where does it fail", "confusion matrix" | `/turing:diagnose` | Analyze |
+| "ablate", "ablation", "remove component", "which features matter", "component impact" | `/turing:ablate` | Analyze |
+| "frontier", "pareto", "tradeoff", "tradeoffs", "multi-objective", "which model is best" | `/turing:frontier` | Analyze |
+| "profile", "profiling", "bottleneck", "slow training", "why is it slow", "timing" | `/turing:profile` | Check |
+| "checkpoint", "checkpoints", "prune checkpoints", "disk space", "resume training" | `/turing:checkpoint` | Check |
 
 ## Sub-commands
 
@@ -52,6 +57,11 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:mode <mode>` | Set research strategy (explore/exploit/replicate) | (inline) |
 | `/turing:preflight` | Pre-flight resource check (VRAM/RAM/disk) | (inline) |
 | `/turing:card` | Generate standardized model card (type, performance, data, limitations, contract) | (inline) |
+| `/turing:diagnose [exp-id]` | Error analysis: failure modes, confused pairs, feature-range bias | (inline) |
+| `/turing:ablate [--components]` | Ablation study: remove components, measure impact, flag dead weight | (inline) |
+| `/turing:frontier [--metrics]` | Pareto frontier: multi-objective tradeoff visualization | (inline) |
+| `/turing:profile [exp-id]` | Computational profiling: timing, memory, throughput, bottleneck detection | (inline) |
+| `/turing:checkpoint <action>` | Smart checkpoint management: list, prune (Pareto), average, resume, stats | (inline) |
 
 ## Proactive Detection
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "1.3.0",
+  "version": "1.5.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -23,6 +23,7 @@ const SUB_COMMANDS = [
   "init", "train", "status", "compare", "sweep", "validate",
   "try", "brief", "suggest", "explore", "design", "logbook", "poster",
   "report", "mode", "preflight", "card", "seed", "reproduce",
+  "diagnose", "ablate", "frontier", "profile", "checkpoint",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -33,6 +33,11 @@ const EXPECTED_COMMANDS = [
   "card/SKILL.md",
   "seed/SKILL.md",
   "reproduce/SKILL.md",
+  "diagnose/SKILL.md",
+  "ablate/SKILL.md",
+  "frontier/SKILL.md",
+  "profile/SKILL.md",
+  "checkpoint/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];