claude-turing 1.3.0 → 1.4.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "1.3.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 19 commands, 2 specialized agents, statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "1.4.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 22 commands, 2 specialized agents, experiment intelligence (error analysis, ablation studies, Pareto frontiers), statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -325,6 +325,9 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:validate [--auto]` | Check metric stability — auto-configure multi-run if noisy |
 | `/turing:seed [N] [--quick]` | Multi-seed study — mean/std/CI, flag seed-sensitive results |
 | `/turing:reproduce <exp-id>` | Reproducibility verification — re-run and check tolerance |
+| `/turing:diagnose [exp-id]` | Error analysis — failure modes, confused pairs, feature-range bias |
+| `/turing:ablate [--components]` | Ablation study — remove components, measure impact, flag dead weight |
+| `/turing:frontier [--metrics]` | Pareto frontier — multi-objective tradeoff visualization |
 | `/turing:card` | Generate a model card — performance, limitations, intended use, artifact contract |
 | `/turing:logbook` | Generate HTML experiment logbook |
 | `/turing:report` | Generate research report |
@@ -514,11 +517,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-19 commands, 2 agents, 8 config files, 34 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor (seed studies + reproducibility), 407 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+22 commands, 2 agents, 8 config files, 37 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence (error analysis + ablation + Pareto frontier), 487 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              18 skill files (core + taste-leverage + reporting + exploration + statistical rigor)
+├── commands/              21 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/ablate.md

@@ -0,0 +1,47 @@
+---
+name: ablate
+description: Run systematic ablation study — remove components one at a time, measure impact, produce publication-ready table with dead-weight flagging.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--components \"X,Y\"] [--seeds 3] [--latex]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Run a systematic ablation study to measure the contribution of each model component.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--components "dropout,feature_X,regularization"` specifies components to ablate
+   - `--seeds 3` runs each ablation 3 times for statistical robustness (uses seed runner)
+   - `--latex` outputs a LaTeX-formatted table instead of markdown
+
+3. **Run ablation study:**
+   ```bash
+   python scripts/ablation_study.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - Show the ablation table: Configuration | Metric | Δ from Full | % Change
+   - Rank by impact (largest Δ first)
+   - Flag **dead-weight** components (removing them improves the metric)
+   - If `--latex`, output ready for copy-paste into a paper
+
+5. **Saved output:** results written to `experiments/ablations/exp-NNN-ablation.yaml`
+
+6. **If no ablatable components detected:** suggest using `--components` explicitly.
+
+## Examples
+
+```
+/turing:ablate                                    # Auto-detect components
+/turing:ablate exp-042                            # Specific experiment
+/turing:ablate --components "dropout,subsample"   # Specific components
+/turing:ablate --seeds 3                          # Multi-seed for robustness
+/turing:ablate --latex                            # LaTeX table output
+```

package/commands/diagnose.md

@@ -0,0 +1,52 @@
+---
+name: diagnose
+description: Error analysis — cluster failure cases, identify systematic failure modes, and suggest targeted fixes with auto-queued hypotheses.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--auto-queue] [--top 5]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Analyze where and why the model fails, beyond aggregate metrics.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Generate predictions if needed:**
+   Check if `experiments/predictions/exp-NNN-preds.yaml` exists. If not, run:
+   ```bash
+   python train.py --predict-only --output experiments/predictions/
+   ```
+   The predictions file must contain `y_true`, `y_pred`, `task_type`, and optionally `features`.
+
+3. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--auto-queue` auto-queues hypotheses from failure modes into `hypotheses.yaml`
+   - `--top 5` limits to top N failure modes (default 5)
+
+4. **Run error analysis:**
+   ```bash
+   python scripts/diagnose_errors.py $ARGUMENTS
+   ```
+
+5. **Report results:**
+   - **Classification:** confusion matrix, most-confused pairs, per-class P/R/F1, low-recall classes
+   - **Regression:** residual stats, P90/P95 errors, feature-range bias, systematic bias
+   - **Failure modes:** ranked by impact, with suggested fixes
+   - **Auto-hypotheses:** if `--auto-queue`, shows queued hypotheses targeting weaknesses
+
+6. **Saved output:** report written to `experiments/diagnoses/exp-NNN-diagnosis.yaml`
+
+7. **If no predictions file exists:** instruct user to run the model on validation set first.
+
+## Examples
+
+```
+/turing:diagnose                    # Analyze best experiment
+/turing:diagnose exp-042            # Specific experiment
+/turing:diagnose --auto-queue       # Queue fix hypotheses
+/turing:diagnose --top 10           # Top 10 failure modes
+```

package/commands/frontier.md

@@ -0,0 +1,45 @@
+---
+name: frontier
+description: Visualize Pareto frontier across multiple objectives — answers "which model is actually best?" when there are tradeoffs.
+disable-model-invocation: true
+argument-hint: "[--metrics \"accuracy,train_seconds,n_params\"] [--ascii]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Visualize the Pareto frontier across multiple objectives from experiment history.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--metrics "accuracy,train_seconds,n_params"` specifies metrics to analyze
+   - Without `--metrics`, uses primary metric + train_seconds from config
+   - `--ascii` generates an ASCII scatter plot (2D projection)
+
+3. **Run Pareto analysis:**
+   ```bash
+   python scripts/pareto_frontier.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **Pareto-optimal experiments:** table with all metrics and what each is best at
+   - **Dominated experiments:** with their nearest Pareto neighbor
+   - **ASCII scatter plot** (if `--ascii`): 2D projection with * for Pareto, · for dominated
+   - Summary: "N Pareto-optimal of M experiments across K metrics"
+
+5. **Saved output:** results written to `experiments/frontiers/frontier-YYYY-MM-DD.yaml`
+
+6. **If no experiments have all requested metrics:** suggest which metrics are available.
+
+## Examples
+
+```
+/turing:frontier                                              # Default: metric vs time
+/turing:frontier --metrics "accuracy,train_seconds"           # 2D frontier
+/turing:frontier --metrics "accuracy,train_seconds,n_params"  # 3D frontier
+/turing:frontier --ascii                                      # With scatter plot
+```

package/commands/turing.md

@@ -28,6 +28,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "mode", "explore", "exploit", "replicate", "strategy" | `/turing:mode` | Strategy |
 | "preflight", "resources", "VRAM", "memory", "can I run", "OOM", "GPU" | `/turing:preflight` | Check |
 | "card", "model card", "document model", "model documentation" | `/turing:card` | Document |
+| "diagnose", "error analysis", "failure modes", "where does it fail", "confusion matrix" | `/turing:diagnose` | Analyze |
+| "ablate", "ablation", "remove component", "which features matter", "component impact" | `/turing:ablate` | Analyze |
+| "frontier", "pareto", "tradeoff", "tradeoffs", "multi-objective", "which model is best" | `/turing:frontier` | Analyze |
 
 ## Sub-commands
 
@@ -52,6 +55,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:mode <mode>` | Set research strategy (explore/exploit/replicate) | (inline) |
 | `/turing:preflight` | Pre-flight resource check (VRAM/RAM/disk) | (inline) |
 | `/turing:card` | Generate standardized model card (type, performance, data, limitations, contract) | (inline) |
+| `/turing:diagnose [exp-id]` | Error analysis: failure modes, confused pairs, feature-range bias | (inline) |
+| `/turing:ablate [--components]` | Ablation study: remove components, measure impact, flag dead weight | (inline) |
+| `/turing:frontier [--metrics]` | Pareto frontier: multi-objective tradeoff visualization | (inline) |
 
 ## Proactive Detection
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -23,6 +23,7 @@ const SUB_COMMANDS = [
   "init", "train", "status", "compare", "sweep", "validate",
   "try", "brief", "suggest", "explore", "design", "logbook", "poster",
   "report", "mode", "preflight", "card", "seed", "reproduce",
+  "diagnose", "ablate", "frontier",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -33,6 +33,9 @@ const EXPECTED_COMMANDS = [
   "card/SKILL.md",
   "seed/SKILL.md",
   "reproduce/SKILL.md",
+  "diagnose/SKILL.md",
+  "ablate/SKILL.md",
+  "frontier/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];