claude-turing 1.2.0 → 1.4.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "1.2.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 17 commands, 2 specialized agents, tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "1.4.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 22 commands, 2 specialized agents, experiment intelligence (error analysis, ablation studies, Pareto frontiers), statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -323,6 +323,11 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | Command | What it does |
 |---------|-------------|
 | `/turing:validate [--auto]` | Check metric stability — auto-configure multi-run if noisy |
+| `/turing:seed [N] [--quick]` | Multi-seed study — mean/std/CI, flag seed-sensitive results |
+| `/turing:reproduce <exp-id>` | Reproducibility verification — re-run and check tolerance |
+| `/turing:diagnose [exp-id]` | Error analysis — failure modes, confused pairs, feature-range bias |
+| `/turing:ablate [--components]` | Ablation study — remove components, measure impact, flag dead weight |
+| `/turing:frontier [--metrics]` | Pareto frontier — multi-objective tradeoff visualization |
 | `/turing:card` | Generate a model card — performance, limitations, intended use, artifact contract |
 | `/turing:logbook` | Generate HTML experiment logbook |
 | `/turing:report` | Generate research report |
@@ -392,6 +397,32 @@ After N experiments with no meaningful improvement, the agent stops and reports
 
 For noisy metrics, `/turing:validate` runs the pipeline multiple times and measures variance. If the coefficient of variation exceeds 5%, it auto-configures multi-run evaluation so the agent can't be rewarded for lucky single runs.
 
+## Statistical Rigor
+
+> *"Stop publishing lucky seeds. Start publishing distributions."*
+
+Before claiming a result, run a seed study:
+
+```
+/turing:seed              # 5 seeds on best experiment
+/turing:seed --quick      # 3 seeds for fast check
+/turing:seed 10           # 10 seeds for thorough study
+```
+
+This runs the same experiment across multiple random seeds and reports mean +/- std with 95% confidence intervals. If the coefficient of variation exceeds 5%, the result is flagged as **seed-sensitive** — meaning you should report the distribution, not a single number.
+
+To verify an experiment can be reproduced:
+
+```
+/turing:reproduce exp-042                  # Default: 3 runs, 2% tolerance
+/turing:reproduce exp-042 --strict         # Exact match required
+/turing:reproduce exp-042 --tolerance 0.05 # Custom tolerance
+```
+
+This re-runs the experiment from the logged config and checks that metrics fall within tolerance. It also detects environment drift — if library versions have changed since the original run, you'll know before a reviewer tells you.
+
+Seed study results automatically appear in `/turing:brief` and `/turing:card`.
+
 ## Tree-Search Hypothesis Exploration
 
 > *"The learned coin-flipper weaves through the quadrillion-coin room with a preternatural air."*
@@ -486,11 +517,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-17 commands, 2 agents, 8 config files, 31 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, 379 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+22 commands, 2 agents, 8 config files, 37 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence (error analysis + ablation + Pareto frontier), 487 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              16 skill files (core + taste-leverage + reporting + exploration)
+├── commands/              21 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/ablate.md

@@ -0,0 +1,47 @@
+---
+name: ablate
+description: Run systematic ablation study — remove components one at a time, measure impact, produce publication-ready table with dead-weight flagging.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--components \"X,Y\"] [--seeds 3] [--latex]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Run a systematic ablation study to measure the contribution of each model component.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--components "dropout,feature_X,regularization"` specifies components to ablate
+   - `--seeds 3` runs each ablation 3 times for statistical robustness (uses seed runner)
+   - `--latex` outputs a LaTeX-formatted table instead of markdown
+
+3. **Run ablation study:**
+   ```bash
+   python scripts/ablation_study.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - Show the ablation table: Configuration | Metric | Δ from Full | % Change
+   - Rank by impact (largest Δ first)
+   - Flag **dead-weight** components (removing them improves the metric)
+   - If `--latex`, output ready for copy-paste into a paper
+
+5. **Saved output:** results written to `experiments/ablations/exp-NNN-ablation.yaml`
+
+6. **If no ablatable components detected:** suggest using `--components` explicitly.
+
+## Examples
+
+```
+/turing:ablate                                    # Auto-detect components
+/turing:ablate exp-042                            # Specific experiment
+/turing:ablate --components "dropout,subsample"   # Specific components
+/turing:ablate --seeds 3                          # Multi-seed for robustness
+/turing:ablate --latex                            # LaTeX table output
+```

package/commands/diagnose.md

@@ -0,0 +1,52 @@
+---
+name: diagnose
+description: Error analysis — cluster failure cases, identify systematic failure modes, and suggest targeted fixes with auto-queued hypotheses.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--auto-queue] [--top 5]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Analyze where and why the model fails, beyond aggregate metrics.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Generate predictions if needed:**
+   Check if `experiments/predictions/exp-NNN-preds.yaml` exists. If not, run:
+   ```bash
+   python train.py --predict-only --output experiments/predictions/
+   ```
+   The predictions file must contain `y_true`, `y_pred`, `task_type`, and optionally `features`.
+
+3. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--auto-queue` auto-queues hypotheses from failure modes into `hypotheses.yaml`
+   - `--top 5` limits to top N failure modes (default 5)
+
+4. **Run error analysis:**
+   ```bash
+   python scripts/diagnose_errors.py $ARGUMENTS
+   ```
+
+5. **Report results:**
+   - **Classification:** confusion matrix, most-confused pairs, per-class P/R/F1, low-recall classes
+   - **Regression:** residual stats, P90/P95 errors, feature-range bias, systematic bias
+   - **Failure modes:** ranked by impact, with suggested fixes
+   - **Auto-hypotheses:** if `--auto-queue`, shows queued hypotheses targeting weaknesses
+
+6. **Saved output:** report written to `experiments/diagnoses/exp-NNN-diagnosis.yaml`
+
+7. **If no predictions file exists:** instruct user to run the model on validation set first.
+
+## Examples
+
+```
+/turing:diagnose                    # Analyze best experiment
+/turing:diagnose exp-042            # Specific experiment
+/turing:diagnose --auto-queue       # Queue fix hypotheses
+/turing:diagnose --top 10           # Top 10 failure modes
+```

package/commands/frontier.md

@@ -0,0 +1,45 @@
+---
+name: frontier
+description: Visualize Pareto frontier across multiple objectives — answers "which model is actually best?" when there are tradeoffs.
+disable-model-invocation: true
+argument-hint: "[--metrics \"accuracy,train_seconds,n_params\"] [--ascii]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Visualize the Pareto frontier across multiple objectives from experiment history.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--metrics "accuracy,train_seconds,n_params"` specifies metrics to analyze
+   - Without `--metrics`, uses primary metric + train_seconds from config
+   - `--ascii` generates an ASCII scatter plot (2D projection)
+
+3. **Run Pareto analysis:**
+   ```bash
+   python scripts/pareto_frontier.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **Pareto-optimal experiments:** table with all metrics and what each is best at
+   - **Dominated experiments:** with their nearest Pareto neighbor
+   - **ASCII scatter plot** (if `--ascii`): 2D projection with * for Pareto, · for dominated
+   - Summary: "N Pareto-optimal of M experiments across K metrics"
+
+5. **Saved output:** results written to `experiments/frontiers/frontier-YYYY-MM-DD.yaml`
+
+6. **If no experiments have all requested metrics:** suggest which metrics are available.
+
+## Examples
+
+```
+/turing:frontier                                              # Default: metric vs time
+/turing:frontier --metrics "accuracy,train_seconds"           # 2D frontier
+/turing:frontier --metrics "accuracy,train_seconds,n_params"  # 3D frontier
+/turing:frontier --ascii                                      # With scatter plot
+```

package/commands/reproduce.md

@@ -0,0 +1,48 @@
+---
+name: reproduce
+description: Verify reproducibility of a specific experiment by re-running from logged config and checking metrics fall within tolerance.
+disable-model-invocation: true
+argument-hint: "<exp-id> [--tolerance 0.02] [--strict] [--runs 3]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Verify that a logged experiment can be reproduced with consistent results.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First argument is the experiment ID (required), e.g. `exp-042`
+   - `--tolerance 0.02` sets the relative tolerance (default 2%)
+   - `--strict` requires exact float match (1e-6), overrides tolerance
+   - `--runs 3` sets number of reproduction runs (default 3, 1 for strict)
+
+3. **Run reproducibility verification:**
+   ```bash
+   python scripts/reproduce_experiment.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **reproducible:** metrics match exactly (deterministic algorithm)
+   - **approximately_reproducible:** metrics within tolerance or original falls in 95% CI
+   - **not_reproducible:** metrics outside tolerance and CI
+   - **environment_changed:** metrics diverge AND library versions differ
+   - Show environment diff if present (Python version, package versions)
+
+5. **Saved output:** report written to `experiments/reproductions/exp-NNN-repro.yaml`
+
+6. **If experiment ID not found:** list available experiment IDs from `experiments/log.jsonl`
+
+7. **If no training pipeline exists:** suggest `/turing:init` first.
+
+## Examples
+
+```
+/turing:reproduce exp-042                 # Default: 3 runs, 2% tolerance
+/turing:reproduce exp-042 --strict        # Exact match required
+/turing:reproduce exp-042 --tolerance 0.05 --runs 5   # Lenient, more runs
+```

package/commands/seed.md

@@ -0,0 +1,47 @@
+---
+name: seed
+description: Run multi-seed study on an experiment to compute mean/std/CI and flag seed-sensitive results. Prevents publishing lucky seeds.
+disable-model-invocation: true
+argument-hint: "[N] [--quick] [--exp-id <id>]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Run a multi-seed study to verify that experiment results are robust across random seeds.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - A bare number (e.g., `5`) sets the seed count
+   - `--quick` runs 3 seeds instead of 5
+   - `--exp-id exp-042` targets a specific experiment (defaults to best)
+   - `--seed-list 42,123,456` uses specific seed values
+
+3. **Run seed study:**
+   ```bash
+   python scripts/seed_runner.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - Show the per-seed results table
+   - Show mean +/- std with 95% CI
+   - **STABLE (CV < 5%):** result is robust, safe to report
+   - **SEED-SENSITIVE (CV >= 5%):** result varies too much across seeds — do not report single-seed numbers
+   - If seed-sensitive, recommend reporting as mean +/- std over N seeds
+
+5. **Saved output:** results are written to `experiments/seed_studies/exp-NNN-seeds.yaml`
+
+6. **If no training pipeline exists:** suggest `/turing:init` first.
+
+## Examples
+
+```
+/turing:seed              # 5 seeds on best experiment
+/turing:seed --quick      # 3 seeds for fast check
+/turing:seed 10           # 10 seeds for thorough study
+/turing:seed --exp-id exp-042   # Specific experiment
+```

package/commands/turing.md

@@ -20,12 +20,17 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "poster", "presentation", "one-pager", "visual summary" | `/turing:poster` | Document |
 | "report", "write-up", "findings", "document results" | `/turing:report` | Document |
 | "validate", "stability", "check variance", "noisy" | `/turing:validate` | Validate |
+| "seed", "seed study", "multi-seed", "lucky seed", "seed sensitivity" | `/turing:seed` | Validate |
+| "reproduce", "reproducibility", "verify results", "re-run experiment", "repro" | `/turing:reproduce` | Validate |
 | "suggest", "what model", "recommend", "which architecture", "literature" | `/turing:suggest` | Research |
 | "explore hypotheses", "tree search", "treequest", "search hypothesis space", "MCTS" | `/turing:explore` | Research |
 | "design", "plan experiment", "how should I test", "experiment design" | `/turing:design` | Design |
 | "mode", "explore", "exploit", "replicate", "strategy" | `/turing:mode` | Strategy |
 | "preflight", "resources", "VRAM", "memory", "can I run", "OOM", "GPU" | `/turing:preflight` | Check |
 | "card", "model card", "document model", "model documentation" | `/turing:card` | Document |
+| "diagnose", "error analysis", "failure modes", "where does it fail", "confusion matrix" | `/turing:diagnose` | Analyze |
+| "ablate", "ablation", "remove component", "which features matter", "component impact" | `/turing:ablate` | Analyze |
+| "frontier", "pareto", "tradeoff", "tradeoffs", "multi-objective", "which model is best" | `/turing:frontier` | Analyze |
 
 ## Sub-commands
 
@@ -39,6 +44,8 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:brief` | Generate structured research intelligence report | @ml-evaluator |
 | `/turing:init` | Scaffold a new ML project | (inline) |
 | `/turing:validate` | Check metric stability, auto-fix if noisy | (inline) |
+| `/turing:seed [N] [--quick]` | Multi-seed study: mean/std/CI, flag seed-sensitive results | (inline) |
+| `/turing:reproduce <exp-id>` | Reproducibility verification with tolerance checking | (inline) |
 | `/turing:suggest` | Literature-grounded model architecture suggestions | (inline, uses WebSearch) |
 | `/turing:explore` | Tree-search hypothesis exploration via AB-MCTS | (inline) |
 | `/turing:design <hyp-id>` | Generate structured experiment design from hypothesis | (inline, uses WebSearch) |
@@ -48,6 +55,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:mode <mode>` | Set research strategy (explore/exploit/replicate) | (inline) |
 | `/turing:preflight` | Pre-flight resource check (VRAM/RAM/disk) | (inline) |
 | `/turing:card` | Generate standardized model card (type, performance, data, limitations, contract) | (inline) |
+| `/turing:diagnose [exp-id]` | Error analysis: failure modes, confused pairs, feature-range bias | (inline) |
+| `/turing:ablate [--components]` | Ablation study: remove components, measure impact, flag dead weight | (inline) |
+| `/turing:frontier [--metrics]` | Pareto frontier: multi-objective tradeoff visualization | (inline) |
 
 ## Proactive Detection
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "1.2.0",
+  "version": "1.4.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -22,7 +22,8 @@ const PLUGIN_ROOT = join(__dirname, "..");
 const SUB_COMMANDS = [
   "init", "train", "status", "compare", "sweep", "validate",
   "try", "brief", "suggest", "explore", "design", "logbook", "poster",
-  "report", "mode", "preflight", "card",
+  "report", "mode", "preflight", "card", "seed", "reproduce",
+  "diagnose", "ablate", "frontier",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -31,6 +31,11 @@ const EXPECTED_COMMANDS = [
   "mode/SKILL.md",
   "preflight/SKILL.md",
   "card/SKILL.md",
+  "seed/SKILL.md",
+  "reproduce/SKILL.md",
+  "diagnose/SKILL.md",
+  "ablate/SKILL.md",
+  "frontier/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];

package/templates/config.yaml

@@ -19,6 +19,16 @@ evaluation:
   # Set to false for metrics where higher is better (accuracy, f1, auc)
   lower_is_better: false  # {{METRIC_DIRECTION}} -- change to true if lower is better
 
+  # Multi-seed configuration (Phase 10.1: /turing:seed)
+  # Seeds used for seed studies — diverse values for good coverage
+  seed_seeds: [42, 123, 456, 789, 1024, 1337, 2048, 3141, 4096, 7919]
+  seed_study_n_runs: 5        # Default number of seeds for /turing:seed
+  seed_sensitivity_threshold: 5.0  # CV% above this = seed-sensitive
+
+  # Reproducibility configuration (Phase 10.2: /turing:reproduce)
+  reproduce_tolerance: 0.02   # 2% relative tolerance for approximate match
+  reproduce_n_runs: 3         # Default reproduction runs for stochastic algorithms
+
 convergence:
   patience: 3                    # Consecutive non-improvements before stopping
   improvement_threshold: 0.005   # 0.5% relative improvement required

package/templates/program.md

@@ -170,6 +170,11 @@ The autoresearch experiment loop. Each iteration is one experiment — one hypot
    - N consecutive non-improvements (`config.yaml` → `convergence.patience`) = STOP
    - `max_iterations` reached = STOP
    - Report final best model and recommend next steps
+   - **Before declaring final results**, run a seed study to verify robustness:
+     ```bash
+     python scripts/seed_runner.py --quick
+     ```
+     If CV > 5%, the result is seed-sensitive — report mean ± std, not a single-seed number.
 
 10. **REPEAT** — return to step 1.