claude-turing 2.5.0 → 3.1.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "2.5.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 39 commands, 2 specialized agents, scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "3.1.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 44 commands, 2 specialized agents, pre-training intelligence (sanity checks + baseline generation + leakage detection), meta-intelligence (cross-project knowledge transfer + methodology audit), scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -350,6 +350,11 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:scale [--axis]` | Scaling law estimator — power-law fit, full-scale predictions, diminishing returns verdict |
 | `/turing:budget <action>` | Compute budget manager — set limits, track allocation, auto-shift explore/exploit |
 | `/turing:distill <exp-id>` | Model compression — distill teacher into smaller student with accuracy/size tradeoff |
+| `/turing:transfer [--from]` | Cross-project knowledge transfer — find similar projects, surface what worked |
+| `/turing:audit [--strict]` | Pre-submission methodology audit — data leakage, baselines, seeds, ablations, reproducibility |
+| `/turing:sanity [--quick]` | Pre-training sanity checks — initial loss, single-batch overfit, gradient flow, output validation |
+| `/turing:baseline [--methods]` | Automatic baseline generation — random, majority/mean, linear, k-NN |
+| `/turing:leak [--deep]` | Targeted leakage detection — single-feature tests, correlation, train/test overlap |
 
 And for fully hands-off operation:
 
@@ -534,11 +539,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-39 commands, 2 agents, 10 config files, 58 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+44 commands, 2 agents, 10 config files, 63 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), pre-training intelligence (sanity + baseline + leak), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              38 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency)
+├── commands/              43 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency + meta-intelligence + pre-training intelligence)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/audit.md

@@ -0,0 +1,56 @@
+---
+name: audit
+description: Pre-submission methodology audit — catch data leakage, missing baselines, cherry-picked seeds, and incomplete ablations before a reviewer does.
+disable-model-invocation: true
+argument-hint: "[--strict] [--checklist neurips]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+A reviewer checklist you run before submitting. Catches methodology mistakes that cause desk rejections.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--strict` — treat warnings as failures
+   - `--checklist neurips|icml|iclr` — add venue-specific checks
+   - `--json` — raw JSON output
+
+3. **Run methodology audit:**
+   ```bash
+   python scripts/methodology_audit.py $ARGUMENTS
+   ```
+
+4. **Checks performed:**
+   - **Data leakage** (critical): verify prepare.py/evaluate.py separation
+   - **CV strategy** (critical): verify appropriate cross-validation for data type
+   - **Seed sensitivity** (high): seed studies exist for best experiments
+   - **Ablation completeness** (high): ablation studies performed
+   - **Baseline comparison** (high): simple baselines in experiment log
+   - **Reproducibility** (high): best result successfully reproduced
+   - **Hyperparameter budget** (medium): total tuning cost documented
+   - **Regression stability** (medium): regression checks performed
+
+5. **Verdicts:**
+   - **PASS** — ready for submission
+   - **PASS (with warnings)** — address before submission
+   - **NEEDS WORK** — fix failures first
+   - **FAIL** — critical issues found
+
+6. **Actions:** each failure suggests the `/turing:` command to fix it
+
+7. **Venue checklists:** `--checklist neurips` adds NeurIPS-specific checks (broader impact, reproducibility checklist, code availability)
+
+8. **Saved output:** report in `experiments/audits/audit-YYYY-MM-DD.yaml`
+
+## Examples
+
+```
+/turing:audit                          # Standard audit
+/turing:audit --strict                 # Warnings become failures
+/turing:audit --checklist neurips      # NeurIPS submission checklist
+```

package/commands/baseline.md

@@ -0,0 +1,45 @@
+---
+name: baseline
+description: Automatic baseline generation — random, majority/mean, linear, k-NN baselines in 60 seconds. Every experiment needs a "is this better than dumb?" reference.
+disable-model-invocation: true
+argument-hint: "[--methods all|simple|linear] [--data data.npz]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Generate trivial baselines so you always know if your model is meaningfully better than simple approaches.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--methods all|simple|linear` — baseline group (default: all)
+   - `--data data.npz` — data file with X and y arrays
+   - `--json` — raw JSON output
+
+3. **Run baseline generation:**
+   ```bash
+   python scripts/generate_baselines.py $ARGUMENTS
+   ```
+
+4. **Baselines generated:**
+   - **Classification:** Random, Majority class, Stratified random, Logistic Regression, k-NN
+   - **Regression:** Random, Mean predictor, Median predictor, Ridge Regression, k-NN
+   - Each evaluated with the same protocol as real experiments
+
+5. **Report includes:** comparison table with metric values and notes (floor, ceiling, reference)
+
+6. **Integration:** satisfies the "baseline comparison" check in `/turing:audit`
+
+7. **Saved output:** report in `experiments/baselines/baselines-*.yaml`
+
+## Examples
+
+```
+/turing:baseline                           # All baselines
+/turing:baseline --methods simple          # Just random + majority
+/turing:baseline --data data/processed.npz # With actual data
+```

package/commands/leak.md

@@ -0,0 +1,47 @@
+---
+name: leak
+description: Targeted leakage detection — probe for data leakage with single-feature tests, correlation checks, and train/test overlap detection.
+disable-model-invocation: true
+argument-hint: "[--deep] [--features feature_1,feature_2]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Actively probe for data leakage. The #1 cause of "too good to be true" results.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--deep` — run full single-feature analysis (slow but thorough)
+   - `--features "feat_1,feat_2"` — check specific features
+   - `--json` — raw JSON output
+
+3. **Run leakage scan:**
+   ```bash
+   python scripts/leakage_detector.py $ARGUMENTS
+   ```
+
+4. **Checks performed:**
+   - **Feature-target correlation:** flag features with >0.95 correlation to target
+   - **Single-feature predictiveness (--deep):** train on each feature alone, flag any that achieve >80% of full model performance
+   - **Train/test overlap:** hash-based deduplication across splits
+
+5. **Verdicts:**
+   - **CLEAN** — no leakage detected
+   - **SUSPICIOUS** — warnings to review
+   - **LEAKAGE DETECTED** — critical flags found
+
+6. **Integration:** satisfies the "data leakage" check in `/turing:audit`
+
+7. **Saved output:** report in `experiments/leakage/leak-*.yaml`
+
+## Examples
+
+```
+/turing:leak                    # Standard correlation + overlap checks
+/turing:leak --deep             # Full single-feature analysis
+```

package/commands/sanity.md

@@ -0,0 +1,48 @@
+---
+name: sanity
+description: Pre-training sanity checks — catch broken data loaders, misconfigured losses, and dead gradients in 30 seconds before wasting hours.
+disable-model-invocation: true
+argument-hint: "[--quick] [--verbose]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Run a battery of fast checks before committing to a full training run. Catches wiring bugs in seconds.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--quick` — skip single-batch overfit test (fastest, ~5 seconds)
+   - `--verbose` — show detailed check output
+   - `--json` — raw JSON output
+
+3. **Run sanity checks:**
+   ```bash
+   python scripts/sanity_checks.py $ARGUMENTS
+   ```
+
+4. **Checks performed:**
+   - **Data pipeline** (critical): first batch loads, shapes match, no NaN/Inf
+   - **Initial loss** (high): loss at initialization matches theory (e.g., -log(1/C) for cross-entropy)
+   - **Gradient flow** (high): all parameters have non-zero, non-exploding gradients
+   - **Single-batch overfit** (critical): model can memorize 1 batch in 50 steps — if not, something is broken
+   - **Output validation** (high): predictions are non-NaN, non-constant, reasonable range
+   - **Config consistency** (medium): learning rate, batch size in reasonable ranges
+
+5. **Verdicts:**
+   - **PASS** — safe to proceed
+   - **PASS (with warnings)** — review before training
+   - **FAIL** — do not proceed, fix issues first
+
+6. **Saved output:** report in `experiments/sanity/sanity-*.yaml`
+
+## Examples
+
+```
+/turing:sanity                    # Full check (~30 seconds)
+/turing:sanity --quick            # Skip overfit test (~5 seconds)
+```

package/commands/transfer.md

@@ -0,0 +1,54 @@
+---
+name: transfer
+description: Cross-project knowledge transfer — find similar prior projects and surface what worked. Builds institutional ML memory.
+disable-model-invocation: true
+argument-hint: "[--from project-path] [--auto]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Find similar prior projects and surface what worked. "Last time you had tabular classification with class imbalance, LightGBM beat everything by 3%."
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--from ~/projects/fraud-detection` — transfer from a specific project
+   - `--auto` — auto-queue hypotheses from recommendations
+   - `--index ~/.turing/project_index.yaml` — custom index path
+   - `--json` — raw JSON output
+
+3. **Run knowledge transfer:**
+   ```bash
+   python scripts/knowledge_transfer.py $ARGUMENTS
+   ```
+
+4. **Report includes:**
+   - Similar prior projects ranked by similarity score
+   - Per project: task type, winner model, key insights
+   - Suggested hypotheses from winning strategies
+   - Auto-queued hypotheses (with `--auto`)
+
+5. **Similarity matching** uses:
+   - Task type (classification/regression) — highest weight
+   - Dataset size (log-scale comparison)
+   - Feature types (tabular/image/text)
+   - Class balance characteristics
+   - Dimensionality
+
+6. **Project index** at `~/.turing/project_index.yaml` — local only, never uploaded
+
+7. **If no similar projects found:** suggest running on more projects first or specifying one with `--from`
+
+8. **Saved output:** report in `experiments/transfers/transfer-*.yaml`
+
+## Examples
+
+```
+/turing:transfer                                    # Search index for similar projects
+/turing:transfer --from ~/projects/fraud-detection  # Transfer from specific project
+/turing:transfer --auto                             # Auto-queue hypotheses
+```

package/commands/turing.md

@@ -48,6 +48,11 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "scale", "scaling law", "how much data", "is more data worth it", "power law", "data efficiency" | `/turing:scale` | Analyze |
 | "budget", "compute budget", "how many experiments", "spending limit", "stop after" | `/turing:budget` | Manage |
 | "distill", "compress", "smaller model", "student model", "knowledge distillation", "model compression" | `/turing:distill` | Deploy |
+| "transfer", "what worked before", "similar project", "cross-project", "institutional knowledge", "prior projects" | `/turing:transfer` | Research |
+| "audit", "methodology check", "pre-submission", "reviewer checklist", "data leakage", "missing baselines" | `/turing:audit` | Validate |
+| "sanity", "sanity check", "pre-training", "is it broken", "before training", "quick check" | `/turing:sanity` | Check |
+| "baseline", "baselines", "trivial baseline", "majority class", "is it better than random" | `/turing:baseline` | Analyze |
+| "leak", "leakage", "data leakage scan", "suspicious feature", "train test overlap" | `/turing:leak` | Validate |
 
 ## Sub-commands
 
@@ -92,6 +97,11 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:scale [--axis]` | Scaling law estimator: fit power law, predict full-scale performance | (inline) |
 | `/turing:budget <action>` | Compute budget manager: set limits, track allocation, auto-shift modes | (inline) |
 | `/turing:distill <exp-id>` | Model compression: distill teacher into smaller student model | (inline) |
+| `/turing:transfer [--from]` | Cross-project knowledge transfer: find similar prior projects, surface what worked | (inline) |
+| `/turing:audit [--strict]` | Pre-submission methodology audit: data leakage, baselines, seeds, ablations, reproducibility | (inline) |
+| `/turing:sanity [--quick]` | Pre-training sanity checks: initial loss, overfit test, gradient flow, output validation | (inline) |
+| `/turing:baseline [--methods]` | Automatic baseline generation: random, majority/mean, linear, k-NN | (inline) |
+| `/turing:leak [--deep]` | Targeted leakage detection: single-feature tests, correlation, train/test overlap | (inline) |
 
 ## Proactive Detection
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "2.5.0",
+  "version": "3.1.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -28,6 +28,8 @@ const SUB_COMMANDS = [
   "diff", "watch", "regress",
   "ensemble", "stitch", "warm",
   "scale", "budget", "distill",
+  "transfer", "audit",
+  "sanity", "baseline", "leak",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -53,6 +53,11 @@ const EXPECTED_COMMANDS = [
   "scale/SKILL.md",
   "budget/SKILL.md",
   "distill/SKILL.md",
+  "transfer/SKILL.md",
+  "audit/SKILL.md",
+  "sanity/SKILL.md",
+  "baseline/SKILL.md",
+  "leak/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];