claude-turing 2.5.0 → 3.0.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "2.5.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 39 commands, 2 specialized agents, scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "3.0.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 41 commands, 2 specialized agents, meta-intelligence (cross-project knowledge transfer + methodology audit), scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -350,6 +350,8 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:scale [--axis]` | Scaling law estimator — power-law fit, full-scale predictions, diminishing returns verdict |
 | `/turing:budget <action>` | Compute budget manager — set limits, track allocation, auto-shift explore/exploit |
 | `/turing:distill <exp-id>` | Model compression — distill teacher into smaller student with accuracy/size tradeoff |
+| `/turing:transfer [--from]` | Cross-project knowledge transfer — find similar projects, surface what worked |
+| `/turing:audit [--strict]` | Pre-submission methodology audit — data leakage, baselines, seeds, ablations, reproducibility |
 
 And for fully hands-off operation:
 
@@ -534,11 +536,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-39 commands, 2 agents, 10 config files, 58 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+41 commands, 2 agents, 10 config files, 60 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              38 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency)
+├── commands/              40 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency + meta-intelligence)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/audit.md

@@ -0,0 +1,56 @@
+---
+name: audit
+description: Pre-submission methodology audit — catch data leakage, missing baselines, cherry-picked seeds, and incomplete ablations before a reviewer does.
+disable-model-invocation: true
+argument-hint: "[--strict] [--checklist neurips]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+A reviewer checklist you run before submitting. Catches methodology mistakes that cause desk rejections.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--strict` — treat warnings as failures
+   - `--checklist neurips|icml|iclr` — add venue-specific checks
+   - `--json` — raw JSON output
+
+3. **Run methodology audit:**
+   ```bash
+   python scripts/methodology_audit.py $ARGUMENTS
+   ```
+
+4. **Checks performed:**
+   - **Data leakage** (critical): verify prepare.py/evaluate.py separation
+   - **CV strategy** (critical): verify appropriate cross-validation for data type
+   - **Seed sensitivity** (high): seed studies exist for best experiments
+   - **Ablation completeness** (high): ablation studies performed
+   - **Baseline comparison** (high): simple baselines in experiment log
+   - **Reproducibility** (high): best result successfully reproduced
+   - **Hyperparameter budget** (medium): total tuning cost documented
+   - **Regression stability** (medium): regression checks performed
+
+5. **Verdicts:**
+   - **PASS** — ready for submission
+   - **PASS (with warnings)** — address before submission
+   - **NEEDS WORK** — fix failures first
+   - **FAIL** — critical issues found
+
+6. **Actions:** each failure suggests the `/turing:` command to fix it
+
+7. **Venue checklists:** `--checklist neurips` adds NeurIPS-specific checks (broader impact, reproducibility checklist, code availability)
+
+8. **Saved output:** report in `experiments/audits/audit-YYYY-MM-DD.yaml`
+
+## Examples
+
+```
+/turing:audit                          # Standard audit
+/turing:audit --strict                 # Warnings become failures
+/turing:audit --checklist neurips      # NeurIPS submission checklist
+```

package/commands/transfer.md

@@ -0,0 +1,54 @@
+---
+name: transfer
+description: Cross-project knowledge transfer — find similar prior projects and surface what worked. Builds institutional ML memory.
+disable-model-invocation: true
+argument-hint: "[--from project-path] [--auto]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Find similar prior projects and surface what worked. "Last time you had tabular classification with class imbalance, LightGBM beat everything by 3%."
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - `--from ~/projects/fraud-detection` — transfer from a specific project
+   - `--auto` — auto-queue hypotheses from recommendations
+   - `--index ~/.turing/project_index.yaml` — custom index path
+   - `--json` — raw JSON output
+
+3. **Run knowledge transfer:**
+   ```bash
+   python scripts/knowledge_transfer.py $ARGUMENTS
+   ```
+
+4. **Report includes:**
+   - Similar prior projects ranked by similarity score
+   - Per project: task type, winner model, key insights
+   - Suggested hypotheses from winning strategies
+   - Auto-queued hypotheses (with `--auto`)
+
+5. **Similarity matching** uses:
+   - Task type (classification/regression) — highest weight
+   - Dataset size (log-scale comparison)
+   - Feature types (tabular/image/text)
+   - Class balance characteristics
+   - Dimensionality
+
+6. **Project index** at `~/.turing/project_index.yaml` — local only, never uploaded
+
+7. **If no similar projects found:** suggest running on more projects first or specifying one with `--from`
+
+8. **Saved output:** report in `experiments/transfers/transfer-*.yaml`
+
+## Examples
+
+```
+/turing:transfer                                    # Search index for similar projects
+/turing:transfer --from ~/projects/fraud-detection  # Transfer from specific project
+/turing:transfer --auto                             # Auto-queue hypotheses
+```

package/commands/turing.md

@@ -48,6 +48,8 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "scale", "scaling law", "how much data", "is more data worth it", "power law", "data efficiency" | `/turing:scale` | Analyze |
 | "budget", "compute budget", "how many experiments", "spending limit", "stop after" | `/turing:budget` | Manage |
 | "distill", "compress", "smaller model", "student model", "knowledge distillation", "model compression" | `/turing:distill` | Deploy |
+| "transfer", "what worked before", "similar project", "cross-project", "institutional knowledge", "prior projects" | `/turing:transfer` | Research |
+| "audit", "methodology check", "pre-submission", "reviewer checklist", "data leakage", "missing baselines" | `/turing:audit` | Validate |
 
 ## Sub-commands
 
@@ -92,6 +94,8 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:scale [--axis]` | Scaling law estimator: fit power law, predict full-scale performance | (inline) |
 | `/turing:budget <action>` | Compute budget manager: set limits, track allocation, auto-shift modes | (inline) |
 | `/turing:distill <exp-id>` | Model compression: distill teacher into smaller student model | (inline) |
+| `/turing:transfer [--from]` | Cross-project knowledge transfer: find similar prior projects, surface what worked | (inline) |
+| `/turing:audit [--strict]` | Pre-submission methodology audit: data leakage, baselines, seeds, ablations, reproducibility | (inline) |
 
 ## Proactive Detection
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "2.5.0",
+  "version": "3.0.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -28,6 +28,7 @@ const SUB_COMMANDS = [
   "diff", "watch", "regress",
   "ensemble", "stitch", "warm",
   "scale", "budget", "distill",
+  "transfer", "audit",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -53,6 +53,8 @@ const EXPECTED_COMMANDS = [
   "scale/SKILL.md",
   "budget/SKILL.md",
   "distill/SKILL.md",
+  "transfer/SKILL.md",
+  "audit/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];

package/templates/scripts/generate_brief.py

@@ -355,6 +355,22 @@ def load_scaling_results(scaling_dir: str = "experiments/scaling") -> list[dict]
     return reports
 
 
+def load_audit_report(audit_dir: str = "experiments/audits") -> dict | None:
+    """Load the most recent audit report."""
+    path = Path(audit_dir)
+    if not path.exists():
+        return None
+    files = sorted(path.glob("audit-*.yaml"))
+    if not files:
+        return None
+    try:
+        with open(files[-1]) as f:
+            report = yaml.safe_load(f)
+            return report if isinstance(report, dict) else None
+    except (yaml.YAMLError, OSError):
+        return None
+
+
 def format_brief(
     campaign: dict,
     best: dict | None,
@@ -376,6 +392,7 @@ def format_brief(
     ensemble_results: list[dict] | None = None,
     budget_status: dict | None = None,
     scaling_results: list[dict] | None = None,
+    audit_report: dict | None = None,
 ) -> str:
     """Format the research briefing as markdown."""
     direction = "lower" if lower_is_better else "higher"
@@ -635,6 +652,28 @@ def format_brief(
             reason = verdict.get("reason", "")
             lines.append(f"- **{v.upper()}**: {reason}")
 
+    # Methodology audit
+    if audit_report and audit_report.get("score"):
+        score = audit_report["score"]
+        verdict = audit_report.get("verdict", "?")
+        verdict_labels = {
+            "pass": "PASS",
+            "pass_with_warnings": "PASS (warnings)",
+            "needs_work": "NEEDS WORK",
+            "fail": "FAIL",
+        }
+        lines.extend(["", "## Methodology Audit", ""])
+        lines.append(
+            f"**{verdict_labels.get(verdict, verdict.upper())}** — "
+            f"{score.get('pass', 0)}/{score.get('checkable', 0)} checks passed, "
+            f"{score.get('fail', 0)} failure(s)"
+        )
+        actions = audit_report.get("actions", [])
+        if actions:
+            lines.append("")
+            for a in actions[:3]:
+                lines.append(f"- Fix: `{a['fix']}` ({a['check']})")
+
     # Regression check history (stability)
     if regression_checks:
         lines.extend(["", "## Stability", ""])
@@ -728,6 +767,7 @@ def generate_brief(
     ensemble_results = load_ensemble_results()
     budget_status = load_budget_status(log_path=log_path)
     scaling_results = load_scaling_results()
+    audit_report = load_audit_report()
 
     return format_brief(
         campaign, best, trajectory, model_types, hypotheses,
@@ -743,6 +783,7 @@ def generate_brief(
         ensemble_results=ensemble_results if ensemble_results else None,
         budget_status=budget_status,
         scaling_results=scaling_results if scaling_results else None,
+        audit_report=audit_report,
     )