claude-turing 4.0.0 → 4.1.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "4.0.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 63 commands, 2 specialized agents, research communication (cite + present + changelog), experiment archaeology (trend + flashback + archive + annotate + search + template + replay), model surgery (prune + quantize + merge + surgery), feature & training intelligence, model debugging, pre-training intelligence, meta-intelligence, scaling & efficiency, model composition, deep analysis, experiment orchestration, literature + paper, model export, profiling, checkpoints, experiment intelligence, statistical rigor, tree-search, cost-performance, model cards, hypothesis database, novelty guard, anti-cheating, taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "4.1.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 66 commands, 2 specialized agents, collaboration (onboard + share + review), research communication (cite + present + changelog), experiment archaeology (trend + flashback + archive + annotate + search + template + replay), model surgery (prune + quantize + merge + surgery), feature & training intelligence, model debugging, pre-training intelligence, meta-intelligence, scaling & efficiency, model composition, deep analysis, experiment orchestration, literature + paper, model export, profiling, checkpoints, experiment intelligence, statistical rigor, tree-search, cost-performance, model cards, hypothesis database, novelty guard, anti-cheating, taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -374,6 +374,9 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:cite <action>` | Citation & attribution manager — track papers, audit missing citations, generate BibTeX |
 | `/turing:present [--figures]` | Presentation figures — training curves, comparisons, ablation, Pareto, sensitivity |
 | `/turing:changelog [--audience]` | Model changelog — version-grouped improvements for technical or stakeholder audiences |
+| `/turing:onboard [--audience]` | Project onboarding — walkthrough for new collaborators |
+| `/turing:share <exp-ids...>` | Experiment packaging — portable archive with manifest |
+| `/turing:review [--venue]` | Peer review simulation — weaknesses, fix commands, score |
 
 And for fully hands-off operation:
 
@@ -558,7 +561,7 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-63 commands, 2 agents, 10 config files, 82 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), pre-training intelligence (sanity + baseline + leak), model debugging (xray + sensitivity + calibrate), feature & training intelligence (feature + curriculum), model surgery (prune + quantize + merge + surgery), experiment archaeology (trend + flashback + archive + annotate + search + template + replay), research communication (cite + present + changelog), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+66 commands, 2 agents, 10 config files, 85 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), pre-training intelligence (sanity + baseline + leak), model debugging (xray + sensitivity + calibrate), feature & training intelligence (feature + curriculum), model surgery (prune + quantize + merge + surgery), experiment archaeology (trend + flashback + archive + annotate + search + template + replay), research communication (cite + present + changelog), collaboration (onboard + share + review), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/

package/commands/onboard.md

@@ -0,0 +1,20 @@
+---
+name: onboard
+description: Project onboarding — generate a walkthrough for new collaborators. Task, history, decisions, next steps.
+disable-model-invocation: true
+argument-hint: "[--audience researcher|engineer|stakeholder] [--depth brief|full]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+5-minute read that replaces a 1-hour onboarding meeting.
+
+## Steps
+1. `source .venv/bin/activate`
+2. `python scripts/generate_onboarding.py $ARGUMENTS`
+3. **Saved:** `ONBOARDING.md`
+
+## Examples
+```
+/turing:onboard
+/turing:onboard --audience engineer --depth brief
+```

package/commands/review.md

@@ -0,0 +1,20 @@
+---
+name: review
+description: Peer review simulation — generate likely reviewer objections with severity ratings and fix commands.
+disable-model-invocation: true
+argument-hint: "[--venue neurips|icml|general] [--harsh]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Simulate a conference reviewer before you submit. Each weakness links to the command that fixes it.
+
+## Steps
+1. `source .venv/bin/activate`
+2. `python scripts/simulate_review.py $ARGUMENTS`
+3. **Saved:** `experiments/reviews/`
+
+## Examples
+```
+/turing:review
+/turing:review --venue neurips --harsh
+```

package/commands/share.md

@@ -0,0 +1,20 @@
+---
+name: share
+description: Experiment packaging — portable archive with config, metrics, seed study, annotations, reproduction instructions.
+disable-model-invocation: true
+argument-hint: "<exp-ids...> [--include model,figures,code]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Package experiments for collaborator handoff or paper supplementary material.
+
+## Steps
+1. `source .venv/bin/activate`
+2. `python scripts/package_experiments.py $ARGUMENTS`
+3. **Saved:** `exports/packages/<name>/`
+
+## Examples
+```
+/turing:share exp-089
+/turing:share exp-042 exp-089 --include model,figures
+```

package/commands/turing.md

@@ -65,6 +65,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "cite", "citation", "bibliography", "bibtex", "attribution", "references" | `/turing:cite` | Record |
 | "present", "figures", "slides", "presentation", "charts", "plots" | `/turing:present` | Document |
 | "changelog", "model changelog", "progress summary", "what improved" | `/turing:changelog` | Document |
+| "onboard", "onboarding", "walkthrough", "new collaborator", "project overview" | `/turing:onboard` | Document |
+| "share", "package", "export experiments", "send results", "portable" | `/turing:share` | Share |
+| "review", "peer review", "reviewer", "simulate review", "weakness" | `/turing:review` | Validate |
 | "trend", "trends", "research direction", "improvement rate", "diminishing returns", "what's working" | `/turing:trend` | Analyze |
 | "flashback", "where was I", "context", "resume", "catch up", "what happened" | `/turing:flashback` | Recall |
 | "archive", "cleanup", "compress old", "disk space", "archive experiments" | `/turing:archive` | Manage |
@@ -140,6 +143,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:cite <action>` | Citation manager: add/list/check/bib for papers, datasets, methods | (inline) |
 | `/turing:present [--figures]` | Presentation figures: training curves, comparisons, ablation, Pareto, sensitivity | (inline) |
 | `/turing:changelog [--audience]` | Model changelog: version-grouped improvements for technical or stakeholder audiences | (inline) |
+| `/turing:onboard [--audience]` | Project onboarding: full walkthrough for new collaborators | (inline) |
+| `/turing:share <exp-ids...>` | Experiment packaging: portable archive with manifest and README | (inline) |
+| `/turing:review [--venue]` | Peer review simulation: weaknesses, questions, fix commands, score | (inline) |
 
 ## Proactive Detection
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "4.0.0",
+  "version": "4.1.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -35,6 +35,7 @@ const SUB_COMMANDS = [
   "prune", "quantize", "merge", "surgery",
   "trend", "flashback", "archive", "annotate", "search", "template", "replay",
   "cite", "present", "changelog",
+  "onboard", "share", "review",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -77,6 +77,9 @@ const EXPECTED_COMMANDS = [
   "cite/SKILL.md",
   "present/SKILL.md",
   "changelog/SKILL.md",
+  "onboard/SKILL.md",
+  "share/SKILL.md",
+  "review/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];

package/templates/scripts/generate_onboarding.py

@@ -0,0 +1,284 @@
+#!/usr/bin/env python3
+"""Project onboarding generator for new collaborators.
+
+Reads config, experiments, annotations, and hypotheses to produce a
+structured walkthrough: task description, what's been tried (grouped
+by family), key decisions, where heading, how to start.
+
+Usage:
+    python scripts/generate_onboarding.py --audience researcher --depth full
+    python scripts/generate_onboarding.py --audience stakeholder --depth brief --json
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+import yaml
+from scripts.turing_io import load_config, load_experiments, load_hypotheses
+
+VALID_AUDIENCES = ["researcher", "engineer", "stakeholder"]
+VALID_DEPTHS = ["brief", "full"]
+DEFAULT_LOG = "experiments/log.jsonl"
+DEFAULT_ANNOTATIONS = "experiments/annotations.yaml"
+DEFAULT_HYPOTHESES = "hypotheses.yaml"
+
+
+def _load_yaml_list(path: str) -> list[dict]:
+    p = Path(path)
+    if not p.exists() or p.stat().st_size == 0:
+        return []
+    with open(p) as f:
+        data = yaml.safe_load(f)
+    return data if isinstance(data, list) else []
+
+
+def _load_yaml_dir(directory: str, glob: str) -> list[dict]:
+    path = Path(directory)
+    if not path.exists():
+        return []
+    items = []
+    for f in sorted(path.glob(glob)):
+        try:
+            with open(f) as fh:
+                d = yaml.safe_load(fh)
+                if d and isinstance(d, dict):
+                    items.append(d)
+        except (yaml.YAMLError, OSError):
+            continue
+    return items
+
+
+def _family_summary(exps: list[dict], metric: str, lower_is_better: bool) -> dict:
+    total = len(exps)
+    kept = [e for e in exps if e.get("status") == "kept"]
+    best_val, best_id = None, None
+    for e in kept:
+        val = e.get("metrics", {}).get(metric)
+        if val is None:
+            continue
+        if best_val is None or (lower_is_better and val < best_val) or (not lower_is_better and val > best_val):
+            best_val, best_id = val, e.get("experiment_id")
+    return {"total": total, "kept": len(kept), "keep_rate": round(len(kept) / total, 2) if total else 0,
+            "best_metric": best_val, "best_experiment": best_id}
+
+
+def _find_best(experiments: list[dict], metric: str, lower_is_better: bool) -> dict | None:
+    best, best_val = None, float("inf") if lower_is_better else float("-inf")
+    for e in experiments:
+        if e.get("status") != "kept":
+            continue
+        val = e.get("metrics", {}).get(metric)
+        if val is not None and ((lower_is_better and val < best_val) or (not lower_is_better and val > best_val)):
+            best_val, best = val, e
+    return best
+
+
+def _extract_decisions(packets: list[dict], annotations: list[dict]) -> list[dict]:
+    decisions = []
+    for pkt in packets:
+        if pkt.get("action") in ("promote", "abandon", "replicate"):
+            decisions.append({"type": "decision", "experiment": pkt.get("experiment_id", "?"),
+                              "action": pkt["action"], "reason": pkt.get("reason", ""),
+                              "date": pkt.get("timestamp", "")[:10]})
+    key_tags = {"decision", "key", "important", "milestone"}
+    for ann in annotations:
+        if set(t.lower() for t in ann.get("tags", [])) & key_tags:
+            decisions.append({"type": "annotation", "experiment": ann.get("experiment_id", "?"),
+                              "text": ann.get("text", ""), "date": ann.get("date", "")[:10]})
+    decisions.sort(key=lambda d: d.get("date", ""), reverse=True)
+    return decisions
+
+
+def _project_direction(hypotheses: list[dict], experiments: list[dict]) -> dict:
+    queued = [h for h in hypotheses if h.get("status") == "queued"]
+    promising = [h for h in hypotheses if h.get("status") == "promising"]
+    recent = experiments[-5:] if len(experiments) >= 5 else experiments
+    recent_kept = sum(1 for e in recent if e.get("status") == "kept")
+    if not experiments:
+        phase = "not_started"
+    elif not queued and not promising:
+        phase = "exhausted"
+    elif recent_kept == 0 and len(recent) >= 3:
+        phase = "plateaued"
+    elif promising:
+        phase = "promising_leads"
+    else:
+        phase = "active_exploration"
+    return {"phase": phase, "queued": queued[:5], "promising": promising[:3],
+            "n_queued": len(queued), "n_promising": len(promising)}
+
+
+def format_onboarding_report(config, experiments, families, best, decisions,
+                             direction, annotations, seeds, audience, depth,
+                             metric, lower_is_better) -> str:
+    d = "lower" if lower_is_better else "higher"
+    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
+    L = ["# Project Onboarding", "", f"*Generated {now} for audience: {audience}*", "", "---", "",
+         "## 1. What This Project Does", ""]
+    data_cfg, eval_cfg = config.get("data", {}), config.get("evaluation", {})
+    L.append(f"**Task:** {config.get('task_description', 'N/A')}")
+    L.append(f"**Dataset:** {data_cfg.get('source', 'unknown')}")
+    L.append(f"**Primary metric:** `{metric}` ({d} is better)")
+    extra = [m for m in eval_cfg.get("metrics", []) if m != metric]
+    if extra:
+        L.append(f"**Additional metrics:** {', '.join(f'`{m}`' for m in extra)}")
+    if depth == "full" and audience != "stakeholder":
+        sr = data_cfg.get("split_ratios", {})
+        if sr:
+            L.append(f"**Data splits:** {' / '.join(f'{k}: {int(v*100)}%' for k, v in sr.items())}")
+        if data_cfg.get("target_column"):
+            L.append(f"**Target column:** `{data_cfg['target_column']}`")
+    L.extend(["", "## 2. What's Been Tried", ""])
+    total, kept_n = len(experiments), sum(1 for e in experiments if e.get("status") == "kept")
+    if total == 0:
+        L.append("No experiments yet. Start with `/turing:train`.")
+    else:
+        L.append(f"**{total} experiments**, **{kept_n} kept** ({round(kept_n/total*100)}% keep rate).")
+        L.append("")
+        if best:
+            ms = ", ".join(f"{k}={v:.4f}" if isinstance(v, float) else f"{k}={v}"
+                           for k, v in best.get("metrics", {}).items())
+            L.extend([f"**Champion:** `{best.get('experiment_id','?')}` "
+                       f"({best.get('config',{}).get('model_type','?')}) — {ms}", ""])
+        L.extend(["### By Family", "", "| Family | Exps | Kept | Best | Status |",
+                   "|--------|------|------|------|--------|"])
+        for name, s in sorted(families.items()):
+            bv = f"{s['best_metric']:.4f}" if s["best_metric"] is not None else "---"
+            st = "Exhausted" if s["keep_rate"] == 0 and s["total"] >= 3 else (
+                 "Productive" if s["keep_rate"] >= 0.5 else "Mixed")
+            L.append(f"| {name} | {s['total']} | {s['kept']} | {bv} | {st} |")
+        L.append("")
+        if depth == "full" and audience in ("researcher", "engineer"):
+            for name, s in sorted(families.items()):
+                if not s["total"]:
+                    continue
+                L.append(f"#### {name}")
+                if s["best_experiment"]:
+                    L.append(f"- Best: `{s['best_experiment']}` ({metric}={s['best_metric']:.4f})")
+                L.append(f"- {s['kept']}/{s['total']} kept ({s['keep_rate']:.0%})")
+                fam_ids = {e.get("experiment_id") for e in experiments if (e.get("family") or "untagged") == name}
+                notes = [a for a in annotations if a.get("experiment_id") in fam_ids]
+                for n in notes[:3]:
+                    L.append(f"  - {n.get('text','')[:80]}")
+                L.append("")
+    L.extend(["## 3. Key Decisions", ""])
+    if not decisions:
+        L.append("No major decisions recorded yet.")
+    else:
+        lim = 5 if depth == "brief" else 15
+        for dec in decisions[:lim]:
+            if dec["type"] == "decision":
+                L.append(f"- **{dec['date']}** `{dec['experiment']}`: **{dec['action']}** — {dec['reason']}")
+            else:
+                L.append(f"- **{dec['date']}** `{dec['experiment']}`: {dec['text'][:100]}")
+        if len(decisions) > lim:
+            L.append(f"  *...and {len(decisions)-lim} more*")
+    L.extend(["", "## 4. Where We're Heading", ""])
+    phases = {"not_started": "Project has not started experiments yet.",
+              "exhausted": "All hypotheses tested. Need fresh ideas.",
+              "plateaued": "Recent experiments not improving. Consider pivoting.",
+              "promising_leads": "Promising directions identified and being pursued.",
+              "active_exploration": "Actively exploring hypothesis space."}
+    L.extend([phases.get(direction["phase"], "Unknown phase."), ""])
+    if direction["n_queued"]:
+        L.append(f"**{direction['n_queued']} hypotheses queued:**")
+        for h in direction["queued"]:
+            p = " (HIGH)" if h.get("priority") == "high" else ""
+            L.append(f"- {h.get('id','?')}: {h.get('description','?')}{p}")
+        L.append("")
+    if direction["n_promising"]:
+        L.append(f"**{direction['n_promising']} promising lead(s):**")
+        for h in direction["promising"]:
+            L.append(f"- {h.get('id','?')}: {h.get('description','?')}")
+        L.append("")
+    sensitive = [s for s in seeds if s.get("seed_sensitive")]
+    if sensitive and audience != "stakeholder":
+        L.append("**Seed sensitivity warnings:**")
+        for s in sensitive:
+            L.append(f"- `{s.get('experiment_id','?')}`: CV={s.get('cv_percent',0):.1f}%")
+        L.append("")
+    L.extend(["## 5. How to Get Started", ""])
+    cmds = {"researcher": [
+        "1. Read `config.yaml` for task and evaluation setup",
+        "2. `/turing:status` — current experiment state",
+        "3. `/turing:brief` — full research intelligence report",
+        "4. Review `hypotheses.yaml` for queued ideas",
+        "5. `/turing:try \"your hypothesis\"` — inject ideas",
+        "6. `/turing:train` — run next experiment",
+    ], "engineer": [
+        "1. `pip install -r requirements.txt`",
+        "2. Review `config.yaml` for data paths",
+        "3. `/turing:status` — where things stand",
+        "4. Check `train.py` for current model",
+        "5. `/turing:train` — execute experiments",
+    ], "stakeholder": [
+        "1. `/turing:brief` — high-level summary",
+        "2. Check champion performance above",
+        "3. Review 'Where We're Heading' for next steps",
+    ]}
+    L.extend(cmds.get(audience, []))
+    L.extend(["", "---", f"*Generated by `/turing:onboard` — {audience}, {depth}*"])
+    return "\n".join(L)
+
+
+def save_onboarding_report(content: str, path: str = "ONBOARDING.md") -> Path:
+    p = Path(path)
+    p.parent.mkdir(parents=True, exist_ok=True)
+    p.write_text(content)
+    return p
+
+
+def generate_onboarding(config_path="config.yaml", log_path=DEFAULT_LOG,
+                         hypotheses_path=DEFAULT_HYPOTHESES,
+                         annotations_path=DEFAULT_ANNOTATIONS,
+                         audience="researcher", depth="full") -> dict:
+    """Generate full onboarding report. Returns dict with report and metadata."""
+    config = load_config(config_path)
+    metric = config.get("evaluation", {}).get("primary_metric", "accuracy")
+    lower = config.get("evaluation", {}).get("lower_is_better", False)
+    experiments = load_experiments(log_path)
+    hypotheses = load_hypotheses(hypotheses_path)
+    annotations = _load_yaml_list(annotations_path)
+    packets = _load_yaml_dir("experiments/decisions", "*.yaml")
+    seeds = _load_yaml_dir("experiments/seed_studies", "*-seeds.yaml")
+    fam_groups = {}
+    for e in experiments:
+        fam_groups.setdefault(e.get("family") or "untagged", []).append(e)
+    families = {n: _family_summary(exps, metric, lower) for n, exps in fam_groups.items()}
+    best = _find_best(experiments, metric, lower)
+    decisions = _extract_decisions(packets, annotations)
+    direction = _project_direction(hypotheses, experiments)
+    report = format_onboarding_report(config, experiments, families, best, decisions,
+                                       direction, annotations, seeds, audience, depth, metric, lower)
+    return {"timestamp": datetime.now(timezone.utc).isoformat(), "audience": audience,
+            "depth": depth, "total_experiments": len(experiments),
+            "project_phase": direction["phase"], "report": report}
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Generate project onboarding for new collaborators")
+    parser.add_argument("--config", default="config.yaml")
+    parser.add_argument("--log", default=DEFAULT_LOG)
+    parser.add_argument("--hypotheses", default=DEFAULT_HYPOTHESES)
+    parser.add_argument("--annotations", default=DEFAULT_ANNOTATIONS)
+    parser.add_argument("--audience", default="researcher", choices=VALID_AUDIENCES)
+    parser.add_argument("--depth", default="full", choices=VALID_DEPTHS)
+    parser.add_argument("--output", default="ONBOARDING.md")
+    parser.add_argument("--json", action="store_true", help="Output raw JSON")
+    args = parser.parse_args()
+    result = generate_onboarding(args.config, args.log, args.hypotheses,
+                                  args.annotations, args.audience, args.depth)
+    if args.json:
+        print(json.dumps(result, indent=2, default=str))
+    else:
+        saved = save_onboarding_report(result["report"], args.output)
+        print(result["report"])
+        print(f"\nSaved to {saved}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

package/templates/scripts/package_experiments.py

@@ -0,0 +1,285 @@
+#!/usr/bin/env python3
+"""Package experiments into portable archive for sharing.
+
+Collects config, metrics, seed studies, annotations, decision packets
+per experiment. Generates manifest.yaml and README.md inside the
+package directory. Does NOT create tar.gz -- just organizes files.
+
+Usage:
+    python scripts/package_experiments.py
+    python scripts/package_experiments.py --experiments exp-042,exp-043
+    python scripts/package_experiments.py --include model,data-hash,figures,code
+    python scripts/package_experiments.py --json
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import shutil
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+import yaml
+from scripts.turing_io import load_config, load_experiments
+
+DEFAULT_LOG = "experiments/log.jsonl"
+DEFAULT_OUTPUT = "exports/packages"
+VALID_INCLUDES = ["model", "data-hash", "figures", "code"]
+
+
+def _load_yaml_list(path: str) -> list[dict]:
+    p = Path(path)
+    if not p.exists() or p.stat().st_size == 0:
+        return []
+    with open(p) as f:
+        data = yaml.safe_load(f)
+    return data if isinstance(data, list) else []
+
+
+def _load_yaml_file(path: Path) -> dict | None:
+    if not path.exists():
+        return None
+    try:
+        with open(path) as f:
+            d = yaml.safe_load(f)
+        return d if isinstance(d, dict) else None
+    except (yaml.YAMLError, OSError):
+        return None
+
+
+def _file_hash(filepath: str) -> str | None:
+    p = Path(filepath)
+    if not p.exists():
+        return None
+    h = hashlib.sha256()
+    with open(p, "rb") as f:
+        for chunk in iter(lambda: f.read(8192), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def collect_experiment_artifacts(exp: dict, includes: list[str]) -> dict:
+    """Collect all artifacts for a single experiment."""
+    eid = exp.get("experiment_id", "unknown")
+    art: dict = {"experiment_id": eid, "status": exp.get("status", "unknown"),
+                 "metrics": exp.get("metrics", {}), "config": exp.get("config", {}),
+                 "description": exp.get("description", ""), "timestamp": exp.get("timestamp", ""),
+                 "family": exp.get("family")}
+    # Seed study
+    seed = _load_yaml_file(Path(f"experiments/seed_studies/{eid}-seeds.yaml"))
+    if seed:
+        art["seed_study"] = {"mean": seed.get("mean"), "std": seed.get("std"),
+                              "cv_percent": seed.get("cv_percent"),
+                              "seed_sensitive": seed.get("seed_sensitive", False)}
+    # Decision packet
+    dec = _load_yaml_file(Path(f"experiments/decisions/{eid}-decision.yaml"))
+    if dec:
+        art["decision"] = {"action": dec.get("action"), "reason": dec.get("reason", "")}
+    # Ablation
+    abl = _load_yaml_file(Path(f"experiments/ablations/{eid}-ablation.yaml"))
+    if abl:
+        art["ablation"] = {"metric": abl.get("metric"),
+                            "n_ablations": len(abl.get("results", []))}
+    # Reproduction
+    repro = _load_yaml_file(Path(f"experiments/reproductions/{eid}-repro.yaml"))
+    if repro:
+        art["reproduction"] = {"verdict": repro.get("verdict"), "reason": repro.get("reason", "")}
+    # Optional includes
+    if "model" in includes:
+        for pat in [f"models/{eid}", f"models/{eid}.*", f"checkpoints/{eid}/*"]:
+            matches = list(Path(".").glob(pat))
+            if matches:
+                art["model_path"] = str(matches[0])
+                break
+    if "data-hash" in includes:
+        dp = exp.get("config", {}).get("data", {}).get("path")
+        if dp:
+            h = _file_hash(dp)
+            if h:
+                art["data_hash"] = h
+    if "figures" in includes:
+        fig_dir = Path(f"experiments/figures/{eid}")
+        art["figures"] = [str(f) for f in fig_dir.glob("*") if f.is_file()] if fig_dir.exists() else []
+    if "code" in includes:
+        art["train_py_hash"] = _file_hash("train.py")
+        snap = Path(f"experiments/code/{eid}")
+        if snap.exists():
+            art["code_snapshot_path"] = str(snap)
+    return art
+
+
+def build_manifest(name: str, config: dict, artifacts: list[dict], includes: list[str]) -> dict:
+    eval_cfg = config.get("evaluation", {})
+    return {
+        "package": {"name": name, "created": datetime.now(timezone.utc).isoformat(),
+                     "generator": "turing:share", "version": "1.0"},
+        "project": {"task": config.get("task_description", ""),
+                     "primary_metric": eval_cfg.get("primary_metric", "accuracy"),
+                     "lower_is_better": eval_cfg.get("lower_is_better", False)},
+        "contents": {"experiments": len(artifacts), "includes": includes,
+                      "has_seed_studies": any(a.get("seed_study") for a in artifacts),
+                      "has_decisions": any(a.get("decision") for a in artifacts)},
+        "experiments": [{"id": a["experiment_id"], "status": a["status"],
+                          "family": a.get("family"), "metrics": a["metrics"]} for a in artifacts],
+    }
+
+
+def build_package_readme(config: dict, artifacts: list[dict], manifest: dict) -> str:
+    metric = manifest["project"]["primary_metric"]
+    d = "lower" if manifest["project"]["lower_is_better"] else "higher"
+    L = [f"# Experiment Package: {manifest['package']['name']}", "",
+         f"*Packaged {manifest['package']['created'][:19]} UTC*", "",
+         "## Project", "", f"- **Task:** {config.get('task_description', 'N/A')}",
+         f"- **Primary metric:** `{metric}` ({d} is better)", "", "## Experiments", "",
+         f"| ID | Status | Family | {metric} |",
+         f"|----|--------|--------|{'---'*max(len(metric)//3,1)}--|"]
+    for a in artifacts:
+        v = a.get("metrics", {}).get(metric)
+        vs = f"{v:.4f}" if isinstance(v, (int, float)) else "---"
+        L.append(f"| {a['experiment_id']} | {a['status']} | {a.get('family','---')} | {vs} |")
+    seeds = [a for a in artifacts if a.get("seed_study")]
+    if seeds:
+        L.extend(["", "## Seed Studies", ""])
+        for a in seeds:
+            s = a["seed_study"]
+            tag = "SEED-SENSITIVE" if s["seed_sensitive"] else "stable"
+            L.append(f"- `{a['experiment_id']}`: mean={s['mean']:.4f} +/- {s['std']:.4f} [{tag}]")
+    decs = [a for a in artifacts if a.get("decision")]
+    if decs:
+        L.extend(["", "## Decisions", ""])
+        for a in decs:
+            L.append(f"- `{a['experiment_id']}`: **{a['decision']['action']}** — {a['decision']['reason']}")
+    L.extend(["", "## Files", "", "- `manifest.yaml` — Machine-readable manifest",
+              "- `README.md` — This file", "- `experiments/` — Per-experiment artifacts",
+              "- `config.yaml` — Project config snapshot", "", "---",
+              "*Generated by `/turing:share`*"])
+    return "\n".join(L)
+
+
+def write_package(pkg_dir: Path, config: dict, artifacts: list[dict],
+                   manifest: dict, readme: str, includes: list[str]) -> None:
+    """Write all package files to the directory."""
+    pkg_dir.mkdir(parents=True, exist_ok=True)
+    with open(pkg_dir / "manifest.yaml", "w") as f:
+        yaml.dump(manifest, f, default_flow_style=False, sort_keys=False)
+    (pkg_dir / "README.md").write_text(readme)
+    with open(pkg_dir / "config.yaml", "w") as f:
+        yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+    exp_dir = pkg_dir / "experiments"
+    exp_dir.mkdir(exist_ok=True)
+    for art in artifacts:
+        sub = exp_dir / art["experiment_id"]
+        sub.mkdir(exist_ok=True)
+        with open(sub / "artifact.yaml", "w") as f:
+            yaml.dump(art, f, default_flow_style=False, sort_keys=False)
+        if "figures" in includes and art.get("figures"):
+            fd = sub / "figures"
+            fd.mkdir(exist_ok=True)
+            for fp in art["figures"]:
+                src = Path(fp)
+                if src.exists():
+                    shutil.copy2(src, fd / src.name)
+        if "code" in includes and art.get("code_snapshot_path"):
+            cs = Path(art["code_snapshot_path"])
+            if cs.exists() and cs.is_dir():
+                shutil.copytree(cs, sub / "code", dirs_exist_ok=True)
+        if "model" in includes and art.get("model_path"):
+            ms = Path(art["model_path"])
+            if ms.exists():
+                md = sub / "model"
+                md.mkdir(exist_ok=True)
+                if ms.is_dir():
+                    shutil.copytree(ms, md, dirs_exist_ok=True)
+                else:
+                    shutil.copy2(ms, md / ms.name)
+
+
+def save_package_report(result: dict, pkg_dir: Path) -> Path:
+    rp = pkg_dir / "package-report.yaml"
+    with open(rp, "w") as f:
+        yaml.dump({"timestamp": result["timestamp"], "package_name": result["package_name"],
+                    "package_dir": str(result["package_dir"]),
+                    "experiments_packaged": result["experiments_packaged"],
+                    "includes": result["includes"]}, f, default_flow_style=False, sort_keys=False)
+    return rp
+
+
+def format_package_report(result: dict) -> str:
+    L = ["# Package Summary", "",
+         f"- **Package:** {result['package_name']}",
+         f"- **Location:** `{result['package_dir']}`",
+         f"- **Experiments:** {result['experiments_packaged']}",
+         f"- **Includes:** {', '.join(result['includes']) or 'metrics only'}", "", "## Contents", ""]
+    for a in result.get("artifacts", []):
+        extras = [k for k in ("seed_study", "decision", "ablation", "reproduction") if a.get(k)]
+        es = f" [{', '.join(extras)}]" if extras else ""
+        L.append(f"- `{a['experiment_id']}` ({a['status']}){es}")
+    return "\n".join(L)
+
+
+def package_experiments(experiment_ids=None, includes=None, config_path="config.yaml",
+                         log_path=DEFAULT_LOG, output_dir=DEFAULT_OUTPUT) -> dict:
+    """Package experiments into a portable directory."""
+    includes = includes or []
+    config = load_config(config_path)
+    experiments = load_experiments(log_path)
+    annotations = _load_yaml_list("experiments/annotations.yaml")
+    if experiment_ids:
+        selected = [e for e in experiments if e.get("experiment_id") in experiment_ids]
+        if not selected:
+            return {"error": f"No matching experiments for: {experiment_ids}"}
+    else:
+        selected = [e for e in experiments if e.get("status") == "kept"]
+        if not selected:
+            return {"error": "No kept experiments to package."}
+    artifacts = [collect_experiment_artifacts(e, includes) for e in selected]
+    for art in artifacts:
+        eid = art["experiment_id"]
+        anns = [a for a in annotations if a.get("experiment_id") == eid]
+        if anns:
+            art["annotations"] = anns
+    ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
+    pkg_name = f"package-{len(artifacts)}exp-{ts}"
+    pkg_dir = Path(output_dir) / pkg_name
+    manifest = build_manifest(pkg_name, config, artifacts, includes)
+    readme = build_package_readme(config, artifacts, manifest)
+    write_package(pkg_dir, config, artifacts, manifest, readme, includes)
+    result = {"timestamp": datetime.now(timezone.utc).isoformat(), "package_name": pkg_name,
+              "package_dir": str(pkg_dir), "experiments_packaged": len(artifacts),
+              "includes": includes, "artifacts": artifacts}
+    save_package_report(result, pkg_dir)
+    return result
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Package experiments into portable archive")
+    parser.add_argument("--experiments", default=None, help="Comma-separated experiment IDs")
+    parser.add_argument("--include", default=None, help="Extras: model,data-hash,figures,code")
+    parser.add_argument("--config", default="config.yaml")
+    parser.add_argument("--log", default=DEFAULT_LOG)
+    parser.add_argument("--output", default=DEFAULT_OUTPUT)
+    parser.add_argument("--json", action="store_true", help="Output raw JSON")
+    args = parser.parse_args()
+    exp_ids = [e.strip() for e in args.experiments.split(",")] if args.experiments else None
+    includes = []
+    if args.include:
+        includes = [i.strip() for i in args.include.split(",")]
+        bad = [i for i in includes if i not in VALID_INCLUDES]
+        if bad:
+            print(f"ERROR: Invalid include(s): {bad}. Valid: {VALID_INCLUDES}", file=sys.stderr)
+            sys.exit(1)
+    result = package_experiments(exp_ids, includes, args.config, args.log, args.output)
+    if "error" in result:
+        print(f"ERROR: {result['error']}", file=sys.stderr)
+        sys.exit(1)
+    if args.json:
+        print(json.dumps(result, indent=2, default=str))
+    else:
+        print(format_package_report(result))
+        print(f"\nPackage saved to: {result['package_dir']}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

package/templates/scripts/scaffold.py

@@ -140,6 +140,9 @@ TEMPLATE_DIRS = {
         "citation_manager.py",
         "generate_figures.py",
         "generate_changelog.py",
+        "generate_onboarding.py",
+        "package_experiments.py",
+        "simulate_review.py",
     ],
     "tests": ["__init__.py", "conftest.py"],
 }
@@ -189,6 +192,8 @@ DIRECTORIES_TO_CREATE = [
     "experiments/replays",
     "experiments/citations",
     "paper/figures",
+    "exports/packages",
+    "experiments/reviews",
     "experiments/logs",
     "models/best",
     "models/archive",

package/templates/scripts/simulate_review.py

@@ -0,0 +1,342 @@
+#!/usr/bin/env python3
+"""Simulated peer review for ML experiment campaigns.
+
+Checks for: missing baselines, missing error bars, missing ablation,
+overclaimed results, missing SOTA comparison, calibration, computational
+cost. Generates structured review with strengths/weaknesses/questions,
+each weakness linked to a /turing: fix command. Scores 1-10.
+
+Usage:
+    python scripts/simulate_review.py
+    python scripts/simulate_review.py --venue neurips --harsh
+    python scripts/simulate_review.py --venue icml --json
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+import yaml
+from scripts.turing_io import load_config, load_experiments
+
+DEFAULT_LOG = "experiments/log.jsonl"
+VALID_VENUES = ["neurips", "icml", "general"]
+SEVERITY_WEIGHTS = {"critical": 3.0, "major": 2.0, "minor": 1.0, "nitpick": 0.3}
+
+
+def _load_yaml_dir(directory: str, glob: str) -> list[dict]:
+    path = Path(directory)
+    if not path.exists():
+        return []
+    items = []
+    for f in sorted(path.glob(glob)):
+        try:
+            with open(f) as fh:
+                d = yaml.safe_load(fh)
+                if d and isinstance(d, dict):
+                    items.append(d)
+        except (yaml.YAMLError, OSError):
+            continue
+    return items
+
+
+def _load_yaml_list(path: str) -> list[dict]:
+    p = Path(path)
+    if not p.exists() or p.stat().st_size == 0:
+        return []
+    with open(p) as f:
+        data = yaml.safe_load(f)
+    return data if isinstance(data, list) else []
+
+
+def _w(id, sev, title, detail, fix, venues=None):
+    """Shorthand weakness constructor."""
+    return {"id": id, "severity": sev, "title": title, "detail": detail,
+            "fix_command": fix, "venue_relevance": venues or ["neurips", "icml", "general"]}
+
+
+# --- Review checks ---
+
+def check_baselines(experiments, config):
+    types = {e.get("config", {}).get("model_type", "") for e in experiments if e.get("status") == "kept"}
+    baselines = {"logistic_regression", "linear_regression", "dummy", "majority_class", "random", "baseline"}
+    if types and not (types & baselines):
+        return _w("missing-baselines", "major", "No simple baseline comparison",
+                   f"Model types: {', '.join(sorted(types))}. No simple baseline to calibrate expectations.",
+                   '/turing:try "Add logistic regression baseline"')
+
+def check_error_bars(experiments, seeds):
+    kept = [e for e in experiments if e.get("status") == "kept"]
+    if not kept:
+        return None
+    studied = {s.get("experiment_id") for s in seeds}
+    unstudied = [e for e in kept if e.get("experiment_id") not in studied]
+    if len(unstudied) == len(kept):
+        return _w("no-error-bars", "critical", "No error bars on any result",
+                   f"{len(kept)} kept experiment(s) with no seed studies. Single-seed results not publishable.",
+                   "/turing:seed")
+    elif unstudied:
+        ids = ", ".join(e.get("experiment_id", "?") for e in unstudied[:5])
+        return _w("partial-error-bars", "minor", "Some experiments lack error bars",
+                   f"{len(unstudied)}/{len(kept)} lack seed studies: {ids}.", "/turing:seed")
+
+def check_ablation(experiments, ablations):
+    if len([e for e in experiments if e.get("status") == "kept"]) >= 2 and not ablations:
+        return _w("no-ablation", "major", "No ablation study",
+                   "No ablation studies found. Component contributions unclear.", "/turing:ablate")
+
+def check_overclaimed(experiments, seeds, metric, lower_is_better):
+    sensitive = [s for s in seeds if s.get("seed_sensitive")]
+    if not sensitive:
+        return None
+    details = "; ".join(f"{s.get('experiment_id','?')}: CV={s.get('cv_percent',0):.1f}%" for s in sensitive)
+    return _w("overclaimed-results", "major", "Seed-sensitive results may be overclaimed",
+               f"{len(sensitive)} experiment(s) show high seed sensitivity: {details}. "
+               "Report mean +/- std instead of point estimates.", "/turing:seed")
+
+def check_sota(experiments, config, annotations):
+    kw = {"sota", "state-of-the-art", "benchmark", "leaderboard", "published"}
+    for ann in annotations:
+        text = ann.get("text", "").lower()
+        if any(k in text for k in kw) or any(k in [t.lower() for t in ann.get("tags", [])] for k in kw):
+            return None
+    if config.get("evaluation", {}).get("reference_score") or config.get("evaluation", {}).get("sota_score"):
+        return None
+    return _w("no-sota-comparison", "minor", "No SOTA or external benchmark comparison",
+               "No reference to published results. Add reference score or annotate with SOTA values.",
+               '/turing:try "Add SOTA comparison from literature"', ["neurips", "icml"])
+
+def check_calibration(cal_results, experiments):
+    kept = [e for e in experiments if e.get("status") == "kept"]
+    if not kept:
+        return None
+    if not cal_results:
+        return _w("no-calibration", "minor", "No calibration analysis",
+                   "Model calibration not assessed. Reviewers expect ECE or reliability diagrams.",
+                   '/turing:try "Add calibration analysis (ECE)"', ["neurips", "icml"])
+    poor = [r for r in cal_results if r.get("ece", 0) > 0.1]
+    if poor:
+        return _w("poor-calibration", "minor", "Model poorly calibrated",
+                   f"{len(poor)} model(s) have ECE > 0.1. Consider temperature scaling.",
+                   '/turing:try "Apply temperature scaling"')
+
+def check_compute_cost(experiments):
+    kept = [e for e in experiments if e.get("status") == "kept"]
+    if not kept:
+        return None
+    has_time = any(e.get("metrics", {}).get("train_seconds") is not None for e in kept)
+    has_env = any(e.get("environment") for e in kept)
+    issues = []
+    if not has_time:
+        issues.append("No training time reported")
+    if not has_env:
+        issues.append("No hardware info recorded")
+    if issues:
+        return _w("no-compute-cost", "major" if not has_time else "minor",
+                   "Computational cost not reported", "; ".join(issues) + ". "
+                   "Reporting compute cost is expected at all major venues.",
+                   '/turing:try "Profile training and report compute cost"')
+
+def check_diversity(experiments):
+    types = {e.get("config", {}).get("model_type", "") for e in experiments if e.get("status") == "kept"}
+    kept_n = sum(1 for e in experiments if e.get("status") == "kept")
+    if len(types) == 1 and kept_n >= 3:
+        return _w("low-diversity", "minor", "Only one model family explored",
+                   f"All {kept_n} kept experiments use {list(types)[0]}. Alternatives not explored.",
+                   '/turing:try "Explore alternative model architecture"')
+
+def check_leakage(experiments, annotations):
+    kw = {"leakage", "leak", "contamination", "suspicious", "too high", "too good"}
+    flagged = [a for a in annotations
+               if any(k in a.get("text", "").lower() for k in kw) or "leakage" in [t.lower() for t in a.get("tags", [])]]
+    if flagged:
+        return _w("leakage-concern", "critical", "Data leakage flagged in annotations",
+                   f"{len(flagged)} annotation(s) mention potential leakage. Must investigate before submission.",
+                   '/turing:try "Investigate and rule out data leakage"')
+
+def check_reproducibility(experiments, config):
+    issues = []
+    if config.get("data", {}).get("random_state") is None:
+        issues.append("No random state in config")
+    rd = Path("experiments/reproductions")
+    if len(experiments) >= 5 and not (rd.exists() and any(rd.glob("*.yaml"))):
+        issues.append("No reproduction checks run")
+    if issues:
+        return _w("reproducibility-gaps", "minor", "Reproducibility not fully verified",
+                   "; ".join(issues) + ".", "/turing:reproduce")
+
+
+# --- Strengths & questions ---
+
+def identify_strengths(experiments, seeds, ablations, config, metric, lower_is_better):
+    S = []
+    kept = [e for e in experiments if e.get("status") == "kept"]
+    types = set(e.get("config", {}).get("model_type", "") for e in kept)
+    if len(kept) >= 5:
+        S.append(f"Thorough experimentation: {len(kept)} successful experiments across {len(types)} type(s).")
+    stable = [s for s in seeds if not s.get("seed_sensitive")]
+    if stable:
+        S.append(f"Seed studies: {len(stable)}/{len(seeds)} experiments show stable results.")
+    if ablations:
+        S.append(f"Ablation analysis provided ({len(ablations)} study/ies).")
+    families = set(e.get("family") for e in kept if e.get("family"))
+    if len(families) >= 3:
+        S.append(f"Systematic exploration of {len(families)} research directions.")
+    if len(experiments) >= 5 and len(kept) / len(experiments) >= 0.4:
+        S.append(f"High experiment efficiency: {len(kept)/len(experiments):.0%} keep rate.")
+    return S or ["Experiments have been initiated on this problem."]
+
+
+def generate_questions(weaknesses, experiments, config, venue):
+    Q, wids = [], {w["id"] for w in weaknesses}
+    qmap = {"missing-baselines": "How does performance compare to a simple baseline?",
+            "no-error-bars": "Can you provide confidence intervals over multiple seeds?",
+            "overclaimed-results": "Can you provide confidence intervals over multiple seeds?",
+            "no-ablation": "What is the contribution of each component?",
+            "no-sota-comparison": "How do results compare to published state-of-the-art?",
+            "no-compute-cost": "What are the computational requirements (GPU hours)?"}
+    for wid, q in qmap.items():
+        if wid in wids:
+            Q.append(q)
+    if venue == "neurips":
+        Q.append("What is the broader impact? Are there negative societal implications?")
+    elif venue == "icml":
+        Q.append("Is there theoretical justification, or is this purely empirical?")
+    return Q
+
+
+def compute_score(strengths, weaknesses, harsh):
+    score = 6.0 + min(len(strengths) * 0.4, 2.0)
+    for w in weaknesses:
+        score -= SEVERITY_WEIGHTS.get(w["severity"], 1.0)
+    if harsh:
+        score -= 1.0
+    return max(1, min(10, round(score)))
+
+
+# --- Formatting ---
+
+def format_review_report(strengths, weaknesses, questions, score, venue, harsh,
+                          config, experiments, metric):
+    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
+    kept = [e for e in experiments if e.get("status") == "kept"]
+    types = set(e.get("config", {}).get("model_type", "") for e in kept)
+    labels = {1: "Strong Reject", 2: "Reject", 3: "Reject", 4: "Weak Reject",
+              5: "Borderline Reject", 6: "Borderline Accept", 7: "Weak Accept",
+              8: "Accept", 9: "Strong Accept", 10: "Strong Accept"}
+    crit = sum(1 for w in weaknesses if w["severity"] == "critical")
+    maj = sum(1 for w in weaknesses if w["severity"] == "major")
+    mn = sum(1 for w in weaknesses if w["severity"] == "minor")
+    L = ["# Simulated Peer Review", "",
+         f"*Generated {now}*",
+         f"*Venue: {venue.upper()} | Mode: {'HARSH' if harsh else 'Standard'} | "
+         f"Score: {score}/10 ({labels.get(score, '?')})*", "", "---", "",
+         "## Summary", "",
+         f"This work presents experiments on {config.get('task_description', 'the given task')} "
+         f"with {len(kept)} successful experiment(s) across {len(types)} model type(s). "
+         f"Primary metric: `{metric}`.", "",
+         "## Score", "", f"**{score}/10** — {labels.get(score, '?')}", "",
+         f"- Strengths: {len(strengths)}",
+         f"- Weaknesses: {len(weaknesses)} ({crit} critical, {maj} major, {mn} minor)", "",
+         "## Strengths", ""]
+    for i, s in enumerate(strengths, 1):
+        L.append(f"**S{i}.** {s}")
+    L.extend(["", "## Weaknesses", ""])
+    for i, w in enumerate(weaknesses, 1):
+        L.extend([f"**W{i}. [{w['severity'].upper()}] {w['title']}**", "",
+                   w["detail"], "", f"*Fix:* `{w['fix_command']}`", ""])
+    if not weaknesses:
+        L.extend(["No significant weaknesses identified.", ""])
+    L.extend(["## Questions for Authors", ""])
+    for i, q in enumerate(questions, 1):
+        L.append(f"**Q{i}.** {q}")
+    critical_major = [w for w in weaknesses if w["severity"] in ("critical", "major")]
+    if critical_major:
+        L.extend(["", "## Recommended Action Plan", "", "Address before submission:", ""])
+        for p, w in enumerate(critical_major, 1):
+            L.append(f"{p}. **[{w['severity'].upper()}]** {w['title']}: `{w['fix_command']}`")
+    L.extend(["", "## Verdict", ""])
+    if score >= 7:
+        L.append("Approaching publication quality. Address minor issues and consider submission.")
+    elif score >= 5:
+        L.append("Borderline. Significant improvements needed. Follow the action plan.")
+    else:
+        L.append("Not ready. Major methodology gaps. Focus on critical and major weaknesses.")
+    L.extend(["", "---", "*Simulated review by `/turing:review` — not a substitute for actual peer review.*"])
+    return "\n".join(L)
+
+
+def save_review_report(result: dict, output_dir="experiments/reviews") -> Path:
+    p = Path(output_dir)
+    p.mkdir(parents=True, exist_ok=True)
+    ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
+    out = p / f"review-{ts}.yaml"
+    with open(out, "w") as f:
+        yaml.dump({"timestamp": result["timestamp"], "venue": result["venue"],
+                    "harsh": result["harsh"], "score": result["score"],
+                    "weaknesses": [{"id": w["id"], "severity": w["severity"],
+                                     "title": w["title"], "fix_command": w["fix_command"]}
+                                    for w in result["weaknesses"]]},
+                   f, default_flow_style=False, sort_keys=False)
+    return out
+
+
+# --- Orchestration ---
+
+def simulate_review(venue="general", harsh=False, config_path="config.yaml",
+                     log_path=DEFAULT_LOG) -> dict:
+    """Run full simulated review pipeline."""
+    config = load_config(config_path)
+    metric = config.get("evaluation", {}).get("primary_metric", "accuracy")
+    lower = config.get("evaluation", {}).get("lower_is_better", False)
+    experiments = load_experiments(log_path)
+    if not experiments:
+        return {"error": "No experiments found. Run /turing:train first.",
+                "timestamp": datetime.now(timezone.utc).isoformat()}
+    seeds = _load_yaml_dir("experiments/seed_studies", "*-seeds.yaml")
+    ablations = _load_yaml_dir("experiments/ablations", "*-ablation.yaml")
+    cal = _load_yaml_dir("experiments/calibration", "*.yaml")
+    annotations = _load_yaml_list("experiments/annotations.yaml")
+    checks = [check_baselines(experiments, config), check_error_bars(experiments, seeds),
+              check_ablation(experiments, ablations), check_overclaimed(experiments, seeds, metric, lower),
+              check_sota(experiments, config, annotations), check_calibration(cal, experiments),
+              check_compute_cost(experiments), check_diversity(experiments),
+              check_leakage(experiments, annotations), check_reproducibility(experiments, config)]
+    weaknesses = [c for c in checks if c and venue in c.get("venue_relevance", ["general"])]
+    sev_order = {"critical": 0, "major": 1, "minor": 2, "nitpick": 3}
+    weaknesses.sort(key=lambda w: sev_order.get(w["severity"], 9))
+    strengths = identify_strengths(experiments, seeds, ablations, config, metric, lower)
+    questions = generate_questions(weaknesses, experiments, config, venue)
+    score = compute_score(strengths, weaknesses, harsh)
+    report = format_review_report(strengths, weaknesses, questions, score, venue, harsh,
+                                   config, experiments, metric)
+    return {"timestamp": datetime.now(timezone.utc).isoformat(), "venue": venue, "harsh": harsh,
+            "score": score, "strengths": strengths, "weaknesses": weaknesses,
+            "questions": questions, "report": report}
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Simulate peer review of experiment campaign")
+    parser.add_argument("--venue", default="general", choices=VALID_VENUES)
+    parser.add_argument("--harsh", action="store_true", help="Stricter review criteria")
+    parser.add_argument("--config", default="config.yaml")
+    parser.add_argument("--log", default=DEFAULT_LOG)
+    parser.add_argument("--json", action="store_true", help="Output raw JSON")
+    args = parser.parse_args()
+    result = simulate_review(args.venue, args.harsh, args.config, args.log)
+    if "error" in result and "report" not in result:
+        print(f"ERROR: {result['error']}", file=sys.stderr)
+        sys.exit(1)
+    if args.json:
+        print(json.dumps(result, indent=2, default=str))
+    else:
+        print(result["report"])
+        saved = save_review_report(result)
+        print(f"\nReview saved to {saved}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()