claude-turing 1.2.0 → 1.3.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "1.2.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 17 commands, 2 specialized agents, tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "1.3.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 19 commands, 2 specialized agents, statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -323,6 +323,8 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | Command | What it does |
 |---------|-------------|
 | `/turing:validate [--auto]` | Check metric stability — auto-configure multi-run if noisy |
+| `/turing:seed [N] [--quick]` | Multi-seed study — mean/std/CI, flag seed-sensitive results |
+| `/turing:reproduce <exp-id>` | Reproducibility verification — re-run and check tolerance |
 | `/turing:card` | Generate a model card — performance, limitations, intended use, artifact contract |
 | `/turing:logbook` | Generate HTML experiment logbook |
 | `/turing:report` | Generate research report |
@@ -392,6 +394,32 @@ After N experiments with no meaningful improvement, the agent stops and reports
 
 For noisy metrics, `/turing:validate` runs the pipeline multiple times and measures variance. If the coefficient of variation exceeds 5%, it auto-configures multi-run evaluation so the agent can't be rewarded for lucky single runs.
 
+## Statistical Rigor
+
+> *"Stop publishing lucky seeds. Start publishing distributions."*
+
+Before claiming a result, run a seed study:
+
+```
+/turing:seed              # 5 seeds on best experiment
+/turing:seed --quick      # 3 seeds for fast check
+/turing:seed 10           # 10 seeds for thorough study
+```
+
+This runs the same experiment across multiple random seeds and reports mean +/- std with 95% confidence intervals. If the coefficient of variation exceeds 5%, the result is flagged as **seed-sensitive** — meaning you should report the distribution, not a single number.
+
+To verify an experiment can be reproduced:
+
+```
+/turing:reproduce exp-042                  # Default: 3 runs, 2% tolerance
+/turing:reproduce exp-042 --strict         # Exact match required
+/turing:reproduce exp-042 --tolerance 0.05 # Custom tolerance
+```
+
+This re-runs the experiment from the logged config and checks that metrics fall within tolerance. It also detects environment drift — if library versions have changed since the original run, you'll know before a reviewer tells you.
+
+Seed study results automatically appear in `/turing:brief` and `/turing:card`.
+
 ## Tree-Search Hypothesis Exploration
 
 > *"The learned coin-flipper weaves through the quadrillion-coin room with a preternatural air."*
@@ -486,11 +514,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-17 commands, 2 agents, 8 config files, 31 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, 379 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+19 commands, 2 agents, 8 config files, 34 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor (seed studies + reproducibility), 407 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              16 skill files (core + taste-leverage + reporting + exploration)
+├── commands/              18 skill files (core + taste-leverage + reporting + exploration + statistical rigor)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/reproduce.md

@@ -0,0 +1,48 @@
+---
+name: reproduce
+description: Verify reproducibility of a specific experiment by re-running from logged config and checking metrics fall within tolerance.
+disable-model-invocation: true
+argument-hint: "<exp-id> [--tolerance 0.02] [--strict] [--runs 3]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Verify that a logged experiment can be reproduced with consistent results.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First argument is the experiment ID (required), e.g. `exp-042`
+   - `--tolerance 0.02` sets the relative tolerance (default 2%)
+   - `--strict` requires exact float match (1e-6), overrides tolerance
+   - `--runs 3` sets number of reproduction runs (default 3, 1 for strict)
+
+3. **Run reproducibility verification:**
+   ```bash
+   python scripts/reproduce_experiment.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **reproducible:** metrics match exactly (deterministic algorithm)
+   - **approximately_reproducible:** metrics within tolerance or original falls in 95% CI
+   - **not_reproducible:** metrics outside tolerance and CI
+   - **environment_changed:** metrics diverge AND library versions differ
+   - Show environment diff if present (Python version, package versions)
+
+5. **Saved output:** report written to `experiments/reproductions/exp-NNN-repro.yaml`
+
+6. **If experiment ID not found:** list available experiment IDs from `experiments/log.jsonl`
+
+7. **If no training pipeline exists:** suggest `/turing:init` first.
+
+## Examples
+
+```
+/turing:reproduce exp-042                 # Default: 3 runs, 2% tolerance
+/turing:reproduce exp-042 --strict        # Exact match required
+/turing:reproduce exp-042 --tolerance 0.05 --runs 5   # Lenient, more runs
+```

package/commands/seed.md

@@ -0,0 +1,47 @@
+---
+name: seed
+description: Run multi-seed study on an experiment to compute mean/std/CI and flag seed-sensitive results. Prevents publishing lucky seeds.
+disable-model-invocation: true
+argument-hint: "[N] [--quick] [--exp-id <id>]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Run a multi-seed study to verify that experiment results are robust across random seeds.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - A bare number (e.g., `5`) sets the seed count
+   - `--quick` runs 3 seeds instead of 5
+   - `--exp-id exp-042` targets a specific experiment (defaults to best)
+   - `--seed-list 42,123,456` uses specific seed values
+
+3. **Run seed study:**
+   ```bash
+   python scripts/seed_runner.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - Show the per-seed results table
+   - Show mean +/- std with 95% CI
+   - **STABLE (CV < 5%):** result is robust, safe to report
+   - **SEED-SENSITIVE (CV >= 5%):** result varies too much across seeds — do not report single-seed numbers
+   - If seed-sensitive, recommend reporting as mean +/- std over N seeds
+
+5. **Saved output:** results are written to `experiments/seed_studies/exp-NNN-seeds.yaml`
+
+6. **If no training pipeline exists:** suggest `/turing:init` first.
+
+## Examples
+
+```
+/turing:seed              # 5 seeds on best experiment
+/turing:seed --quick      # 3 seeds for fast check
+/turing:seed 10           # 10 seeds for thorough study
+/turing:seed --exp-id exp-042   # Specific experiment
+```

package/commands/turing.md

@@ -20,6 +20,8 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "poster", "presentation", "one-pager", "visual summary" | `/turing:poster` | Document |
 | "report", "write-up", "findings", "document results" | `/turing:report` | Document |
 | "validate", "stability", "check variance", "noisy" | `/turing:validate` | Validate |
+| "seed", "seed study", "multi-seed", "lucky seed", "seed sensitivity" | `/turing:seed` | Validate |
+| "reproduce", "reproducibility", "verify results", "re-run experiment", "repro" | `/turing:reproduce` | Validate |
 | "suggest", "what model", "recommend", "which architecture", "literature" | `/turing:suggest` | Research |
 | "explore hypotheses", "tree search", "treequest", "search hypothesis space", "MCTS" | `/turing:explore` | Research |
 | "design", "plan experiment", "how should I test", "experiment design" | `/turing:design` | Design |
@@ -39,6 +41,8 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:brief` | Generate structured research intelligence report | @ml-evaluator |
 | `/turing:init` | Scaffold a new ML project | (inline) |
 | `/turing:validate` | Check metric stability, auto-fix if noisy | (inline) |
+| `/turing:seed [N] [--quick]` | Multi-seed study: mean/std/CI, flag seed-sensitive results | (inline) |
+| `/turing:reproduce <exp-id>` | Reproducibility verification with tolerance checking | (inline) |
 | `/turing:suggest` | Literature-grounded model architecture suggestions | (inline, uses WebSearch) |
 | `/turing:explore` | Tree-search hypothesis exploration via AB-MCTS | (inline) |
 | `/turing:design <hyp-id>` | Generate structured experiment design from hypothesis | (inline, uses WebSearch) |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -22,7 +22,7 @@ const PLUGIN_ROOT = join(__dirname, "..");
 const SUB_COMMANDS = [
   "init", "train", "status", "compare", "sweep", "validate",
   "try", "brief", "suggest", "explore", "design", "logbook", "poster",
-  "report", "mode", "preflight", "card",
+  "report", "mode", "preflight", "card", "seed", "reproduce",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -31,6 +31,8 @@ const EXPECTED_COMMANDS = [
   "mode/SKILL.md",
   "preflight/SKILL.md",
   "card/SKILL.md",
+  "seed/SKILL.md",
+  "reproduce/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];

package/templates/config.yaml

@@ -19,6 +19,16 @@ evaluation:
   # Set to false for metrics where higher is better (accuracy, f1, auc)
   lower_is_better: false  # {{METRIC_DIRECTION}} -- change to true if lower is better
 
+  # Multi-seed configuration (Phase 10.1: /turing:seed)
+  # Seeds used for seed studies — diverse values for good coverage
+  seed_seeds: [42, 123, 456, 789, 1024, 1337, 2048, 3141, 4096, 7919]
+  seed_study_n_runs: 5        # Default number of seeds for /turing:seed
+  seed_sensitivity_threshold: 5.0  # CV% above this = seed-sensitive
+
+  # Reproducibility configuration (Phase 10.2: /turing:reproduce)
+  reproduce_tolerance: 0.02   # 2% relative tolerance for approximate match
+  reproduce_n_runs: 3         # Default reproduction runs for stochastic algorithms
+
 convergence:
   patience: 3                    # Consecutive non-improvements before stopping
   improvement_threshold: 0.005   # 0.5% relative improvement required

package/templates/program.md

@@ -170,6 +170,11 @@ The autoresearch experiment loop. Each iteration is one experiment — one hypot
    - N consecutive non-improvements (`config.yaml` → `convergence.patience`) = STOP
    - `max_iterations` reached = STOP
    - Report final best model and recommend next steps
+   - **Before declaring final results**, run a seed study to verify robustness:
+     ```bash
+     python scripts/seed_runner.py --quick
+     ```
+     If CV > 5%, the result is seed-sensitive — report mean ± std, not a single-seed number.
 
 10. **REPEAT** — return to step 1.
 

package/templates/scripts/generate_brief.py

@@ -25,6 +25,7 @@ import yaml
 
 from scripts.cost_frontier import compute_pareto_frontier, load_cost_data, _format_seconds
 from scripts.turing_io import load_config, load_experiments, load_hypotheses
+from scripts.seed_runner import CV_THRESHOLD
 
 
 def compute_campaign_summary(experiments: list[dict]) -> dict:
@@ -211,6 +212,40 @@ def detect_environment_drift(experiments: list[dict]) -> list[str]:
     return warnings
 
 
+def load_seed_studies(seed_dir: str = "experiments/seed_studies") -> list[dict]:
+    """Load all seed study results from YAML files."""
+    path = Path(seed_dir)
+    if not path.exists():
+        return []
+    studies = []
+    for f in sorted(path.glob("*-seeds.yaml")):
+        try:
+            with open(f) as fh:
+                study = yaml.safe_load(fh)
+                if study and isinstance(study, dict):
+                    studies.append(study)
+        except (yaml.YAMLError, OSError):
+            continue
+    return studies
+
+
+def load_reproductions(repro_dir: str = "experiments/reproductions") -> list[dict]:
+    """Load all reproduction reports from YAML files."""
+    path = Path(repro_dir)
+    if not path.exists():
+        return []
+    reports = []
+    for f in sorted(path.glob("*-repro.yaml")):
+        try:
+            with open(f) as fh:
+                report = yaml.safe_load(fh)
+                if report and isinstance(report, dict):
+                    reports.append(report)
+        except (yaml.YAMLError, OSError):
+            continue
+    return reports
+
+
 def format_brief(
     campaign: dict,
     best: dict | None,
@@ -223,6 +258,8 @@ def format_brief(
     env_warnings: list[str] | None = None,
     cost_data: list | None = None,
     cost_frontier: list | None = None,
+    seed_studies: list[dict] | None = None,
+    reproductions: list[dict] | None = None,
 ) -> str:
     """Format the research briefing as markdown."""
     direction = "lower" if lower_is_better else "higher"
@@ -361,6 +398,44 @@ def format_brief(
                     f"The {pct:.1f}% improvement costs {ratio:.0f}x more compute.",
                 ])
 
+    # Seed studies
+    if seed_studies:
+        lines.extend(["", "## Seed Studies", ""])
+        for study in seed_studies:
+            exp_id = study.get("experiment_id", "?")
+            sensitive = study.get("seed_sensitive", False)
+            status = "SEED-SENSITIVE" if sensitive else "STABLE"
+            lines.append(
+                f"- **{exp_id}:** {study.get('metric', metric)} = "
+                f"{study.get('mean', 0):.4f} +/- {study.get('std', 0):.4f} "
+                f"(CV={study.get('cv_percent', 0):.1f}%) — **{status}**"
+            )
+            if sensitive:
+                lines.append(
+                    f"  - 95% CI: [{study['ci_95'][0]:.4f}, {study['ci_95'][1]:.4f}] "
+                    f"over {len(study.get('seeds_run', []))} seeds"
+                )
+        if any(s.get("seed_sensitive") for s in seed_studies):
+            lines.extend(["", "*Some results are seed-sensitive. Report distributions, not point estimates.*"])
+
+    # Reproduction reports
+    if reproductions:
+        lines.extend(["", "## Reproducibility", ""])
+        verdict_markers = {
+            "reproducible": "PASS",
+            "approximately_reproducible": "PASS (approx)",
+            "not_reproducible": "FAIL",
+            "environment_changed": "WARN (env)",
+        }
+        for report in reproductions:
+            exp_id = report.get("experiment_id", "?")
+            verdict = report.get("verdict", "unknown")
+            marker = verdict_markers.get(verdict, verdict)
+            lines.append(f"- **{exp_id}:** {marker} — {report.get('reason', 'N/A')}")
+        failed = [r for r in reproductions if r.get("verdict") in ("not_reproducible", "environment_changed")]
+        if failed:
+            lines.extend(["", f"*{len(failed)} experiment(s) failed reproducibility checks.*"])
+
     lines.extend([
         "",
         "## Recommendations",
@@ -420,11 +495,17 @@ def generate_brief(
     cost_records = load_cost_data(log_path, metric)
     pareto = compute_pareto_frontier(cost_records, lower_is_better) if cost_records else []
 
+    # Load seed studies and reproduction reports
+    seed_studies = load_seed_studies()
+    reproductions = load_reproductions()
+
     return format_brief(
         campaign, best, trajectory, model_types, hypotheses,
         metric, lower_is_better, failures, env_warnings,
         cost_data=cost_records if cost_records else None,
         cost_frontier=pareto if cost_records else None,
+        seed_studies=seed_studies if seed_studies else None,
+        reproductions=reproductions if reproductions else None,
     )
 
 

package/templates/scripts/generate_model_card.py

@@ -243,6 +243,31 @@ def generate_card(
     else:
         lines.append("No experiments completed yet.")
 
+    # --- Seed Study ---
+    if best:
+        seed_study_path = Path("experiments/seed_studies") / f"{best.get('experiment_id', 'unknown')}-seeds.yaml"
+        if seed_study_path.exists():
+            import yaml
+            with open(seed_study_path) as f:
+                seed_study = yaml.safe_load(f) or {}
+            if seed_study and "mean" in seed_study:
+                sensitive = seed_study.get("seed_sensitive", False)
+                status = "SEED-SENSITIVE" if sensitive else "STABLE"
+                lines.extend([
+                    "",
+                    "### Seed Study",
+                    "",
+                    f"- **Status:** {status}",
+                    f"- **{metric}:** {seed_study['mean']:.4f} +/- {seed_study.get('std', 0):.4f}",
+                ])
+                if "ci_95" in seed_study:
+                    ci = seed_study["ci_95"]
+                    lines.append(f"- **95% CI:** [{ci[0]:.4f}, {ci[1]:.4f}]")
+                lines.append(f"- **CV:** {seed_study.get('cv_percent', 0):.2f}%")
+                lines.append(f"- **Seeds tested:** {len(seed_study.get('seeds_run', []))}")
+                if sensitive:
+                    lines.append("- *Result varies significantly across seeds. Report distribution, not point estimate.*")
+
     # --- Training History ---
     lines.extend([
         "",

package/templates/scripts/leaderboard.py

@@ -503,6 +503,16 @@ def main() -> None:
             print()
             print(footer)
 
+    # Show seed study status for #1 if available
+    if ranked and args.fmt not in ("csv",):
+        from scripts.turing_io import load_seed_study
+        best_id = ranked[0].get("experiment_id")
+        if best_id:
+            study = load_seed_study(best_id)
+            if study and "mean" in study:
+                sensitive = "SEED-SENSITIVE" if study.get("seed_sensitive") else "STABLE"
+                print(f"\n  Seed study: {metric}={study['mean']:.4f}±{study.get('std',0):.4f} ({sensitive})")
+
 
 if __name__ == "__main__":
     main()