claude-turing 1.5.0 → 2.0.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "1.5.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 24 commands, 2 specialized agents, performance profiling, smart Pareto-based checkpoint management, experiment intelligence (error analysis, ablation studies, Pareto frontiers), statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "2.0.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 25 commands, 2 specialized agents, production model export (6 formats, equivalence verification, latency benchmarking), performance profiling, smart Pareto-based checkpoint management, experiment intelligence (error analysis, ablation studies, Pareto frontiers), statistical rigor (multi-seed studies, reproducibility verification), tree-search hypothesis exploration (TreeQuest AB-MCTS), cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -330,6 +330,7 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:frontier [--metrics]` | Pareto frontier — multi-objective tradeoff visualization |
 | `/turing:profile [exp-id]` | Computational profiling — timing, memory, throughput, bottleneck detection |
 | `/turing:checkpoint <action>` | Smart checkpoint management — list, prune (Pareto), average, resume, stats |
+| `/turing:export [--format]` | Export model to production format with equivalence check + latency benchmark |
 | `/turing:card` | Generate a model card — performance, limitations, intended use, artifact contract |
 | `/turing:logbook` | Generate HTML experiment logbook |
 | `/turing:report` | Generate research report |
@@ -519,11 +520,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-24 commands, 2 agents, 8 config files, 39 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling + smart checkpoints, 542 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+25 commands, 2 agents, 8 config files, 44 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, 611 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              23 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance)
+├── commands/              24 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/export.md

@@ -0,0 +1,48 @@
+---
+name: export
+description: Export model to production format with equivalence verification, latency benchmarking, and deployment model card.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--format joblib|xgboost_json|onnx|torchscript|tflite]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Export a trained model to a production-ready format.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - First argument can be an experiment ID (e.g., `exp-042`); defaults to best
+   - `--format joblib|xgboost_json|onnx|torchscript|tflite` specifies export format (auto-detected if omitted)
+   - `--skip-equivalence` skips inference equivalence check
+   - `--skip-latency` skips latency benchmark
+   - `--samples 100` sets test sample count
+
+3. **Run export pipeline:**
+   ```bash
+   python scripts/export_model.py $ARGUMENTS
+   ```
+
+4. **Report results:**
+   - **Export:** format, file size, output path, dependencies
+   - **Equivalence:** verdict (equivalent/approximately_equivalent/divergent), max delta
+   - **Latency:** p50/p95/p99 ms, speedup vs original
+   - **Model Card:** metrics, seed study, equivalence, latency, dependencies
+
+5. **Output:** exported model + model_card.yaml written to `exports/exp-NNN/`
+
+6. **If model file not found:** suggest checking models/best/ directory.
+
+## Examples
+
+```
+/turing:export                                     # Best experiment, default format
+/turing:export exp-042                             # Specific experiment
+/turing:export --format xgboost_json               # Native XGBoost JSON
+/turing:export --format onnx                       # ONNX format
+/turing:export --skip-equivalence --skip-latency   # Fast export
+```

package/commands/turing.md

@@ -31,6 +31,7 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "diagnose", "error analysis", "failure modes", "where does it fail", "confusion matrix" | `/turing:diagnose` | Analyze |
 | "ablate", "ablation", "remove component", "which features matter", "component impact" | `/turing:ablate` | Analyze |
 | "frontier", "pareto", "tradeoff", "tradeoffs", "multi-objective", "which model is best" | `/turing:frontier` | Analyze |
+| "export", "deploy", "production", "onnx", "torchscript", "tflite", "ship model" | `/turing:export` | Deploy |
 | "profile", "profiling", "bottleneck", "slow training", "why is it slow", "timing" | `/turing:profile` | Check |
 | "checkpoint", "checkpoints", "prune checkpoints", "disk space", "resume training" | `/turing:checkpoint` | Check |
 
@@ -60,6 +61,7 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:diagnose [exp-id]` | Error analysis: failure modes, confused pairs, feature-range bias | (inline) |
 | `/turing:ablate [--components]` | Ablation study: remove components, measure impact, flag dead weight | (inline) |
 | `/turing:frontier [--metrics]` | Pareto frontier: multi-objective tradeoff visualization | (inline) |
+| `/turing:export [exp-id] [--format]` | Export model to production format with equivalence check + latency benchmark | (inline) |
 | `/turing:profile [exp-id]` | Computational profiling: timing, memory, throughput, bottleneck detection | (inline) |
 | `/turing:checkpoint <action>` | Smart checkpoint management: list, prune (Pareto), average, resume, stats | (inline) |
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "1.5.0",
+  "version": "2.0.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -23,7 +23,7 @@ const SUB_COMMANDS = [
   "init", "train", "status", "compare", "sweep", "validate",
   "try", "brief", "suggest", "explore", "design", "logbook", "poster",
   "report", "mode", "preflight", "card", "seed", "reproduce",
-  "diagnose", "ablate", "frontier", "profile", "checkpoint",
+  "diagnose", "ablate", "frontier", "profile", "checkpoint", "export",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -38,6 +38,7 @@ const EXPECTED_COMMANDS = [
   "frontier/SKILL.md",
   "profile/SKILL.md",
   "checkpoint/SKILL.md",
+  "export/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];

package/templates/scripts/equivalence_checker.py

@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+"""Inference equivalence verification for model exports.
+
+Compares outputs between the original model and the exported model
+to verify they produce identical (or near-identical) results.
+
+Verdicts:
+    equivalent                 — max delta < 1e-6 (float precision)
+    approximately_equivalent   — max delta < tolerance (default 1e-5)
+    divergent                  — max delta >= tolerance
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+
+
+FLOAT32_TOLERANCE = 1e-5
+QUANTIZED_TOLERANCE = 1e-3
+EXACT_TOLERANCE = 1e-6
+
+
+def compare_outputs(
+    original_outputs: list | np.ndarray,
+    exported_outputs: list | np.ndarray,
+    tolerance: float = FLOAT32_TOLERANCE,
+) -> dict:
+    """Compare outputs from original and exported models.
+
+    Args:
+        original_outputs: Predictions from original model.
+        exported_outputs: Predictions from exported model.
+        tolerance: Maximum allowed difference.
+
+    Returns:
+        Dict with verdict, max_delta, mean_delta, and per-sample details.
+    """
+    orig = np.array(original_outputs, dtype=np.float64).flatten()
+    exported = np.array(exported_outputs, dtype=np.float64).flatten()
+
+    if orig.shape != exported.shape:
+        return {
+            "verdict": "divergent",
+            "reason": f"Shape mismatch: original {orig.shape} vs exported {exported.shape}",
+            "max_delta": float("inf"),
+            "mean_delta": float("inf"),
+            "n_samples": 0,
+            "n_divergent": 0,
+        }
+
+    deltas = np.abs(orig - exported)
+    max_delta = float(np.max(deltas))
+    mean_delta = float(np.mean(deltas))
+    n_divergent = int(np.sum(deltas >= tolerance))
+
+    if max_delta < EXACT_TOLERANCE:
+        verdict = "equivalent"
+        reason = f"Exact match (max delta {max_delta:.2e} < {EXACT_TOLERANCE})"
+    elif max_delta < tolerance:
+        verdict = "approximately_equivalent"
+        reason = f"Within tolerance (max delta {max_delta:.2e} < {tolerance})"
+    else:
+        verdict = "divergent"
+        reason = (
+            f"Max delta {max_delta:.2e} exceeds tolerance {tolerance}. "
+            f"{n_divergent} of {len(orig)} samples diverge."
+        )
+
+    return {
+        "verdict": verdict,
+        "reason": reason,
+        "max_delta": round(max_delta, 10),
+        "mean_delta": round(mean_delta, 10),
+        "n_samples": len(orig),
+        "n_divergent": n_divergent,
+        "tolerance": tolerance,
+    }
+
+
+def run_equivalence_check(
+    original_predict_fn,
+    exported_predict_fn,
+    test_data: np.ndarray | list,
+    tolerance: float = FLOAT32_TOLERANCE,
+) -> dict:
+    """Run equivalence check by predicting on test data with both models.
+
+    Args:
+        original_predict_fn: Callable that takes input data and returns predictions.
+        exported_predict_fn: Callable for the exported model.
+        test_data: Input data to predict on.
+        tolerance: Maximum allowed difference.
+
+    Returns:
+        Equivalence check result dict.
+    """
+    try:
+        original_preds = original_predict_fn(test_data)
+    except Exception as e:
+        return {"verdict": "error", "reason": f"Original model prediction failed: {e}"}
+
+    try:
+        exported_preds = exported_predict_fn(test_data)
+    except Exception as e:
+        return {"verdict": "error", "reason": f"Exported model prediction failed: {e}"}
+
+    return compare_outputs(original_preds, exported_preds, tolerance)
+
+
+def generate_test_data(n_samples: int = 100, n_features: int = 10, seed: int = 42) -> np.ndarray:
+    """Generate random test data for equivalence checking.
+
+    Args:
+        n_samples: Number of test samples.
+        n_features: Number of features per sample.
+        seed: Random seed for reproducibility.
+
+    Returns:
+        Array of shape (n_samples, n_features).
+    """
+    rng = np.random.RandomState(seed)
+    return rng.randn(n_samples, n_features).astype(np.float32)
+
+
+def format_equivalence_report(result: dict) -> str:
+    """Format equivalence check result as readable text."""
+    verdict = result.get("verdict", "unknown")
+    reason = result.get("reason", "")
+
+    verdict_markers = {
+        "equivalent": "PASS (exact)",
+        "approximately_equivalent": "PASS (approx)",
+        "divergent": "FAIL",
+        "error": "ERROR",
+    }
+    marker = verdict_markers.get(verdict, verdict)
+
+    lines = [
+        f"## Equivalence Check: {marker}",
+        "",
+        f"*{reason}*",
+        "",
+    ]
+
+    if result.get("n_samples"):
+        lines.extend([
+            f"- **Samples tested:** {result['n_samples']}",
+            f"- **Max delta:** {result['max_delta']:.2e}",
+            f"- **Mean delta:** {result['mean_delta']:.2e}",
+            f"- **Divergent samples:** {result['n_divergent']}",
+            f"- **Tolerance:** {result['tolerance']:.0e}",
+        ])
+
+    return "\n".join(lines)

package/templates/scripts/export_card.py

@@ -0,0 +1,183 @@
+#!/usr/bin/env python3
+"""Deployment model card generation for exported models.
+
+Produces a structured model card with metrics, seed study results,
+export format, equivalence check, latency benchmarks, and dependencies.
+"""
+
+from __future__ import annotations
+
+import yaml
+from datetime import datetime, timezone
+from pathlib import Path
+
+from scripts.turing_io import load_config, load_seed_study
+
+
+def generate_export_card(
+    experiment: dict,
+    export_result: dict,
+    equivalence: dict | None = None,
+    latency: dict | None = None,
+    config: dict | None = None,
+) -> dict:
+    """Generate a deployment model card for an exported model.
+
+    Args:
+        experiment: Original experiment dict from log.jsonl.
+        export_result: Result from export_formats.export_model().
+        equivalence: Result from equivalence_checker.compare_outputs().
+        latency: Latency comparison from latency_benchmark.compare_latency().
+        config: Project config dict.
+
+    Returns:
+        Model card dict.
+    """
+    exp_id = experiment.get("experiment_id", "unknown")
+    metrics = experiment.get("metrics", {})
+    exp_config = experiment.get("config", {})
+    model_type = exp_config.get("model_type", config.get("model", {}).get("type", "unknown") if config else "unknown")
+
+    eval_cfg = config.get("evaluation", {}) if config else {}
+    primary_metric = eval_cfg.get("primary_metric", "accuracy")
+    task_desc = config.get("task_description", eval_cfg.get("primary_metric", "N/A")) if config else "N/A"
+
+    card = {
+        "name": f"{exp_id}-{model_type}",
+        "experiment_id": exp_id,
+        "task": task_desc,
+        "model_type": model_type,
+        "primary_metric": primary_metric,
+        "metrics": {k: round(v, 4) if isinstance(v, float) else v for k, v in metrics.items()},
+        "export_format": export_result.get("format", "unknown"),
+        "export_path": export_result.get("path"),
+        "size_mb": export_result.get("size_mb", 0),
+        "dependencies": export_result.get("dependencies", []),
+        "training_date": experiment.get("timestamp", "unknown"),
+        "export_date": datetime.now(timezone.utc).isoformat(),
+    }
+
+    # Seed study (if available)
+    seed_study = load_seed_study(exp_id)
+    if seed_study and "mean" in seed_study:
+        card["seed_study"] = {
+            "mean": seed_study["mean"],
+            "std": seed_study.get("std", 0),
+            "cv_percent": seed_study.get("cv_percent", 0),
+            "seed_sensitive": seed_study.get("seed_sensitive", False),
+            "seeds_tested": len(seed_study.get("seeds_run", [])),
+        }
+
+    # Equivalence check
+    if equivalence:
+        card["equivalence"] = {
+            "verdict": equivalence.get("verdict", "unknown"),
+            "max_delta": equivalence.get("max_delta", 0),
+            "n_samples_tested": equivalence.get("n_samples", 0),
+        }
+
+    # Latency benchmark
+    if latency and latency.get("verdict") != "error":
+        card["inference_latency"] = {
+            "exported_p50_ms": latency.get("exported_p50_ms"),
+            "exported_p95_ms": latency.get("exported_p95_ms"),
+            "original_p50_ms": latency.get("original_p50_ms"),
+            "speedup": latency.get("speedup_ratio"),
+        }
+
+    # Environment
+    env = experiment.get("environment")
+    if env:
+        card["training_environment"] = {
+            "python_version": env.get("python_version"),
+            "gpu": env.get("gpu_name") or env.get("gpu"),
+        }
+
+    return card
+
+
+def save_export_card(card: dict, output_dir: str) -> Path:
+    """Save export model card to YAML file."""
+    out_path = Path(output_dir)
+    out_path.mkdir(parents=True, exist_ok=True)
+    filepath = out_path / "model_card.yaml"
+    with open(filepath, "w") as f:
+        yaml.dump(card, f, default_flow_style=False, sort_keys=False)
+    return filepath
+
+
+def format_export_card(card: dict) -> str:
+    """Format export model card as readable markdown."""
+    lines = [
+        f"# Export Model Card: {card.get('name', 'unknown')}",
+        "",
+        f"- **Experiment:** {card.get('experiment_id', '?')}",
+        f"- **Task:** {card.get('task', 'N/A')}",
+        f"- **Model type:** {card.get('model_type', '?')}",
+        f"- **Export format:** {card.get('export_format', '?')}",
+        f"- **Size:** {card.get('size_mb', 0):.2f} MB",
+        f"- **Dependencies:** {', '.join(card.get('dependencies', []))}",
+        "",
+        "## Metrics",
+        "",
+    ]
+
+    for metric, value in card.get("metrics", {}).items():
+        if isinstance(value, float):
+            lines.append(f"- **{metric}:** {value:.4f}")
+        else:
+            lines.append(f"- **{metric}:** {value}")
+
+    # Seed study
+    seed = card.get("seed_study")
+    if seed:
+        status = "SEED-SENSITIVE" if seed.get("seed_sensitive") else "STABLE"
+        lines.extend([
+            "",
+            "## Seed Study",
+            "",
+            f"- **Status:** {status}",
+            f"- **Mean ± Std:** {seed['mean']:.4f} ± {seed.get('std', 0):.4f}",
+            f"- **CV:** {seed.get('cv_percent', 0):.2f}%",
+            f"- **Seeds tested:** {seed.get('seeds_tested', 0)}",
+        ])
+
+    # Equivalence
+    eq = card.get("equivalence")
+    if eq:
+        verdict_markers = {
+            "equivalent": "PASS (exact)",
+            "approximately_equivalent": "PASS (approx)",
+            "divergent": "FAIL",
+        }
+        marker = verdict_markers.get(eq["verdict"], eq["verdict"])
+        lines.extend([
+            "",
+            "## Equivalence",
+            "",
+            f"- **Verdict:** {marker}",
+            f"- **Max delta:** {eq.get('max_delta', 0):.2e}",
+            f"- **Samples tested:** {eq.get('n_samples_tested', 0)}",
+        ])
+
+    # Latency
+    lat = card.get("inference_latency")
+    if lat:
+        lines.extend([
+            "",
+            "## Inference Latency",
+            "",
+            f"- **Exported p50:** {lat.get('exported_p50_ms', 0):.2f} ms",
+            f"- **Exported p95:** {lat.get('exported_p95_ms', 0):.2f} ms",
+        ])
+        if lat.get("original_p50_ms"):
+            lines.append(f"- **Original p50:** {lat['original_p50_ms']:.2f} ms")
+        if lat.get("speedup"):
+            lines.append(f"- **Speedup:** {lat['speedup']:.1f}x")
+
+    lines.extend([
+        "",
+        f"*Exported: {card.get('export_date', 'unknown')}*",
+    ])
+
+    return "\n".join(lines)