claude-turing 3.1.0 → 3.2.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "3.1.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 44 commands, 2 specialized agents, pre-training intelligence (sanity checks + baseline generation + leakage detection), meta-intelligence (cross-project knowledge transfer + methodology audit), scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "3.2.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 47 commands, 2 specialized agents, model debugging (xray + sensitivity + calibration), pre-training intelligence (sanity checks + baseline generation + leakage detection), meta-intelligence (cross-project knowledge transfer + methodology audit), scaling & efficiency (scaling laws + compute budget + model distillation), model composition (ensemble + pipeline stitch + warm-start), deep analysis (experiment diff + live training monitor + regression gate), experiment orchestration (batch queue + smart retry + branching), literature integration + paper drafting, production model export, performance profiling, smart checkpoints, experiment intelligence, statistical rigor, tree-search hypothesis exploration, cost-performance frontier, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -355,6 +355,9 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | `/turing:sanity [--quick]` | Pre-training sanity checks — initial loss, single-batch overfit, gradient flow, output validation |
 | `/turing:baseline [--methods]` | Automatic baseline generation — random, majority/mean, linear, k-NN |
 | `/turing:leak [--deep]` | Targeted leakage detection — single-feature tests, correlation, train/test overlap |
+| `/turing:xray [exp-id]` | Internal model diagnostics — gradient flow, dead neurons, weight distributions, tree analysis |
+| `/turing:sensitivity [exp-id]` | Hyperparameter sensitivity — rank parameters by impact, detect non-monotonic responses |
+| `/turing:calibrate [exp-id]` | Probability calibration — ECE/MCE, reliability diagrams, Platt/isotonic/temperature scaling |
 
 And for fully hands-off operation:
 
@@ -539,11 +542,11 @@ Each project gets independent config, data, experiments, models, and agent memor
 
 ## Architecture of Turing Itself
 
-44 commands, 2 agents, 10 config files, 63 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), pre-training intelligence (sanity + baseline + leak), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+47 commands, 2 agents, 10 config files, 66 template scripts, model registry, artifact contract, cost-performance frontier, model cards, tree-search exploration, statistical rigor, experiment intelligence, performance profiling, smart checkpoints, production model export, literature integration, paper section drafting, experiment orchestration (queue + retry + fork), deep analysis (diff + watch + regress), model composition (ensemble + stitch + warm), scaling & efficiency (scale + budget + distill), meta-intelligence (transfer + audit), pre-training intelligence (sanity + baseline + leak), model debugging (xray + sensitivity + calibrate), 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
-├── commands/              43 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency + meta-intelligence + pre-training intelligence)
+├── commands/              46 skill files (core + taste-leverage + reporting + exploration + statistical rigor + experiment intelligence + performance + deployment + research workflow + orchestration + deep analysis + model composition + scaling & efficiency + meta-intelligence + pre-training intelligence + model debugging)
 ├── agents/                2 agents (researcher: read/write, evaluator: read-only)
 ├── config/                8 files (lifecycle, taxonomy, archetypes, novelty aliases)
 ├── templates/             Scaffolded into user projects by /turing:init

package/commands/calibrate.md

@@ -0,0 +1,47 @@
+---
+name: calibrate
+description: Probability calibration — measure ECE, plot reliability diagrams, apply Platt scaling or isotonic regression.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--method platt|isotonic|temperature|auto]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Make model probabilities trustworthy. Does 80% confidence actually mean 80% correct?
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - Optional experiment ID
+   - `--method platt|isotonic|temperature|auto` — calibration method (default: auto)
+   - `--json` — raw JSON output
+
+3. **Run calibration:**
+   ```bash
+   python scripts/calibration.py $ARGUMENTS
+   ```
+
+4. **Report includes:**
+   - ECE/MCE before calibration
+   - Reliability diagram (predicted vs actual per bin)
+   - Calibration method comparison table
+   - Verdict: ALREADY CALIBRATED / IMPROVED / NO IMPROVEMENT
+
+5. **Methods:**
+   - **Platt:** logistic regression on logits
+   - **Isotonic:** non-parametric (more flexible, needs more data)
+   - **Temperature:** single scalar T parameter
+   - **Auto:** tries all, picks lowest ECE
+
+6. **Saved output:** report in `experiments/calibration/<exp-id>-calibration.yaml`
+
+## Examples
+
+```
+/turing:calibrate exp-042                  # Auto-select best method
+/turing:calibrate exp-042 --method platt   # Platt scaling only
+```

package/commands/sensitivity.md

@@ -0,0 +1,41 @@
+---
+name: sensitivity
+description: Hyperparameter sensitivity analysis — rank parameters by impact, identify which matter and which are noise.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--params learning_rate,max_depth]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+Which hyperparameters actually matter? Stop wasting time on the ones that don't.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - Optional experiment ID
+   - `--params "learning_rate,max_depth"` — specific parameters to analyze
+   - `--json` — raw JSON output
+
+3. **Run sensitivity analysis:**
+   ```bash
+   python scripts/sensitivity_analysis.py $ARGUMENTS
+   ```
+
+4. **Report includes:**
+   - Per-parameter sensitivity ranking: HIGH / MED / LOW / NONE
+   - Metric range for each parameter sweep
+   - Monotonicity detection (is there a sweet spot?)
+   - Recommendations: focus tuning on X, stop tuning Y
+
+5. **Saved output:** report in `experiments/sensitivity/<exp-id>-sensitivity.yaml`
+
+## Examples
+
+```
+/turing:sensitivity exp-042                           # All tunable params
+/turing:sensitivity --params "learning_rate,max_depth" # Specific params
+```

package/commands/turing.md

@@ -53,6 +53,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "sanity", "sanity check", "pre-training", "is it broken", "before training", "quick check" | `/turing:sanity` | Check |
 | "baseline", "baselines", "trivial baseline", "majority class", "is it better than random" | `/turing:baseline` | Analyze |
 | "leak", "leakage", "data leakage scan", "suspicious feature", "train test overlap" | `/turing:leak` | Validate |
+| "xray", "model internals", "dead neurons", "gradient flow", "weight distribution", "inside the model" | `/turing:xray` | Analyze |
+| "sensitivity", "which params matter", "hyperparameter importance", "parameter ranking" | `/turing:sensitivity` | Analyze |
+| "calibrate", "calibration", "ECE", "reliability diagram", "overconfident", "probability calibration" | `/turing:calibrate` | Analyze |
 
 ## Sub-commands
 
@@ -102,6 +105,9 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:sanity [--quick]` | Pre-training sanity checks: initial loss, overfit test, gradient flow, output validation | (inline) |
 | `/turing:baseline [--methods]` | Automatic baseline generation: random, majority/mean, linear, k-NN | (inline) |
 | `/turing:leak [--deep]` | Targeted leakage detection: single-feature tests, correlation, train/test overlap | (inline) |
+| `/turing:xray [exp-id]` | Internal model diagnostics: gradient flow, dead neurons, weight distributions, tree analysis | (inline) |
+| `/turing:sensitivity [exp-id]` | Hyperparameter sensitivity analysis: rank parameters by impact, detect non-monotonic responses | (inline) |
+| `/turing:calibrate [exp-id]` | Probability calibration: ECE/MCE, reliability diagrams, Platt/isotonic/temperature scaling | (inline) |
 
 ## Proactive Detection
 

package/commands/xray.md

@@ -0,0 +1,43 @@
+---
+name: xray
+description: Internal model diagnostics — gradient flow, dead neurons, activation stats, weight distributions, tree depth analysis.
+disable-model-invocation: true
+argument-hint: "[exp-id] [--layer encoder.layer.2] [--compare exp-a exp-b]"
+allowed-tools: Read, Bash(*), Grep, Glob
+---
+
+See inside the model. When it underperforms, the fix depends on *why*.
+
+## Steps
+
+1. **Activate environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Parse arguments from `$ARGUMENTS`:**
+   - Optional experiment ID
+   - `--layer "name"` — focus on specific layer
+   - `--compare exp-a exp-b` — side-by-side diagnostics
+   - `--json` — raw JSON output
+
+3. **Run model diagnostics:**
+   ```bash
+   python scripts/model_xray.py $ARGUMENTS
+   ```
+
+4. **Diagnostics by model type:**
+   - **Neural networks:** gradient magnitudes, activation stats, dead neuron %, weight distributions, gradient-to-weight ratio
+   - **Tree models:** depth utilization, leaf purity, feature split dominance
+   - **scikit-learn:** coefficient magnitudes, feature importance concentration
+
+5. **Issues detected:** dead gradients, vanishing/exploding gradients, dead neurons, sparse weights, feature dominance, overfitting risk
+
+6. **Saved output:** report in `experiments/xrays/<exp-id>-xray.yaml`
+
+## Examples
+
+```
+/turing:xray exp-042              # Full diagnostics
+/turing:xray                      # Best experiment
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "3.1.0",
+  "version": "3.2.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -30,6 +30,7 @@ const SUB_COMMANDS = [
   "scale", "budget", "distill",
   "transfer", "audit",
   "sanity", "baseline", "leak",
+  "xray", "sensitivity", "calibrate",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -58,6 +58,9 @@ const EXPECTED_COMMANDS = [
   "sanity/SKILL.md",
   "baseline/SKILL.md",
   "leak/SKILL.md",
+  "xray/SKILL.md",
+  "sensitivity/SKILL.md",
+  "calibrate/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];

package/templates/scripts/calibration.py

@@ -0,0 +1,364 @@
+#!/usr/bin/env python3
+"""Probability calibration for the autoresearch pipeline.
+
+Measures whether model probabilities are well-calibrated, computes ECE/MCE,
+generates reliability diagrams, and applies post-hoc calibration (Platt
+scaling, isotonic regression, temperature scaling).
+
+Usage:
+    python scripts/calibration.py exp-042
+    python scripts/calibration.py exp-042 --method platt
+    python scripts/calibration.py exp-042 --method auto
+    python scripts/calibration.py --json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+from scripts.turing_io import load_config, load_experiments
+
+DEFAULT_LOG_PATH = "experiments/log.jsonl"
+DEFAULT_N_BINS = 10
+CALIBRATION_METHODS = ["platt", "isotonic", "temperature"]
+
+
+# --- Calibration Metrics ---
+
+
+def compute_ece(
+    probabilities: np.ndarray,
+    labels: np.ndarray,
+    n_bins: int = DEFAULT_N_BINS,
+) -> float:
+    """Compute Expected Calibration Error.
+
+    ECE = sum(|bin_accuracy - bin_confidence| * bin_size / total)
+    """
+    if len(probabilities) == 0:
+        return 0.0
+
+    bin_boundaries = np.linspace(0, 1, n_bins + 1)
+    ece = 0.0
+
+    for i in range(n_bins):
+        mask = (probabilities >= bin_boundaries[i]) & (probabilities < bin_boundaries[i + 1])
+        if i == n_bins - 1:
+            mask = (probabilities >= bin_boundaries[i]) & (probabilities <= bin_boundaries[i + 1])
+
+        bin_size = np.sum(mask)
+        if bin_size == 0:
+            continue
+
+        bin_accuracy = np.mean(labels[mask])
+        bin_confidence = np.mean(probabilities[mask])
+        ece += abs(bin_accuracy - bin_confidence) * bin_size / len(probabilities)
+
+    return round(float(ece), 6)
+
+
+def compute_mce(
+    probabilities: np.ndarray,
+    labels: np.ndarray,
+    n_bins: int = DEFAULT_N_BINS,
+) -> float:
+    """Compute Maximum Calibration Error."""
+    if len(probabilities) == 0:
+        return 0.0
+
+    bin_boundaries = np.linspace(0, 1, n_bins + 1)
+    max_gap = 0.0
+
+    for i in range(n_bins):
+        mask = (probabilities >= bin_boundaries[i]) & (probabilities < bin_boundaries[i + 1])
+        if i == n_bins - 1:
+            mask = (probabilities >= bin_boundaries[i]) & (probabilities <= bin_boundaries[i + 1])
+
+        if np.sum(mask) == 0:
+            continue
+
+        bin_accuracy = np.mean(labels[mask])
+        bin_confidence = np.mean(probabilities[mask])
+        max_gap = max(max_gap, abs(bin_accuracy - bin_confidence))
+
+    return round(float(max_gap), 6)
+
+
+def compute_reliability_diagram(
+    probabilities: np.ndarray,
+    labels: np.ndarray,
+    n_bins: int = DEFAULT_N_BINS,
+) -> list[dict]:
+    """Compute reliability diagram data."""
+    if len(probabilities) == 0:
+        return []
+
+    bin_boundaries = np.linspace(0, 1, n_bins + 1)
+    bins = []
+
+    for i in range(n_bins):
+        lo = bin_boundaries[i]
+        hi = bin_boundaries[i + 1]
+        mask = (probabilities >= lo) & (probabilities < hi)
+        if i == n_bins - 1:
+            mask = (probabilities >= lo) & (probabilities <= hi)
+
+        bin_size = int(np.sum(mask))
+        if bin_size == 0:
+            bins.append({"bin": f"[{lo:.1f}-{hi:.1f}]", "predicted": None,
+                        "actual": None, "gap": None, "n": 0})
+            continue
+
+        predicted = float(np.mean(probabilities[mask]))
+        actual = float(np.mean(labels[mask]))
+        gap = actual - predicted
+
+        bins.append({
+            "bin": f"[{lo:.1f}-{hi:.1f}]",
+            "predicted": round(predicted, 4),
+            "actual": round(actual, 4),
+            "gap": round(gap, 4),
+            "n": bin_size,
+        })
+
+    return bins
+
+
+# --- Calibration Methods ---
+
+
+def platt_scaling(
+    logits: np.ndarray,
+    labels: np.ndarray,
+) -> dict:
+    """Apply Platt scaling (logistic regression on logits)."""
+    from scipy.special import expit
+
+    # Fit logistic regression: P(y=1|f) = sigmoid(a*f + b)
+    # Simple gradient descent for a, b
+    a, b = 1.0, 0.0
+    lr = 0.01
+    for _ in range(1000):
+        pred = expit(a * logits + b)
+        pred = np.clip(pred, 1e-7, 1 - 1e-7)
+        grad_a = np.mean((pred - labels) * logits)
+        grad_b = np.mean(pred - labels)
+        a -= lr * grad_a
+        b -= lr * grad_b
+
+    calibrated = expit(a * logits + b)
+    return {"method": "platt", "params": {"a": round(float(a), 6), "b": round(float(b), 6)},
+            "calibrated_probabilities": calibrated}
+
+
+def isotonic_calibration(
+    probabilities: np.ndarray,
+    labels: np.ndarray,
+) -> dict:
+    """Apply isotonic regression calibration."""
+    from sklearn.isotonic import IsotonicRegression
+
+    iso = IsotonicRegression(out_of_bounds="clip")
+    calibrated = iso.fit_transform(probabilities, labels)
+    return {"method": "isotonic", "params": {},
+            "calibrated_probabilities": np.clip(calibrated, 0, 1)}
+
+
+def temperature_scaling(
+    logits: np.ndarray,
+    labels: np.ndarray,
+) -> dict:
+    """Apply temperature scaling (single parameter T)."""
+    from scipy.special import expit
+
+    best_t = 1.0
+    best_ece = float("inf")
+
+    for t in np.arange(0.5, 5.0, 0.1):
+        scaled = expit(logits / t)
+        ece = compute_ece(scaled, labels)
+        if ece < best_ece:
+            best_ece = ece
+            best_t = t
+
+    calibrated = expit(logits / best_t)
+    return {"method": "temperature", "params": {"T": round(float(best_t), 2)},
+            "calibrated_probabilities": calibrated}
+
+
+# --- Full Pipeline ---
+
+
+def calibrate_model(
+    probabilities: np.ndarray | None = None,
+    logits: np.ndarray | None = None,
+    labels: np.ndarray | None = None,
+    method: str = "auto",
+    exp_id: str | None = None,
+    config_path: str = "config.yaml",
+) -> dict:
+    """Run calibration analysis and optionally apply post-hoc calibration."""
+    if (probabilities is None and logits is None) or labels is None:
+        return {"error": "Provide probabilities (or logits) and labels for calibration"}
+
+    if probabilities is None and logits is not None:
+        from scipy.special import expit
+        probabilities = expit(logits)
+
+    # Before calibration
+    ece_before = compute_ece(probabilities, labels)
+    mce_before = compute_mce(probabilities, labels)
+    reliability = compute_reliability_diagram(probabilities, labels)
+
+    # Determine overconfidence
+    overconfident_bins = [b for b in reliability if b.get("gap") is not None and b["gap"] < -0.05 and b["n"] > 0]
+
+    report = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "experiment_id": exp_id,
+        "before": {"ece": ece_before, "mce": mce_before},
+        "reliability_diagram": reliability,
+        "overconfident_bins": len(overconfident_bins),
+    }
+
+    # Apply calibration
+    methods_to_try = CALIBRATION_METHODS if method == "auto" else [method]
+    results = []
+
+    for m in methods_to_try:
+        try:
+            if m == "platt" and logits is not None:
+                cal = platt_scaling(logits, labels)
+            elif m == "isotonic":
+                cal = isotonic_calibration(probabilities, labels)
+            elif m == "temperature" and logits is not None:
+                cal = temperature_scaling(logits, labels)
+            else:
+                continue
+
+            ece_after = compute_ece(cal["calibrated_probabilities"], labels)
+            results.append({
+                "method": m,
+                "ece_after": ece_after,
+                "improvement": round(ece_before - ece_after, 6),
+                "params": cal.get("params", {}),
+            })
+        except Exception:
+            continue
+
+    # Find best method
+    best = None
+    if results:
+        best = min(results, key=lambda r: r["ece_after"])
+
+    report["calibration_results"] = results
+    report["best_method"] = best
+
+    # Verdict
+    if ece_before < 0.02:
+        report["verdict"] = "already_calibrated"
+        report["reason"] = f"ECE {ece_before:.4f} is already low — calibration not needed"
+    elif best and best["improvement"] > 0.01:
+        report["verdict"] = "improved"
+        report["reason"] = f"{best['method']} reduces ECE from {ece_before:.4f} to {best['ece_after']:.4f}"
+    elif best:
+        report["verdict"] = "marginal_improvement"
+        report["reason"] = f"Best method ({best['method']}) improves ECE by only {best['improvement']:.4f}"
+    else:
+        report["verdict"] = "no_improvement"
+        report["reason"] = "No calibration method improved ECE"
+
+    return report
+
+
+# --- Report Formatting ---
+
+
+def save_calibration_report(report: dict, output_dir: str = "experiments/calibration") -> Path:
+    out_path = Path(output_dir)
+    out_path.mkdir(parents=True, exist_ok=True)
+    exp_id = report.get("experiment_id", "unknown")
+    filepath = out_path / f"{exp_id}-calibration.yaml"
+    clean = json.loads(json.dumps(report, default=str))
+    with open(filepath, "w") as f:
+        yaml.dump(clean, f, default_flow_style=False, sort_keys=False)
+    return filepath
+
+
+def format_calibration_report(report: dict) -> str:
+    if "error" in report:
+        return f"ERROR: {report['error']}"
+
+    exp_id = report.get("experiment_id", "?")
+    before = report.get("before", {})
+
+    lines = [f"# Calibration: {exp_id}", "",
+             f"*Generated {report.get('generated_at', 'N/A')[:19]}*", "",
+             f"**ECE before:** {before.get('ece', '?')}",
+             f"**MCE before:** {before.get('mce', '?')}", ""]
+
+    # Reliability diagram
+    diagram = report.get("reliability_diagram", [])
+    if diagram:
+        lines.extend(["## Reliability Diagram", "",
+                      "| Bin | Predicted | Actual | Gap |",
+                      "|-----|-----------|--------|-----|"])
+        for b in diagram:
+            if b["predicted"] is not None:
+                gap_marker = " overconfident" if b["gap"] is not None and b["gap"] < -0.05 else ""
+                lines.append(f"| {b['bin']} | {b['predicted']:.4f} | {b['actual']:.4f} | {b['gap']:+.4f}{gap_marker} |")
+        lines.append("")
+
+    # Calibration results
+    results = report.get("calibration_results", [])
+    if results:
+        lines.extend(["## Calibration Methods", "",
+                      "| Method | ECE After | Improvement |",
+                      "|--------|-----------|-------------|"])
+        best = report.get("best_method", {})
+        for r in results:
+            marker = " BEST" if r["method"] == best.get("method") else ""
+            lines.append(f"| {r['method']} | {r['ece_after']:.4f} | {r['improvement']:+.4f} |{marker}")
+        lines.append("")
+
+    # Verdict
+    verdict = report.get("verdict", "?")
+    labels = {"already_calibrated": "ALREADY CALIBRATED", "improved": "IMPROVED",
+              "marginal_improvement": "MARGINAL IMPROVEMENT", "no_improvement": "NO IMPROVEMENT"}
+    lines.extend(["## Verdict", "", f"**{labels.get(verdict, verdict.upper())}**", "",
+                  report.get("reason", "")])
+
+    return "\n".join(lines)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Probability calibration")
+    parser.add_argument("exp_id", nargs="?", help="Experiment ID")
+    parser.add_argument("--method", choices=CALIBRATION_METHODS + ["auto"], default="auto")
+    parser.add_argument("--config", default="config.yaml")
+    parser.add_argument("--log", default=DEFAULT_LOG_PATH)
+    parser.add_argument("--json", action="store_true")
+    args = parser.parse_args()
+
+    # Without data, show usage
+    report = calibrate_model(exp_id=args.exp_id, method=args.method, config_path=args.config)
+
+    if "error" not in report:
+        filepath = save_calibration_report(report)
+        print(f"Saved to {filepath}", file=sys.stderr)
+
+    if args.json:
+        print(json.dumps(report, indent=2, default=str))
+    else:
+        print(format_calibration_report(report))
+
+
+if __name__ == "__main__":
+    main()

package/templates/scripts/model_xray.py

@@ -0,0 +1,317 @@
+#!/usr/bin/env python3
+"""Internal model diagnostics for the autoresearch pipeline.
+
+Inspects model internals: gradient flow per layer, activation statistics,
+dead neurons, weight distributions, decision path analysis. Answers
+"what is the model doing internally?" rather than "what are its predictions?"
+
+Usage:
+    python scripts/model_xray.py exp-042
+    python scripts/model_xray.py exp-042 --layer "encoder.layer.2"
+    python scripts/model_xray.py --compare exp-042 exp-053
+    python scripts/model_xray.py --json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+from scripts.turing_io import load_config, load_experiments
+
+DEFAULT_LOG_PATH = "experiments/log.jsonl"
+DEAD_NEURON_THRESHOLD = 0.01  # Activation below this = dead
+EXPLODING_GRADIENT_RATIO = 100  # Gradient > N * mean = exploding
+NEAR_ZERO_WEIGHT = 0.001  # Weight below this = pruning candidate
+
+
+# --- Neural Network Diagnostics ---
+
+
+def diagnose_neural_layers(layer_stats: list[dict]) -> dict:
+    """Analyze neural network layer statistics.
+
+    Args:
+        layer_stats: List of dicts with name, grad_mean, grad_max, act_mean,
+                     act_std, dead_pct, weight_mean, weight_std.
+
+    Returns:
+        Diagnosis dict with per-layer analysis and detected issues.
+    """
+    if not layer_stats:
+        return {"layers": [], "issues": [], "model_type": "neural"}
+
+    issues = []
+    analyzed = []
+
+    # Compute global gradient mean for relative comparison
+    grad_means = [abs(l.get("grad_mean", 0)) for l in layer_stats if l.get("grad_mean") is not None]
+    global_grad_mean = np.mean(grad_means) if grad_means else 0
+
+    for layer in layer_stats:
+        name = layer.get("name", "?")
+        analysis = {"name": name}
+
+        # Gradient analysis
+        grad_mean = abs(layer.get("grad_mean", 0))
+        grad_max = abs(layer.get("grad_max", 0))
+        analysis["grad_mean"] = grad_mean
+        analysis["grad_max"] = grad_max
+
+        if grad_mean == 0 and grad_max == 0:
+            issues.append({"layer": name, "issue": "dead_gradient", "severity": "high",
+                          "message": f"{name}: zero gradients — layer is not learning"})
+        elif global_grad_mean > 0 and grad_mean < global_grad_mean / EXPLODING_GRADIENT_RATIO:
+            ratio = global_grad_mean / grad_mean if grad_mean > 0 else float("inf")
+            issues.append({"layer": name, "issue": "vanishing_gradient", "severity": "high",
+                          "message": f"{name}: gradient {ratio:.0f}x weaker than average — possible vanishing gradient"})
+        elif global_grad_mean > 0 and grad_max > EXPLODING_GRADIENT_RATIO * global_grad_mean:
+            issues.append({"layer": name, "issue": "exploding_gradient", "severity": "critical",
+                          "message": f"{name}: gradient max {grad_max:.2e} is {grad_max/global_grad_mean:.0f}x the average — exploding gradient"})
+
+        # Activation analysis
+        dead_pct = layer.get("dead_pct", 0)
+        analysis["dead_pct"] = dead_pct
+        if dead_pct > 20:
+            issues.append({"layer": name, "issue": "dead_neurons", "severity": "high",
+                          "message": f"{name}: {dead_pct:.0f}% dead neurons — consider batch norm or layer width reduction"})
+        elif dead_pct > 5:
+            issues.append({"layer": name, "issue": "dying_neurons", "severity": "medium",
+                          "message": f"{name}: {dead_pct:.0f}% near-dead neurons"})
+
+        # Weight analysis
+        weight_std = layer.get("weight_std", 0)
+        near_zero_pct = layer.get("near_zero_pct", 0)
+        analysis["weight_std"] = weight_std
+        analysis["near_zero_pct"] = near_zero_pct
+        if near_zero_pct > 50:
+            issues.append({"layer": name, "issue": "sparse_weights", "severity": "medium",
+                          "message": f"{name}: {near_zero_pct:.0f}% near-zero weights — pruning candidate"})
+
+        analyzed.append(analysis)
+
+    return {"layers": analyzed, "issues": issues, "model_type": "neural"}
+
+
+# --- Tree Model Diagnostics ---
+
+
+def diagnose_tree_model(tree_stats: dict) -> dict:
+    """Analyze tree-based model statistics.
+
+    Args:
+        tree_stats: Dict with n_trees, avg_depth, max_depth_allowed,
+                    feature_split_freq, leaf_purity.
+
+    Returns:
+        Diagnosis dict.
+    """
+    issues = []
+
+    n_trees = tree_stats.get("n_trees", 0)
+    avg_depth = tree_stats.get("avg_depth", 0)
+    max_depth = tree_stats.get("max_depth_allowed", 0)
+    feature_splits = tree_stats.get("feature_split_freq", {})
+    leaf_purity = tree_stats.get("leaf_purity", 0)
+
+    # Depth utilization
+    if max_depth > 0 and avg_depth > 0:
+        utilization = avg_depth / max_depth
+        if utilization < 0.5:
+            issues.append({"issue": "underutilized_depth", "severity": "medium",
+                          "message": f"Trees use only {utilization:.0%} of allowed depth ({avg_depth:.1f}/{max_depth}) — consider reducing max_depth"})
+        elif utilization > 0.95:
+            issues.append({"issue": "depth_saturated", "severity": "medium",
+                          "message": f"Trees use {utilization:.0%} of allowed depth — consider increasing max_depth"})
+
+    # Feature dominance
+    if feature_splits:
+        total_splits = sum(feature_splits.values())
+        if total_splits > 0:
+            top_feature = max(feature_splits, key=feature_splits.get)
+            top_pct = feature_splits[top_feature] / total_splits
+            if top_pct > 0.5:
+                issues.append({"issue": "feature_dominance", "severity": "medium",
+                              "message": f"Feature '{top_feature}' dominates {top_pct:.0%} of splits — check for leakage or engineering opportunity"})
+
+    # Leaf purity
+    if leaf_purity > 0.99:
+        issues.append({"issue": "overfitting_risk", "severity": "medium",
+                      "message": f"Leaf purity {leaf_purity:.4f} — model may be overfitting"})
+
+    return {
+        "model_type": "tree",
+        "n_trees": n_trees,
+        "avg_depth": avg_depth,
+        "max_depth_allowed": max_depth,
+        "depth_utilization": round(avg_depth / max_depth, 3) if max_depth > 0 else None,
+        "feature_split_freq": feature_splits,
+        "leaf_purity": leaf_purity,
+        "issues": issues,
+    }
+
+
+# --- sklearn Diagnostics ---
+
+
+def diagnose_sklearn_model(model_stats: dict) -> dict:
+    """Analyze scikit-learn model statistics.
+
+    Args:
+        model_stats: Dict with model_type, coefficients, feature_importances.
+    """
+    issues = []
+    model_type = model_stats.get("model_type", "unknown")
+
+    coefficients = model_stats.get("coefficients", [])
+    if coefficients:
+        coef_arr = np.array(coefficients)
+        max_coef = float(np.max(np.abs(coef_arr)))
+        near_zero = float(np.mean(np.abs(coef_arr) < NEAR_ZERO_WEIGHT))
+
+        if max_coef > 100:
+            issues.append({"issue": "large_coefficients", "severity": "high",
+                          "message": f"Max coefficient magnitude {max_coef:.1f} — consider regularization"})
+        if near_zero > 0.5:
+            issues.append({"issue": "sparse_coefficients", "severity": "medium",
+                          "message": f"{near_zero:.0%} near-zero coefficients — feature selection may help"})
+
+    importances = model_stats.get("feature_importances", [])
+    if importances:
+        imp_arr = np.array(importances)
+        if len(imp_arr) > 0 and np.std(imp_arr) > 0:
+            top_k = min(3, len(imp_arr))
+            top_indices = np.argsort(imp_arr)[-top_k:]
+            top_total = float(np.sum(imp_arr[top_indices]))
+            if top_total > 0.8:
+                issues.append({"issue": "importance_concentrated", "severity": "medium",
+                              "message": f"Top {top_k} features account for {top_total:.0%} of importance"})
+
+    return {
+        "model_type": model_type,
+        "n_coefficients": len(coefficients),
+        "n_importances": len(importances),
+        "issues": issues,
+    }
+
+
+# --- Full X-Ray Pipeline ---
+
+
+def xray_model(
+    exp_id: str | None = None,
+    layer_stats: list[dict] | None = None,
+    tree_stats: dict | None = None,
+    sklearn_stats: dict | None = None,
+    config_path: str = "config.yaml",
+    log_path: str = DEFAULT_LOG_PATH,
+) -> dict:
+    """Run model diagnostics."""
+    config = load_config(config_path)
+    model_type_hint = config.get("model", {}).get("type", "")
+
+    diagnosis = None
+    if layer_stats is not None:
+        diagnosis = diagnose_neural_layers(layer_stats)
+    elif tree_stats is not None:
+        diagnosis = diagnose_tree_model(tree_stats)
+    elif sklearn_stats is not None:
+        diagnosis = diagnose_sklearn_model(sklearn_stats)
+    else:
+        diagnosis = {"model_type": "unknown", "issues": [],
+                     "note": "No model stats provided. Run with model-specific stats for full diagnostics."}
+
+    return {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "experiment_id": exp_id,
+        "diagnosis": diagnosis,
+        "n_issues": len(diagnosis.get("issues", [])),
+    }
+
+
+# --- Report Formatting ---
+
+
+def save_xray_report(report: dict, output_dir: str = "experiments/xrays") -> Path:
+    out_path = Path(output_dir)
+    out_path.mkdir(parents=True, exist_ok=True)
+    exp_id = report.get("experiment_id", "unknown")
+    filepath = out_path / f"{exp_id}-xray.yaml"
+    with open(filepath, "w") as f:
+        yaml.dump(report, f, default_flow_style=False, sort_keys=False)
+    return filepath
+
+
+def format_xray_report(report: dict) -> str:
+    if "error" in report:
+        return f"ERROR: {report['error']}"
+
+    diag = report.get("diagnosis", {})
+    model_type = diag.get("model_type", "?")
+    exp_id = report.get("experiment_id", "?")
+    issues = diag.get("issues", [])
+
+    lines = [f"# X-Ray: {exp_id} ({model_type})", "",
+             f"*Generated {report.get('generated_at', 'N/A')[:19]}*", ""]
+
+    # Neural layer table
+    layers = diag.get("layers", [])
+    if layers:
+        lines.extend(["## Layer Analysis", "",
+                      "| Layer | Grad Mean | Grad Max | Dead % | Weight Std |",
+                      "|-------|-----------|----------|--------|------------|"])
+        for l in layers:
+            lines.append(f"| {l['name']} | {l.get('grad_mean', 0):.2e} | {l.get('grad_max', 0):.2e} | {l.get('dead_pct', 0):.0f}% | {l.get('weight_std', 0):.4f} |")
+        lines.append("")
+
+    # Tree stats
+    if model_type == "tree":
+        lines.extend(["## Tree Statistics", "",
+                      f"- **Trees:** {diag.get('n_trees', '?')}",
+                      f"- **Avg depth:** {diag.get('avg_depth', '?')}/{diag.get('max_depth_allowed', '?')}",
+                      f"- **Leaf purity:** {diag.get('leaf_purity', '?')}", ""])
+
+    # Issues
+    if issues:
+        lines.extend(["## Issues Detected", ""])
+        for i in issues:
+            sev = i.get("severity", "?").upper()
+            lines.append(f"- **[{sev}]** {i.get('message', 'N/A')}")
+    else:
+        lines.extend(["## Issues Detected", "", "No issues found."])
+
+    if diag.get("note"):
+        lines.extend(["", f"*{diag['note']}*"])
+
+    return "\n".join(lines)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Internal model diagnostics")
+    parser.add_argument("exp_id", nargs="?", help="Experiment ID")
+    parser.add_argument("--config", default="config.yaml")
+    parser.add_argument("--log", default=DEFAULT_LOG_PATH)
+    parser.add_argument("--json", action="store_true")
+    args = parser.parse_args()
+
+    report = xray_model(exp_id=args.exp_id, config_path=args.config, log_path=args.log)
+
+    if "error" not in report:
+        filepath = save_xray_report(report)
+        print(f"Saved to {filepath}", file=sys.stderr)
+
+    if args.json:
+        print(json.dumps(report, indent=2, default=str))
+    else:
+        print(format_xray_report(report))
+
+
+if __name__ == "__main__":
+    main()

package/templates/scripts/scaffold.py

@@ -121,6 +121,9 @@ TEMPLATE_DIRS = {
         "sanity_checks.py",
         "generate_baselines.py",
         "leakage_detector.py",
+        "model_xray.py",
+        "sensitivity_analysis.py",
+        "calibration.py",
     ],
     "tests": ["__init__.py", "conftest.py"],
 }
@@ -154,6 +157,9 @@ DIRECTORIES_TO_CREATE = [
     "experiments/sanity",
     "experiments/baselines",
     "experiments/leakage",
+    "experiments/xrays",
+    "experiments/sensitivity",
+    "experiments/calibration",
     "experiments/logs",
     "models/best",
     "models/archive",

package/templates/scripts/sensitivity_analysis.py

@@ -0,0 +1,335 @@
+#!/usr/bin/env python3
+"""Hyperparameter sensitivity analysis for the autoresearch pipeline.
+
+Varies each hyperparameter individually while holding others fixed,
+measures the metric response, and ranks hyperparameters by sensitivity.
+Answers "which hyperparameters actually matter?"
+
+Usage:
+    python scripts/sensitivity_analysis.py exp-042
+    python scripts/sensitivity_analysis.py --params "learning_rate,max_depth"
+    python scripts/sensitivity_analysis.py --json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+from scripts.turing_io import load_config, load_experiments
+
+DEFAULT_LOG_PATH = "experiments/log.jsonl"
+DEFAULT_N_POINTS = 5
+SENSITIVITY_THRESHOLDS = {"HIGH": 0.02, "MED": 0.005, "LOW": 0.002}
+DEFAULT_MULTIPLIERS = [0.5, 0.75, 1.0, 1.5, 2.0]
+
+
+# --- Sweep Generation ---
+
+
+def generate_sweep(
+    param_name: str,
+    current_value: float | int,
+    n_points: int = DEFAULT_N_POINTS,
+    multipliers: list[float] | None = None,
+) -> list[dict]:
+    """Generate sweep values for a hyperparameter.
+
+    Returns list of {value, multiplier} dicts.
+    """
+    if multipliers is None:
+        multipliers = DEFAULT_MULTIPLIERS[:n_points]
+
+    points = []
+    for m in multipliers:
+        if isinstance(current_value, int):
+            val = max(1, int(current_value * m))
+        else:
+            val = current_value * m
+        points.append({
+            "value": val,
+            "multiplier": round(m, 2),
+            "is_current": abs(m - 1.0) < 0.01,
+        })
+
+    return points
+
+
+def extract_tunable_params(config: dict) -> dict:
+    """Extract tunable hyperparameters from config."""
+    hyperparams = config.get("model", {}).get("hyperparams", {})
+
+    tunable = {}
+    for key, val in hyperparams.items():
+        if isinstance(val, (int, float)) and key not in ("seed", "random_state", "verbose"):
+            tunable[key] = val
+
+    return tunable
+
+
+# --- Sensitivity Scoring ---
+
+
+def compute_sensitivity(
+    param_name: str,
+    sweep_results: list[dict],
+    primary_metric: str,
+) -> dict:
+    """Compute sensitivity score for a hyperparameter.
+
+    Args:
+        param_name: Hyperparameter name.
+        sweep_results: List of {value, metric_value} dicts.
+        primary_metric: Name of the primary metric.
+
+    Returns:
+        Sensitivity dict with score, level, range, best value, monotonicity.
+    """
+    if not sweep_results or len(sweep_results) < 2:
+        return {"param": param_name, "sensitivity": 0, "level": "NONE",
+                "reason": "Insufficient sweep data"}
+
+    values = [r.get("value") for r in sweep_results]
+    metrics = [r.get("metric_value") for r in sweep_results
+               if r.get("metric_value") is not None]
+
+    if len(metrics) < 2:
+        return {"param": param_name, "sensitivity": 0, "level": "NONE",
+                "reason": "Insufficient metric data"}
+
+    metric_range = max(metrics) - min(metrics)
+    metric_mean = np.mean(metrics)
+
+    # Normalized sensitivity
+    sensitivity = metric_range / abs(metric_mean) if metric_mean != 0 else metric_range
+
+    # Classify level
+    if sensitivity > SENSITIVITY_THRESHOLDS["HIGH"]:
+        level = "HIGH"
+    elif sensitivity > SENSITIVITY_THRESHOLDS["MED"]:
+        level = "MED"
+    elif sensitivity > SENSITIVITY_THRESHOLDS["LOW"]:
+        level = "LOW"
+    else:
+        level = "NONE"
+
+    # Check monotonicity
+    monotonic = _check_monotonicity(metrics)
+
+    # Best value
+    best_idx = np.argmax(metrics)
+    best_value = values[best_idx] if best_idx < len(values) else None
+
+    return {
+        "param": param_name,
+        "current_value": next((r["value"] for r in sweep_results if r.get("is_current")), None),
+        "sensitivity": round(float(sensitivity), 6),
+        "metric_range": round(float(metric_range), 6),
+        "metric_min": round(float(min(metrics)), 6),
+        "metric_max": round(float(max(metrics)), 6),
+        "level": level,
+        "best_value": best_value,
+        "monotonic": monotonic,
+    }
+
+
+def _check_monotonicity(values: list[float]) -> str:
+    """Check if values are monotonically increasing, decreasing, or non-monotonic."""
+    if len(values) < 2:
+        return "unknown"
+
+    diffs = [values[i + 1] - values[i] for i in range(len(values) - 1)]
+    all_pos = all(d >= 0 for d in diffs)
+    all_neg = all(d <= 0 for d in diffs)
+
+    if all_pos:
+        return "increasing"
+    elif all_neg:
+        return "decreasing"
+    else:
+        return "non_monotonic"
+
+
+def rank_sensitivities(sensitivities: list[dict]) -> list[dict]:
+    """Rank parameters by sensitivity (highest first)."""
+    return sorted(sensitivities, key=lambda s: s.get("sensitivity", 0), reverse=True)
+
+
+# --- Recommendations ---
+
+
+def generate_recommendations(ranked: list[dict]) -> list[str]:
+    """Generate tuning recommendations from sensitivity ranking."""
+    recs = []
+
+    high = [s for s in ranked if s["level"] == "HIGH"]
+    none = [s for s in ranked if s["level"] == "NONE"]
+
+    if high:
+        names = ", ".join(s["param"] for s in high)
+        recs.append(f"Focus tuning on {names}")
+
+    if none:
+        names = ", ".join(s["param"] for s in none)
+        recs.append(f"Stop tuning {names} — they don't matter for this model")
+
+    non_mono = [s for s in ranked if s.get("monotonic") == "non_monotonic" and s["level"] in ("HIGH", "MED")]
+    if non_mono:
+        for s in non_mono:
+            recs.append(f"{s['param']} has a non-monotonic relationship — there's an optimal sweet spot around {s.get('best_value')}")
+
+    return recs
+
+
+# --- Full Pipeline ---
+
+
+def sensitivity_analysis(
+    exp_id: str | None = None,
+    params: list[str] | None = None,
+    sweep_data: dict[str, list[dict]] | None = None,
+    config_path: str = "config.yaml",
+    log_path: str = DEFAULT_LOG_PATH,
+) -> dict:
+    """Run sensitivity analysis.
+
+    Args:
+        exp_id: Experiment ID to analyze.
+        params: Specific parameters to analyze.
+        sweep_data: Pre-computed sweep results {param: [{value, metric_value}]}.
+        config_path: Path to config.yaml.
+        log_path: Path to experiment log.
+
+    Returns:
+        Sensitivity analysis report.
+    """
+    config = load_config(config_path)
+    eval_cfg = config.get("evaluation", {})
+    primary_metric = eval_cfg.get("primary_metric", "accuracy")
+
+    sensitivities = []
+
+    if sweep_data:
+        # Analyze pre-computed sweep data
+        for param, results in sweep_data.items():
+            sens = compute_sensitivity(param, results, primary_metric)
+            sensitivities.append(sens)
+    else:
+        # Generate sweep plan (actual execution done by agent)
+        tunable = extract_tunable_params(config)
+        if params:
+            tunable = {k: v for k, v in tunable.items() if k in params}
+
+        if not tunable:
+            return {"error": "No tunable hyperparameters found in config"}
+
+        sweep_plans = {}
+        for param, value in tunable.items():
+            sweep_plans[param] = generate_sweep(param, value)
+
+        return {
+            "action": "plan",
+            "generated_at": datetime.now(timezone.utc).isoformat(),
+            "primary_metric": primary_metric,
+            "experiment_id": exp_id,
+            "sweep_plans": sweep_plans,
+            "n_experiments_needed": sum(len(s) for s in sweep_plans.values()),
+            "message": f"Sweep {len(sweep_plans)} parameters × {DEFAULT_N_POINTS} values each",
+        }
+
+    ranked = rank_sensitivities(sensitivities)
+    recommendations = generate_recommendations(ranked)
+
+    return {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "primary_metric": primary_metric,
+        "experiment_id": exp_id,
+        "sensitivities": ranked,
+        "recommendations": recommendations,
+    }
+
+
+# --- Report Formatting ---
+
+
+def save_sensitivity_report(report: dict, output_dir: str = "experiments/sensitivity") -> Path:
+    out_path = Path(output_dir)
+    out_path.mkdir(parents=True, exist_ok=True)
+    exp_id = report.get("experiment_id", "unknown")
+    filepath = out_path / f"{exp_id}-sensitivity.yaml"
+    with open(filepath, "w") as f:
+        yaml.dump(report, f, default_flow_style=False, sort_keys=False)
+    return filepath
+
+
+def format_sensitivity_report(report: dict) -> str:
+    if "error" in report:
+        return f"ERROR: {report['error']}"
+
+    if report.get("action") == "plan":
+        plans = report.get("sweep_plans", {})
+        lines = ["# Sensitivity Analysis Plan", "",
+                 f"**{report.get('n_experiments_needed', 0)} experiments** needed for {len(plans)} parameters", ""]
+        for param, points in plans.items():
+            vals = ", ".join(str(p["value"]) for p in points)
+            lines.append(f"- **{param}:** [{vals}]")
+        return "\n".join(lines)
+
+    metric = report.get("primary_metric", "metric")
+    exp_id = report.get("experiment_id", "?")
+
+    lines = [f"# Hyperparameter Sensitivity Analysis ({exp_id})", "",
+             f"*Generated {report.get('generated_at', 'N/A')[:19]}*", "",
+             f"| Parameter | Current | Range Tested | {metric} Range | Sensitivity |",
+             "|-----------|---------|-------------|----------------|-------------|"]
+
+    for s in report.get("sensitivities", []):
+        current = s.get("current_value", "?")
+        metric_range = f"{s['metric_min']:.4f}–{s['metric_max']:.4f}" if s.get("metric_min") is not None else "N/A"
+        sens = f"{s['level']} ({s['sensitivity']:.4f})"
+        lines.append(f"| {s['param']} | {current} | — | {metric_range} | {sens} |")
+
+    recs = report.get("recommendations", [])
+    if recs:
+        lines.extend(["", "## Recommendations", ""])
+        for r in recs:
+            lines.append(f"- {r}")
+
+    return "\n".join(lines)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Hyperparameter sensitivity analysis")
+    parser.add_argument("exp_id", nargs="?", help="Experiment ID")
+    parser.add_argument("--params", help="Comma-separated parameter names")
+    parser.add_argument("--config", default="config.yaml")
+    parser.add_argument("--log", default=DEFAULT_LOG_PATH)
+    parser.add_argument("--json", action="store_true")
+    args = parser.parse_args()
+
+    params = [p.strip() for p in args.params.split(",")] if args.params else None
+
+    report = sensitivity_analysis(
+        exp_id=args.exp_id, params=params,
+        config_path=args.config, log_path=args.log,
+    )
+
+    if "error" not in report:
+        filepath = save_sensitivity_report(report)
+        print(f"Saved to {filepath}", file=sys.stderr)
+
+    if args.json:
+        print(json.dumps(report, indent=2, default=str))
+    else:
+        print(format_sensitivity_report(report))
+
+
+if __name__ == "__main__":
+    main()