claude-turing 1.0.0 → 1.1.0

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "turing",
-  "version": "1.0.0",
-  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 14 commands, 2 specialized agents, structured experiment lifecycle with convergence detection, immutable evaluation infrastructure, novelty guard, decision synthesis, hypothesis database, and safety guardrails that separate the hypothesis space from the measurement apparatus. Inspired by Karpathy's autoresearch and the scientific method itself.",
+  "version": "1.1.0",
+  "description": "Autonomous ML research harness — the autoresearch loop as a formal protocol. 16 commands, 2 specialized agents, cost-performance frontier analysis, model cards, model registry, hypothesis database with novelty guard, anti-cheating guardrails, and the taste-leverage loop. Inspired by Karpathy's autoresearch and the scientific method itself.",
   "author": {
     "name": "pragnition"
   },

package/README.md

@@ -300,8 +300,8 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 
 | Command | What it does |
 |---------|-------------|
-| `/turing:init [--plan]` | Scaffold a new ML project. `--plan` generates a literature-grounded research plan. |
-| `/turing:train [N]` | Run the autonomous experiment loop (optional max iterations) |
+| `/turing:init [--plan]` | Scaffold a new ML project. `--plan` generates a literature-grounded research plan. Supports multiple projects in subdirectories. |
+| `/turing:train [ml/project] [N]` | Run the experiment loop. Auto-detects project from cwd or explicit path. |
 | `/turing:sweep` | Systematic hyperparameter sweep via cartesian product |
 | `/turing:status` | Quick experiment status — best model, convergence state |
 | `/turing:compare <a> <b>` | Side-by-side experiment comparison with causal analysis |
@@ -321,6 +321,7 @@ The index (`hypotheses.yaml`) is the lightweight queue. The detail files (`hypot
 | Command | What it does |
 |---------|-------------|
 | `/turing:validate [--auto]` | Check metric stability — auto-configure multi-run if noisy |
+| `/turing:card` | Generate a model card — performance, limitations, intended use, artifact contract |
 | `/turing:logbook` | Generate HTML experiment logbook |
 | `/turing:report` | Generate research report |
 | `/turing:poster` | Generate research poster |
@@ -389,6 +390,32 @@ After N experiments with no meaningful improvement, the agent stops and reports
 
 For noisy metrics, `/turing:validate` runs the pipeline multiple times and measures variance. If the coefficient of variation exceeds 5%, it auto-configures multi-run evaluation so the agent can't be rewarded for lucky single runs.
 
+## Cost-Performance Frontier
+
+> *"This model is 2% better but takes 10x longer to train. Is that worth it?"*
+
+The briefing now surfaces [Pareto-optimal](https://en.wikipedia.org/wiki/Pareto_efficiency) experiments — the efficient set where no other experiment is both faster AND has a better metric. The cost report tells you the tradeoff in plain language:
+
+```
+Best metric: exp-012 (accuracy=0.893, 2400s)
+Best efficiency: exp-003 (accuracy=0.871, 3s)
+The 2.5% improvement costs 800x more compute.
+```
+
+Run `python scripts/cost_frontier.py` directly, or read the "Cost-Performance Analysis" section in `/turing:brief`.
+
+## Model Cards
+
+When it's time to ship, `/turing:card` generates a standardized model card documenting:
+- Model type, framework, training time
+- Performance metrics (all configured metrics)
+- Training data source and split ratios
+- Limitations (including overfit detection)
+- Intended use and ethical considerations (user fills these in)
+- Artifact contract version for production consumers
+
+Inspired by [Google's Model Cards](https://arxiv.org/abs/1810.03993) and [Hugging Face model cards](https://huggingface.co/docs/hub/model-cards).
+
 ## Installation
 
 ```bash
@@ -404,15 +431,27 @@ claude plugin add /path/to/turing
 ### Quick Start
 
 ```bash
-/turing:init            # Scaffold project (answer 3 prompts)
-/turing:train           # Run experiment loop
-/turing:brief           # Read what happened
-/turing:try "idea"      # Inject your taste
+/turing:init                    # Scaffold project (answer 3 prompts)
+/turing:train                   # Run experiment loop
+/turing:brief                   # Read what happened
+/turing:try "idea"              # Inject your taste
 ```
 
+### Multiple Projects
+
+```bash
+/turing:init                    # Scaffold ml/sentiment
+/turing:init                    # Scaffold ml/churn
+/turing:train ml/sentiment      # Train in specific project
+/turing:brief ml/churn          # Brief for specific project
+cd ml/sentiment && /turing:train  # Auto-detects from cwd
+```
+
+Each project gets independent config, data, experiments, models, and agent memory.
+
 ## Architecture of Turing Itself
 
-15 commands, 2 agents, 8 config files, 25 template scripts, 338 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
+16 commands, 2 agents, 8 config files, 30 template scripts, model registry, artifact contract, cost-performance frontier, model cards, 345 tests, 16 ADRs. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full codemap.
 
 ```
 turing/
@@ -423,6 +462,8 @@ turing/
 │   ├── prepare.py         Data loading (HIDDEN from agent)
 │   ├── evaluate.py        Evaluation harness (HIDDEN from agent)
 │   ├── train.py           Training code (AGENT-EDITABLE)
+│   ├── model_contract.md  Artifact schema for production consumers
+│   ├── model_registry.yaml  Available model architectures + hyperparams
 │   └── scripts/           25 Python scripts (core loop + analysis + infra)
 ├── tests/                 338 tests (unit + integration + anti-pattern + manifest)
 ├── src/                   5 JS installer files (npm deployment)

package/commands/brief.md

@@ -2,12 +2,24 @@
 name: brief
 description: Generate a structured research intelligence report from experiment history — what's been learned, what's promising, what's exhausted, and what the human should consider next. Use --deep for literature-grounded suggestions.
 disable-model-invocation: true
-argument-hint: "[--deep]"
+argument-hint: "[ml/project] [--deep]"
 allowed-tools: Read, Bash(python scripts/*:*, source .venv/bin/activate:*), Grep, Glob, WebSearch, WebFetch
 ---
 
 Generate a research briefing that a human can read in 2 minutes and immediately decide what to inject next.
 
+## Project Detection
+
+Before generating the briefing, detect which project to report on:
+
+0. **Detect project directory:**
+   - If `$ARGUMENTS` contains a path (e.g., `ml/coding`), use that as the project directory
+   - Else if cwd contains `config.yaml` and `train.py`, use cwd
+   - Else search for `ml/*/` subdirectories containing `config.yaml`
+     - If exactly one found, use it
+     - If multiple found, list them and ask the user which to report on
+   - All subsequent commands run from the detected project directory
+
 ## Steps
 
 1. **Generate the briefing:**

package/commands/card.md

@@ -0,0 +1,36 @@
+---
+name: card
+description: Generate a standardized model card documenting the trained model — type, performance, training data, limitations, intended use, and artifact contract.
+disable-model-invocation: true
+allowed-tools: Read, Bash(python scripts/*:*, source .venv/bin/activate:*), Grep, Glob
+---
+
+You generate a standardized model card from the experiment log, model contract, and config.
+
+## Steps
+
+1. **Activate the virtual environment:**
+   ```bash
+   source .venv/bin/activate
+   ```
+
+2. **Run the model card generator:**
+   ```bash
+   python scripts/generate_model_card.py --config config.yaml --log experiments/log.jsonl --contract model_contract.md --output MODEL_CARD.md
+   ```
+
+3. **Read and present the generated card:**
+   - Read `MODEL_CARD.md` and display it to the user.
+   - If no experiments exist yet, inform the user and show the skeleton card.
+
+4. **Suggest next steps:**
+   - Review the **Ethical Considerations** section and fill in bias, fairness, and impact notes.
+   - Review the **Intended Use** section and document what the model is NOT intended for.
+   - If limitations mention overfitting, suggest running `/turing:validate` for stability checks.
+   - If the card looks complete, suggest committing it to version control.
+
+## Error Handling
+
+- If `config.yaml` is missing, tell the user to run `/turing:init` first.
+- If `experiments/log.jsonl` is missing or empty, generate a skeleton card and note that training is needed.
+- If `.venv` doesn't exist, try `python3 scripts/generate_model_card.py` directly.

package/commands/init.md

@@ -121,3 +121,16 @@ If `$ARGUMENTS` contains `--plan`, generate a research plan AFTER scaffolding. T
 ### Integration
 
 The agent's `program.md` OBSERVE step reads `RESEARCH_PLAN.md` (if it exists) for strategic direction. The plan is advisory — the agent can deviate but should note why in `experiment_state.yaml`.
+
+## Multiple Projects
+
+You can scaffold multiple ML projects in the same repository:
+
+```bash
+/turing:init    # First project: prompts for ml_dir (e.g., ml/sentiment)
+/turing:init    # Second project: prompts for ml_dir (e.g., ml/churn)
+```
+
+Each project gets its own directory with independent config, data, experiments, and models. `/turing:train ml/sentiment` or `/turing:train ml/churn` targets a specific project. If you `cd ml/sentiment` first, `/turing:train` auto-detects from cwd.
+
+Agent memory is scoped per project: `.claude/agent-memory/ml-researcher-{project_name}/MEMORY.md`

package/commands/train.md

@@ -12,16 +12,25 @@ Read `program.md` in the ML project directory for the complete protocol. Follow
 
 ## Arguments
 
-`$ARGUMENTS` — if a number, use as max_iterations (stop after N experiments). If empty, run until convergence (as defined in `config.yaml` convergence settings).
+`$ARGUMENTS` — accepts a project path (e.g., `ml/coding`), a number for max_iterations, or both (e.g., `ml/coding 10`). If no number, run until convergence (as defined in `config.yaml` convergence settings).
 
 ## Bootstrap Sequence
 
-0. **Restore memory:** Read `.claude/agent-memory/ml-researcher/MEMORY.md` for prior observations and best results.
-1. **Read protocol:** Read `program.md` completely — it defines the experiment loop, constraints, and output format.
-2. **Bootstrap data:** Check for training data at `config.yaml` → `data.source`. If no splits exist, run `python prepare.py`.
-3. **Bootstrap venv:** `test -d .venv || (python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt)`
-4. **Assess state:** `source .venv/bin/activate && python scripts/show_metrics.py --last 5`
-5. **Begin the loop** from program.md.
+0. **Detect project directory:**
+   - If `$ARGUMENTS` contains a path (e.g., `ml/coding`), use that as the project directory
+   - Else if cwd contains `config.yaml` and `train.py`, use cwd
+   - Else search for `ml/*/` subdirectories containing `config.yaml`
+     - If exactly one found, use it
+     - If multiple found, list them and ask the user which to target
+   - All subsequent commands run from the detected project directory
+   - Memory path: `.claude/agent-memory/ml-researcher-{project_name}/MEMORY.md`
+
+1. **Restore memory:** Read `.claude/agent-memory/ml-researcher-{project_name}/MEMORY.md` for prior observations and best results.
+2. **Read protocol:** Read `program.md` completely — it defines the experiment loop, constraints, and output format.
+3. **Bootstrap data:** Check for training data at `config.yaml` → `data.source`. If no splits exist, run `python prepare.py`.
+4. **Bootstrap venv:** `test -d .venv || (python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt)`
+5. **Assess state:** `source .venv/bin/activate && python scripts/show_metrics.py --last 5`
+6. **Begin the loop** from program.md.
 
 ## The Loop
 

package/commands/turing.md

@@ -9,7 +9,7 @@ You are the Turing ML research router. Detect the user's intent and route to the
 
 | User says... | Route to | Lifecycle phase |
 |---|---|---|
-| "train", "run experiments", "autoresearch", "improve the model", "start training" | `/turing:train` | Execute |
+| "train", "train ml/coding", "train ml/claims", "run experiments", "run experiments in ml/X", "autoresearch", "improve the model", "start training" | `/turing:train` | Execute |
 | "status", "how's training", "experiment results", "current metrics" | `/turing:status` | Observe |
 | "compare", "diff runs", "which is better" | `/turing:compare` | Analyze |
 | "sweep", "grid search", "hyperparameter search", "tune" | `/turing:sweep` | Explore |
@@ -24,12 +24,13 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | "design", "plan experiment", "how should I test", "experiment design" | `/turing:design` | Design |
 | "mode", "explore", "exploit", "replicate", "strategy" | `/turing:mode` | Strategy |
 | "preflight", "resources", "VRAM", "memory", "can I run", "OOM", "GPU" | `/turing:preflight` | Check |
+| "card", "model card", "document model", "model documentation" | `/turing:card` | Document |
 
 ## Sub-commands
 
 | Command | Purpose | Agent |
 |---|---|---|
-| `/turing:train [N]` | Run the autonomous experiment loop | @ml-researcher |
+| `/turing:train [ml/project] [N]` | Run the autonomous experiment loop (auto-detects project from path or cwd) | @ml-researcher |
 | `/turing:status` | Show experiment status, best model, convergence | @ml-evaluator |
 | `/turing:compare <a> <b>` | Side-by-side experiment comparison | @ml-evaluator |
 | `/turing:sweep` | Generate and run hyperparameter sweep | @ml-researcher |
@@ -44,6 +45,7 @@ You are the Turing ML research router. Detect the user's intent and route to the
 | `/turing:report` | Structured markdown research report | (inline) |
 | `/turing:mode <mode>` | Set research strategy (explore/exploit/replicate) | (inline) |
 | `/turing:preflight` | Pre-flight resource check (VRAM/RAM/disk) | (inline) |
+| `/turing:card` | Generate standardized model card (type, performance, data, limitations, contract) | (inline) |
 
 ## Proactive Detection
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-turing",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "type": "module",
   "description": "Autonomous ML research harness for Claude Code. The autoresearch loop as a formal protocol — iteratively trains, evaluates, and improves ML models with structured experiment tracking, convergence detection, immutable evaluation infrastructure, and safety guardrails.",
   "bin": {

package/src/install.js

@@ -22,7 +22,7 @@ const PLUGIN_ROOT = join(__dirname, "..");
 const SUB_COMMANDS = [
   "init", "train", "status", "compare", "sweep", "validate",
   "try", "brief", "suggest", "design", "logbook", "poster",
-  "report", "mode", "preflight",
+  "report", "mode", "preflight", "card",
 ];
 
 export async function install(opts = {}) {

package/src/verify.js

@@ -29,6 +29,7 @@ const EXPECTED_COMMANDS = [
   "report/SKILL.md",
   "mode/SKILL.md",
   "preflight/SKILL.md",
+  "card/SKILL.md",
 ];
 
 const EXPECTED_AGENTS = ["ml-researcher.md", "ml-evaluator.md"];

package/templates/model_contract.md

@@ -0,0 +1,49 @@
+# Model Artifact Contract
+
+Version: 1
+Last updated: {{PROJECT_NAME}} initial scaffold
+
+## Bundle Format
+
+The trained model is saved as a joblib bundle at `models/best/model.joblib` containing:
+
+```python
+{
+    "model": <fitted model object>,
+    "featurizer": <fitted CompositeFeaturizer>,
+    "config": <dict of training config>,
+    "contract_version": 1
+}
+```
+
+## Metadata
+
+`models/best/metadata.json` contains:
+
+```json
+{
+    "contract_version": 1,
+    "model_type": "xgboost",
+    "experiment_id": "exp-001",
+    "metrics": {"{{TARGET_METRIC}}": 0.0},
+    "feature_names": [],
+    "created_at": "ISO-8601"
+}
+```
+
+## Consumer Contract
+
+Any service loading this model expects:
+- `bundle["model"]` has a `.predict()` method accepting a feature matrix
+- `bundle["featurizer"]` has a `.transform(df)` method returning a DataFrame
+- `bundle.get("contract_version", 0)` must equal 1
+
+If `contract_version` doesn't match, the consumer should log a warning and fall back to a default/rules-based approach.
+
+## Breaking Changes
+
+Increment `contract_version` when changing:
+- Feature schema (different featurizer output shape)
+- Label encoding (different label_map)
+- Bundle key names
+- Model input/output format

package/templates/model_registry.yaml

@@ -0,0 +1,69 @@
+# Model Registry for {{PROJECT_NAME}}
+#
+# Catalog of available model architectures. The agent reads this
+# during /turing:suggest and archetype:model_comparison to know
+# what models are available and how to configure them.
+#
+# Add your domain-specific models here.
+
+models:
+  xgboost:
+    name: "XGBoost Classifier"
+    family: "gradient_boosting"
+    task: "{{TARGET_METRIC}}"
+    notes: "Default model. Good for tabular data with mixed feature types."
+    paper: "https://arxiv.org/abs/1603.02754"
+    default_hyperparams:
+      n_estimators: 100
+      max_depth: 4
+      learning_rate: 0.1
+      objective: "multi:softmax"
+
+  lightgbm:
+    name: "LightGBM Classifier"
+    family: "gradient_boosting"
+    task: "{{TARGET_METRIC}}"
+    notes: "Often faster than XGBoost. Leaf-wise growth. Try dart boosting for regularization."
+    paper: "https://papers.nips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html"
+    default_hyperparams:
+      n_estimators: 100
+      max_depth: -1
+      learning_rate: 0.1
+      num_leaves: 31
+
+  random_forest:
+    name: "Random Forest Classifier"
+    family: "ensemble"
+    task: "{{TARGET_METRIC}}"
+    notes: "Bagging ensemble. Good baseline. Less prone to overfitting than single trees."
+    default_hyperparams:
+      n_estimators: 100
+      max_depth: null
+      min_samples_split: 2
+
+  logistic_regression:
+    name: "Logistic Regression"
+    family: "linear"
+    task: "{{TARGET_METRIC}}"
+    notes: "Simple linear baseline. Always try this first — if it works well, your features are strong."
+    default_hyperparams:
+      C: 1.0
+      max_iter: 1000
+
+  mlp:
+    name: "Multi-Layer Perceptron"
+    family: "neural_network"
+    task: "{{TARGET_METRIC}}"
+    notes: "Simple neural network. Try when samples > 2000. Needs feature scaling."
+    default_hyperparams:
+      hidden_layer_sizes: [100, 50]
+      learning_rate_init: 0.001
+      max_iter: 200
+
+# Add your domain-specific models below:
+# e.g., for NLP tasks:
+#   bert-base:
+#     name: "BERT Base"
+#     family: "transformer"
+#     hf_id: "bert-base-uncased"
+#     ...

package/templates/program.md

@@ -71,6 +71,8 @@ The autoresearch experiment loop. Each iteration is one experiment — one hypot
    cat RESEARCH_PLAN.md 2>/dev/null || true
    ```
 
+   Read `model_registry.yaml` to know what model architectures are available, their default hyperparameters, and family groupings. Use this to inform model selection during suggest and model_comparison archetypes.
+
    If `RESEARCH_PLAN.md` exists, use it for strategic direction (which model families to explore, in what order, what budget). The plan is advisory — deviate if evidence warrants, but note why.
 
    For the most recent discarded experiments, read the actual git diff to understand what was tried and failed — do NOT rely on your own memory of what you changed: