neuropt 0.3.0__tar.gz

neuropt-0.3.0/.claude/settings.local.json

@@ -0,0 +1,13 @@
+{
+  "permissions": {
+    "allow": [
+      "WebSearch",
+      "Bash(grep -E \"\\\\.\\(png|gif|ipynb|jsonl|lock\\)$\")",
+      "Bash(.venv/bin/python -c \"import torch; print\\('torch', torch.__version__\\); import torchvision; print\\('torchvision', torchvision.__version__\\)\")",
+      "Bash(.venv/bin/python -c \"import transformers; print\\('transformers', transformers.__version__\\)\" 2>&1)",
+      "Bash(.venv/bin/python -c \"import mlx; print\\(mlx.__version__\\)\")",
+      "Bash(.venv/bin/python -c \"import anthropic; print\\(anthropic.__version__\\)\")",
+      "Bash(git config:*)"
+    ]
+  }
+}

neuropt-0.3.0/.github/workflows/ci.yml

@@ -0,0 +1,48 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Import check
+        run: |
+          uv run python -c "from neuropt import ArchSearch, LogUniform, Uniform, IntUniform, Categorical"
+          uv run python -c "from neuropt.backends import get_default_backend, get_backend_by_name"
+          uv run python -c "from neuropt.introspect import introspect"
+          uv run python -c "from neuropt.cli import app"
+
+      - name: Smoke test (random search, no LLM)
+        run: |
+          uv run python -c "
+          from neuropt import ArchSearch
+          search = ArchSearch(
+              train_fn=lambda cfg: {'score': cfg['lr'] * 10},
+              search_space={'lr': (1e-4, 1e-1), 'x': [1, 2, 3]},
+              backend='none',
+              log_path='/tmp/ci_test.jsonl',
+          )
+          search.run(max_evals=3)
+          assert search.best_score < 1.0
+          print('OK')
+          "

neuropt-0.3.0/.gitignore

@@ -0,0 +1,35 @@
+# Python
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+# Data & results
+data/
+*.jsonl
+benchmark_results.json
+*.tsv
+*.lock
+results/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+
+# Build artifacts
+site/
+
+# OS
+.DS_Store
+
+# Root-level scratch files (old prototypes)
+/log_vs_linear_sampling.png
+/pso_results.png
+/pso_lightning_chat.ipynb
+/sa_lightning_chat.ipynb

neuropt-0.3.0/.python-version

@@ -0,0 +1 @@
+3.13

neuropt-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,37 @@
+# Contributing to neuropt
+
+Thanks for your interest! Here are some ways to help:
+
+## Good first contributions
+
+- **Try it on a new dataset/model** and share your results in an issue
+- **Add a new LLM backend** — see `neuropt/backends/` for the pattern (just implement `generate` and `is_available`)
+- **Improve the prompt** — the system prompt in `arch_search.py` can always be better. If you find phrasing that gets better results, open a PR
+- **Add tests** — we need them, especially for config validation and dedup logic
+
+## Bigger ideas
+
+- **Multi-objective optimization** — optimize for accuracy AND inference speed simultaneously
+- **Model introspection for more layer types** — currently we detect activations, dropout, and batch norm. Width multipliers, attention heads, and other structural changes would be great
+- **Better local LLM support** — the Qwen backend works but has a high parse failure rate on complex search spaces. Prompt engineering or constrained decoding could help
+
+## Setup
+
+```bash
+git clone https://github.com/loevlie/neuropt.git
+cd neuropt
+uv sync
+uv run neuropt --help
+```
+
+## Running tests
+
+```bash
+uv run pytest
+```
+
+## PR guidelines
+
+- Keep changes focused — one thing per PR
+- Add a test if you're changing logic
+- Run `uv run neuropt run examples/train_fashion.py --backend none -n 3` to verify nothing is broken

neuropt-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dennis Loevlie
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

neuropt-0.3.0/PKG-INFO

@@ -0,0 +1,145 @@
+Metadata-Version: 2.4
+Name: neuropt
+Version: 0.3.0
+Summary: LLM-guided ML optimization — point it at a training script, it reads the curves and designs better models
+Project-URL: Homepage, https://github.com/loevlie/neuropt
+Project-URL: Repository, https://github.com/loevlie/neuropt
+Project-URL: Issues, https://github.com/loevlie/neuropt/issues
+Author: Dennis Loevlie
+License: MIT
+License-File: LICENSE
+Keywords: architecture-search,automl,hyperparameter,llm,machine-learning,neural-architecture-search,optimization,pytorch,training-curves
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: cloudpickle>=3.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: typer>=0.9
+Provides-Extra: all
+Requires-Dist: accelerate>=0.25; extra == 'all'
+Requires-Dist: anthropic>=0.40; extra == 'all'
+Requires-Dist: matplotlib>=3.7; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Requires-Dist: pandas>=2.0; extra == 'all'
+Requires-Dist: torch>=2.0; extra == 'all'
+Requires-Dist: torchvision>=0.15; extra == 'all'
+Requires-Dist: transformers>=4.37; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: optuna; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Provides-Extra: llm
+Requires-Dist: anthropic>=0.40; extra == 'llm'
+Provides-Extra: llm-local
+Requires-Dist: accelerate>=0.25; extra == 'llm-local'
+Requires-Dist: transformers>=4.37; extra == 'llm-local'
+Provides-Extra: llm-openai
+Requires-Dist: openai>=1.0; extra == 'llm-openai'
+Provides-Extra: torch
+Requires-Dist: torch>=2.0; extra == 'torch'
+Requires-Dist: torchvision>=0.15; extra == 'torch'
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.7; extra == 'viz'
+Requires-Dist: pandas>=2.0; extra == 'viz'
+Description-Content-Type: text/markdown
+
+# neuropt
+
+<p align="center">
+  <img src="assets/banner.png" alt="Three robot researchers designing neural network architectures" width="700">
+</p>
+
+<p align="center">
+  <em>An LLM reads your training curves and designs your next experiment.</em>
+</p>
+
+---
+
+Point it at a training script, let it run overnight. The LLM sees full per-epoch train/val curves, spots overfitting, and proposes what to try next — like a research assistant who never sleeps and actually reads the loss plots.
+
+### vs Optuna and random search
+
+<p align="center">
+  <img src="assets/benchmark.png" alt="Benchmark: neuropt vs Optuna vs Random" width="700">
+</p>
+
+Same 15-eval budget, 14-parameter CNN search space. These results use **Claude Haiku 4.5** (the smallest and cheapest of their 4.5 models). We expect even stronger results with Sonnet or Opus. Optuna's TPE was configured with `n_startup_trials=3` for a fair comparison (default is 10, which would make it purely random for most of the budget).
+
+## Quick start
+
+```bash
+pip install neuropt[llm]
+export ANTHROPIC_API_KEY="sk-ant-..."
+```
+
+**Option 1** — define what to search over:
+
+```python
+# train.py
+search_space = {
+    "lr": (1e-4, 1e-1),                    # auto-detects log-scale
+    "hidden_dim": (32, 512),                # auto-detects integer
+    "activation": ["relu", "gelu", "silu"], # categorical
+}
+
+def train_fn(config):
+    model = build_my_model(config["hidden_dim"], config["activation"])
+    # ... train, return per-epoch losses for smarter LLM decisions ...
+    return {"score": val_loss, "train_losses": [...], "val_losses": [...]}
+```
+
+**Option 2** — just give it a model, we figure out the rest:
+
+```python
+# train.py
+model = torchvision.models.resnet18(num_classes=10)  # neuropt introspects this
+
+def train_fn(config):
+    m = config["model"].to("cuda")  # deep copy with modifications applied
+    # ... train ...
+    return {"score": val_loss, "train_losses": [...], "val_losses": [...]}
+```
+
+Then run:
+
+```bash
+neuropt run train.py
+```
+
+Runs until Ctrl+C. Crash-safe, resumable. Works in notebooks too:
+
+```python
+from neuropt import ArchSearch
+
+search = ArchSearch(train_fn=train_fn, search_space=search_space, backend="claude")
+search.run(max_evals=50)
+```
+
+## Documentation
+
+See the [full documentation](https://loevlie.github.io/neuropt/) for:
+
+- [How it works](https://loevlie.github.io/neuropt/how-it-works/) — what the LLM sees, training curve analysis
+- [CLI reference](https://loevlie.github.io/neuropt/cli/) — `neuropt run`, `inspect`, `results`
+- [Python API](https://loevlie.github.io/neuropt/api/) — `ArchSearch`, `from_model`, search space types
+- [Examples](https://loevlie.github.io/neuropt/examples/) — CNN search, ResNet tuning
+- [Benchmarks](https://loevlie.github.io/neuropt/benchmarks/) — vs Optuna, random search
+
+## Installation
+
+```bash
+pip install neuropt                # core
+pip install neuropt[llm]           # + Claude API (recommended)
+pip install neuropt[llm-openai]    # + OpenAI API
+pip install neuropt[all]           # everything
+```
+
+## License
+
+MIT

neuropt-0.3.0/README.md

@@ -0,0 +1,94 @@
+# neuropt
+
+<p align="center">
+  <img src="assets/banner.png" alt="Three robot researchers designing neural network architectures" width="700">
+</p>
+
+<p align="center">
+  <em>An LLM reads your training curves and designs your next experiment.</em>
+</p>
+
+---
+
+Point it at a training script, let it run overnight. The LLM sees full per-epoch train/val curves, spots overfitting, and proposes what to try next — like a research assistant who never sleeps and actually reads the loss plots.
+
+### vs Optuna and random search
+
+<p align="center">
+  <img src="assets/benchmark.png" alt="Benchmark: neuropt vs Optuna vs Random" width="700">
+</p>
+
+Same 15-eval budget, 14-parameter CNN search space. These results use **Claude Haiku 4.5** (the smallest and cheapest of their 4.5 models). We expect even stronger results with Sonnet or Opus. Optuna's TPE was configured with `n_startup_trials=3` for a fair comparison (default is 10, which would make it purely random for most of the budget).
+
+## Quick start
+
+```bash
+pip install neuropt[llm]
+export ANTHROPIC_API_KEY="sk-ant-..."
+```
+
+**Option 1** — define what to search over:
+
+```python
+# train.py
+search_space = {
+    "lr": (1e-4, 1e-1),                    # auto-detects log-scale
+    "hidden_dim": (32, 512),                # auto-detects integer
+    "activation": ["relu", "gelu", "silu"], # categorical
+}
+
+def train_fn(config):
+    model = build_my_model(config["hidden_dim"], config["activation"])
+    # ... train, return per-epoch losses for smarter LLM decisions ...
+    return {"score": val_loss, "train_losses": [...], "val_losses": [...]}
+```
+
+**Option 2** — just give it a model, we figure out the rest:
+
+```python
+# train.py
+model = torchvision.models.resnet18(num_classes=10)  # neuropt introspects this
+
+def train_fn(config):
+    m = config["model"].to("cuda")  # deep copy with modifications applied
+    # ... train ...
+    return {"score": val_loss, "train_losses": [...], "val_losses": [...]}
+```
+
+Then run:
+
+```bash
+neuropt run train.py
+```
+
+Runs until Ctrl+C. Crash-safe, resumable. Works in notebooks too:
+
+```python
+from neuropt import ArchSearch
+
+search = ArchSearch(train_fn=train_fn, search_space=search_space, backend="claude")
+search.run(max_evals=50)
+```
+
+## Documentation
+
+See the [full documentation](https://loevlie.github.io/neuropt/) for:
+
+- [How it works](https://loevlie.github.io/neuropt/how-it-works/) — what the LLM sees, training curve analysis
+- [CLI reference](https://loevlie.github.io/neuropt/cli/) — `neuropt run`, `inspect`, `results`
+- [Python API](https://loevlie.github.io/neuropt/api/) — `ArchSearch`, `from_model`, search space types
+- [Examples](https://loevlie.github.io/neuropt/examples/) — CNN search, ResNet tuning
+- [Benchmarks](https://loevlie.github.io/neuropt/benchmarks/) — vs Optuna, random search
+
+## Installation
+
+```bash
+pip install neuropt                # core
+pip install neuropt[llm]           # + Claude API (recommended)
+pip install neuropt[llm-openai]    # + OpenAI API
+pip install neuropt[all]           # everything
+```
+
+## License
+
+MIT

neuropt-0.3.0/docs/api.md

@@ -0,0 +1,122 @@
+# Python API
+
+## ArchSearch
+
+The main class. Use it directly in notebooks or scripts.
+
+### From a search space
+
+```python
+from neuropt import ArchSearch
+
+search = ArchSearch(
+    train_fn=train_fn,
+    search_space={
+        "lr": (1e-4, 1e-1),
+        "n_layers": (2, 8),
+        "activation": ["relu", "gelu", "silu"],
+        "use_bn": [True, False],
+    },
+    backend="claude",
+)
+search.run(max_evals=50)
+
+print(search.best_config)
+print(search.best_score)
+```
+
+### From a model
+
+```python
+from neuropt import ArchSearch
+
+search = ArchSearch.from_model(
+    model=my_model,
+    train_fn=train_fn,
+    backend="claude",
+)
+search.run(max_evals=50)
+```
+
+`from_model` introspects the module tree, finds activations/dropout/batch norm, generates a search space, and wraps your `train_fn` so `config["model"]` contains the modified deep copy.
+
+### Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `train_fn` | required | `config dict → result dict` |
+| `search_space` | required | Dict of param names to ranges/choices |
+| `backend` | `"auto"` | `"auto"`, `"claude"`, `"openai"`, `"qwen"`, `"none"` |
+| `log_path` | `"search.jsonl"` | JSONL log file |
+| `batch_size` | `3` | Configs per LLM call |
+| `device` | `None` | Injected as `config["device"]` |
+| `timeout` | `600` | Max seconds per experiment |
+| `ml_context` | generic | Domain knowledge for the LLM |
+
+### `run(max_evals=None)`
+
+Runs the search loop. If `max_evals` is set, stops after that many experiments. Otherwise runs until Ctrl+C.
+
+### Result attributes
+
+After `run()` completes:
+
+- `search.best_score` — lowest score seen
+- `search.best_config` — config dict that produced it
+- `search.best_accuracy` — accuracy of best config (if returned)
+- `search.total_experiments` — total experiments run
+- `search.llm_success` — LLM calls that produced valid configs
+- `search.llm_fallback` — LLM calls that fell back to random
+
+## train_fn contract
+
+Your function receives a config dict and returns a result dict.
+
+**Required return key:**
+
+- `"score"` — float, lower is better
+
+**Optional return keys (recommended):**
+
+- `"train_losses"` — list of per-epoch training losses
+- `"val_losses"` — list of per-epoch validation losses
+- `"val_accuracies"` — list of per-epoch validation accuracies
+- `"accuracy"` — final accuracy
+- `"n_params"` — model parameter count
+
+The per-epoch lists are what give the LLM its advantage — it can spot overfitting, underfitting, and learning rate issues from the curve shapes.
+
+## Search space types
+
+You can use plain Python types (auto-inferred) or explicit dimension objects.
+
+### Auto-inference from tuples and lists
+
+```python
+search_space = {
+    "lr": (1e-4, 1e-1),              # → LogUniform (name-based)
+    "wd": (1e-6, 1e-2),              # → LogUniform (name-based)
+    "dropout": (0.0, 0.5),           # → Uniform
+    "n_layers": (2, 8),              # → IntUniform (name + int values)
+    "hidden_dim": (32, 512),          # → IntUniform (name + int values)
+    "activation": ["relu", "gelu"],   # → Categorical
+    "use_bn": [True, False],          # → Categorical
+}
+```
+
+Names like `lr`, `learning_rate`, `wd`, `weight_decay` automatically get log-scale sampling. Names like `n_layers`, `hidden_dim`, `num_heads` get integer sampling. Integer tuple values also trigger IntUniform.
+
+### Explicit dimension objects
+
+For full control over ranges and sampling:
+
+```python
+from neuropt import LogUniform, Uniform, IntUniform, Categorical
+
+search_space = {
+    "lr": LogUniform(1e-4, 1e-1),
+    "momentum": Uniform(0.8, 0.99),
+    "depth": IntUniform(2, 8),
+    "optimizer": Categorical(["sgd", "adam", "adamw"]),
+}
+```

neuropt-0.3.0/docs/benchmarks.md

@@ -0,0 +1,56 @@
+# Benchmarks
+
+## 15-eval benchmark (14-parameter CNN search)
+
+All methods get exactly 15 evaluations on the same search space. FashionMNIST, 5 epochs per eval, M1 MacBook.
+
+![Benchmark results](../assets/benchmark.png)
+
+| Method | Best Loss | Best Acc | LLM Fallbacks |
+|--------|-----------|----------|---------------|
+| **LLM (Claude)** | **0.385** | **85.4%** | 0/5 |
+| Optuna TPE | 0.454 | 82.7% | — |
+| Random Search | 0.610 | 76.7% | — |
+| LLM (Qwen local) | 0.637 | 75.0% | 2/5 |
+
+### Convergence
+
+| Eval | LLM (Claude) | Optuna TPE | Random | LLM (Qwen) |
+|------|-------------|------------|--------|-------------|
+| 5 | **0.401** | 0.612 | 0.655 | 0.748 |
+| 10 | **0.399** | 0.454 | 0.655 | 0.637 |
+| 15 | **0.385** | 0.454 | 0.610 | 0.637 |
+
+Claude was ahead of Optuna from eval 5 onward — it started with good architectural priors (residual connections, AdamW, reasonable LR) instead of discovering them through trial and error.
+
+All benchmark results used **Claude Haiku 4.5** (the smallest, cheapest Claude model for pennies per run). We expect stronger results with Sonnet or Opus, which have better reasoning capabilities for complex search spaces.
+
+### What went wrong with the other methods
+
+**Optuna TPE** — 7 out of 15 evals scored above 1.0 (worse than random chance). With 14 parameters, TPE's surrogate model needs many more samples before it becomes useful. It found one good config at eval 8 but couldn't build on it.
+
+**Random search** — Actually outperformed Qwen, which says more about Qwen's parse failures than random's quality. Random got lucky with a few configs but has no mechanism to improve.
+
+**LLM (Qwen local)** — Failed to produce valid JSON on 2 of 5 iterations (40% fallback rate). When it did generate configs, they were reasonable but not as focused as Claude's. The local Qwen backend is experimental — it works for simpler search spaces but struggles with 14-key JSON output.
+
+### Why Claude wins on this search space
+
+With 14 parameters including categorical choices (activation, optimizer, pool type, residual on/off), the search space has complex interactions:
+
+- High LR + no batch norm = training instability
+- Deep networks + no residual = vanishing gradients
+- Dropout + small dataset subset = unnecessary regularization
+
+Claude starts with knowledge of these interactions. Optuna has to discover each one empirically, burning evals on configurations that any ML practitioner would avoid.
+
+### Run it yourself
+
+```bash
+pip install neuropt[llm]
+export ANTHROPIC_API_KEY="sk-ant-..."
+
+python examples/benchmark.py
+python examples/benchmark.py --n-evals 30          # longer run
+python examples/benchmark.py --skip-qwen           # skip local model
+python examples/benchmark.py --skip-qwen --n-evals 50  # thorough comparison
+```

neuropt-0.3.0/docs/cli.md

@@ -0,0 +1,61 @@
+# CLI Reference
+
+## `neuropt run`
+
+Run LLM-guided search on a training script.
+
+```bash
+neuropt run train.py
+neuropt run train.py --backend claude
+neuropt run train.py --backend none -n 50 --log results.jsonl
+```
+
+Your script must define:
+
+- `train_fn(config)` — returns dict with at least `{"score": float}`
+- `search_space` dict **or** `model` (nn.Module) — one of the two
+
+Optional: `ml_context` string with domain knowledge for the LLM.
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--backend` | auto | `auto`, `claude`, `openai`, `qwen`, `none` |
+| `--log` | search.jsonl | Log file path (supports resume) |
+| `-b` / `--batch-size` | 3 | Configs proposed per LLM call |
+| `-n` / `--max-evals` | unlimited | Stop after N experiments |
+| `--device` | auto | `cuda`, `mps`, `cpu` |
+| `--timeout` | 600 | Max seconds per experiment |
+
+## `neuropt inspect`
+
+Show what neuropt would search over for a given model.
+
+```bash
+neuropt inspect train.py
+```
+
+```
+Model: 11,689,512 parameters
+  Activations: ReLU (9 layers)
+  BatchNorm: 20 layers
+
+Search space (5 params):
+  activation: Categorical(['relu', 'gelu', 'silu', 'leaky_relu'])
+  use_batchnorm: Categorical([True, False])
+  lr: LogUniform(0.0001, 0.1)
+  wd: LogUniform(1e-06, 0.01)
+  optimizer: Categorical(['sgd', 'adam', 'adamw'])
+```
+
+Only works with scripts that define a `model` variable.
+
+## `neuropt results`
+
+Analyze a search log.
+
+```bash
+neuropt results search.jsonl
+neuropt results search.jsonl --top 20
+```
+
+Shows total experiments, top N results with configs, and convergence over time.