butchc 0.6.0__tar.gz

butchc-0.6.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 BUTChC contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

butchc-0.6.0/PKG-INFO

@@ -0,0 +1,424 @@
+Metadata-Version: 2.4
+Name: butchc
+Version: 0.6.0
+Summary: Bayesian Update Tree Chained Conditionally — a dependency-free probabilistic black-box hyperparameter optimizer
+License: MIT
+Project-URL: Homepage, https://github.com/AdventuresInDataScience/BUTChC
+Project-URL: Repository, https://github.com/AdventuresInDataScience/BUTChC
+Project-URL: Documentation, https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/api.md
+Project-URL: Changelog, https://github.com/AdventuresInDataScience/BUTChC/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/AdventuresInDataScience/BUTChC/issues
+Keywords: hyperparameter optimization,bayesian optimization,black-box optimization,tree search,machine learning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Provides-Extra: configspace
+Requires-Dist: ConfigSpace>=0.6; extra == "configspace"
+Dynamic: license-file
+
+# BUTChC
+
+**B**ayesian **U**pdate **T**ree **Ch**ained **C**onditionally — a dependency-free, probabilistic black-box hyperparameter optimizer.
+
+BUTChC maintains a probability distribution over a **hierarchical, conditional search space** and refines it from observed objective values. Parameters can depend on choices made higher up the tree: `momentum` exists only when `optimizer=sgd`, `kernel_size` only when `model=cnn`. Invalid combinations are unreachable by construction rather than discovered by trial, and each branch learns its own parameters from only the trials that used it.
+
+No gradients, no differentiability, no assumptions about the objective's internals.
+
+- **Zero dependencies** — Python ≥ 3.8 standard library only
+- **Conditional search spaces** — nested arbitrarily deep via `next_level`
+- **Non-parametric continuous model** — a weighted KDE elite archive, no distributional assumptions
+- **Scale-invariant** — updates use rank, not raw objective, so an objective in the millions behaves like one in `[0, 1]`
+- **Parallel evaluation** — bring your own executor; threads, processes, joblib, dask
+- **Reproducible** — `seed` gives a private RNG, and the answer does not depend on which worker finishes first
+- **Fails fast** — search spaces and hyperparameters are validated before the first objective call
+- **Prunable** — `prune` turns a finished run into a smaller space, to search again or hand to another optimiser
+
+Against TPE across 18 benchmark problems at 30 paired seeds: **13 significant wins, zero significant losses**, at **73–145× lower per-trial cost**.
+
+📖 **[API reference](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/api.md)** · **[Examples](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/examples.md)** · **[Design notes](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/design.md)** · **[Limitations](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/limitations.md)** · **[Changelog](https://github.com/AdventuresInDataScience/BUTChC/blob/main/CHANGELOG.md)**
+
+---
+
+## Install
+
+```bash
+pip install butchc
+pip install "butchc[configspace]"    # optional ConfigSpace interop
+```
+
+---
+
+## Quick start
+
+```python
+from butchc import BUTChC_optimize
+
+searchspace = {
+    # Top-level choice of model family. Each choice unlocks its own
+    # sub-parameters via 'next_level'.
+    'model_type': {
+        'values': ['svm', 'random_forest', 'neural_net'],
+        'next_level': {
+            'svm': {
+                'kernel': {'values': ['rbf', 'linear', 'poly']},
+                'C':      {'min': 0.01, 'max': 100.0, 'log': True},
+                'gamma':  {'min': 1e-4, 'max': 10.0,  'log': True},
+            },
+            'random_forest': {
+                'n_estimators': {'values': [50, 100, 200, 500]},
+                'max_depth':    {'min': 2, 'max': 30, 'int': True},
+                'max_features': {'values': ['sqrt', 'log2']},
+            },
+            'neural_net': {
+                'learning_rate': {'min': 1e-4, 'max': 1e-1, 'log': True},
+                'hidden_units':  {'values': [64, 128, 256, 512]},
+                'dropout':       {'min': 0.0, 'max': 0.5},
+            },
+        },
+    },
+    # Always present, whatever model_type is chosen
+    'preprocessing': {'values': ['standard_scaler', 'min_max', 'none']},
+}
+
+def objective(config):
+    # config carries 'model_type' and 'preprocessing', plus only the params
+    # of the chosen branch, e.g.
+    #   {'model_type': 'svm', 'kernel': 'rbf', 'C': 4.2, 'gamma': 0.01, ...}
+    # 'dropout' never appears in an svm config; 'C' never in a neural_net one.
+    return cross_val_score(build_model(config), X, y).mean()   # higher is better
+
+results = BUTChC_optimize(
+    searchspace = searchspace,
+    objective   = objective,
+    budget      = 150,
+    seed        = 0,
+)
+
+print(results['best_params'])
+print(f"Best score: {results['best_value']:.4f}")
+```
+
+BUTChC always **maximizes**. Negate to minimize.
+
+### Slow objective? Use an executor
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+with ThreadPoolExecutor(max_workers=8) as pool:
+    results = BUTChC_optimize(searchspace, objective, budget=200,
+                              batch=8, executor=pool, seed=0)
+```
+
+### Already have a ConfigSpace?
+
+```python
+results = BUTChC_optimize(configuration_space, objective, budget=200, seed=0)
+```
+
+---
+
+## Defining a search space
+
+A search space is a plain dict — JSON-serializable, and no imports needed to write one.
+
+```python
+'activation':    {'values': ['relu', 'tanh', 'elu']}        # categorical
+'dropout':       {'min': 0.0,  'max': 0.5}                  # linear
+'learning_rate': {'min': 1e-5, 'max': 1e-1, 'log': True}    # log10-uniform
+'n_layers':      {'min': 1,    'max': 6,    'int': True}    # integer-valued
+```
+
+Use `log: True` whenever the range spans more than about one order of magnitude. Sampled linearly, `[1e-5, 1e-1]` places 99.99% of its mass above `1e-3`, leaving the bottom three decades effectively unreachable.
+
+A categorical node can map each of its choices to a sub-searchspace via `next_level`. Those parameters are sampled and updated only when their parent value is chosen:
+
+```python
+'optimizer': {
+    'values': ['adam', 'sgd', 'lbfgs'],
+    'next_level': {
+        'adam': {'lr': {'min': 1e-4, 'max': 1e-2, 'log': True}},
+        'sgd':  {'lr':       {'min': 1e-3, 'max': 1e-1, 'log': True},
+                 'momentum': {'min': 0.0,  'max': 0.99}},
+        # 'lbfgs' takes no sub-params — omitting it is fine
+    },
+}
+```
+
+Each branch keeps its own model, so learning the best `lr` for adam does not interfere with learning the best `lr` for sgd. `next_level` is the only nesting mechanism and it composes, so nesting is arbitrarily deep — [a worked four-level space](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/examples.md#nesting-more-than-one-level) shows how.
+
+This is also how you express an invalid combination. "`penalty=elasticnet` only works with `solver=saga`" becomes a `solver` node whose branches carry different `penalty` values — the invalid pairing is then unreachable, and no budget is spent finding that out. Constraints that cross the tree rather than nest are the exception; see [limitations](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/limitations.md#search-spaces-butchc-cannot-express).
+
+Any node can carry a `prior` and a `prior_strength` measured in pseudo-trials. See the [API reference](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/api.md#search-space-format) for the full format, priors, and the uniqueness rule for names.
+
+---
+
+## How it works
+
+1. **Initialize** a probability tree from the search space. Categorical nodes start uniform; continuous nodes start with an evenly spaced reservoir covering the range.
+
+2. **Sample** by traversing the tree. Categorical nodes draw from a temperature-scaled softmax. Continuous nodes pick a reservoir point weighted by its rank, add Silverman-bandwidth Gaussian jitter, and *reflect* back into range.
+
+3. **Evaluate** the objective.
+
+4. **Rank** the result against every finite objective seen so far, counting ties as half. Continuous nodes apply the `gamma` gate and convert what survives into a `quality` in `[0, 1]`; categorical nodes use the raw rank, ungated.
+
+5. **Update**. Categorical nodes score each choice by its recency-weighted mean rank, smoothed by `alpha` pseudo-visits. Continuous nodes append to an elite archive, evict the worst-scoring entry, and reweight geometrically by rank.
+
+6. **Repeat**, with categorical commitment ramping up over the budget.
+
+Two properties are load-bearing. **Reflection rather than clipping**: jitter clipped to `[min, max]` deposits probability mass on each bound and biases every search toward interval edges. **Rank rather than raw objective**: raw-value weighting makes behaviour depend on units, and one catastrophic outlier could dominate the archive permanently.
+
+Full reasoning in [design notes](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/design.md).
+
+---
+
+## Tuning
+
+| Parameter | Start | Increase if… | Decrease if… |
+|---|---|---|---|
+| `lambda_` | `2.0` | tree adapts too slowly | converging too fast to a suboptimal region |
+| `alpha` | `3.0` | many categorical options, small budget | want faster commitment to early evidence |
+| `temp` | `1.0` | categorical exploration too greedy | budget spent on clearly bad choices |
+| `gamma` | `0.85` | objective is noisy; want only strong trials to count | want more trials contributing signal |
+| `explore` | `0.05` | search collapses to a local optimum early | objective is expensive and smooth |
+| `batch` | `1` | you have idle workers | you have no executor |
+| `budget` | 10× param count | rolling loss has not plateaued | rolling loss flat after 20% of the run |
+
+`lambda_` and `alpha` act on different node types and do not interact: `lambda_` controls how sharply continuous archives concentrate, `alpha` how slowly categorical nodes commit. Note `lambda_` saturates — it acts only through `min(RANK_SHARPNESS * lambda_, MAX_SHARPNESS)`, so any value at or above 3.33 is clipped and does nothing. See the [API reference](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/api.md#butchc_optimize).
+
+The two knobs most worth reaching for are not in this table. `KDE_RESERVOIR_SIZE` (default 25) and `MIN_BANDWIDTH_FRACTION` (default 0.0003) between them decide how hard the continuous model concentrates, and they carry more of the measured gain than anything else. They interact, so retune them together: a **smooth, high-dimensional** space wants the gentler pair (`50` and `0.001`), while conditional and multimodal spaces want the sharp defaults.
+
+### Tuning for your problem's shape
+
+The defaults are an average over problem shapes. `benchmarks/tune.py --regime` (in the repository, not the installed package) sweeps against problems sharing one property and prints what that shape wants: `branched` spaces want faster commitment, `noisy` ones want `explore 0.1`, `multimodal` ones want a short warm-up. The table is in the [API reference](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/api.md#tuning-by-problem-shape).
+
+---
+
+## Results
+
+Median best objective over 30 paired seeds — same budget, same seeds, every
+method. Every problem is a maximization with optimum 0, so nearer zero is
+better (Styblinski's offset is rounded, so it can read fractionally above 0).
+Re-running the commands below from a clone of the repository reproduces every
+table in this section.
+
+```bash
+python benchmarks/evaluate.py 30
+python benchmarks/evaluate.py 10 --suite heldout --methods tpe,butchc
+```
+
+Random search is a low bar, so the comparator carried throughout is TPE — same
+niche, and what a user choosing against BUTChC would actually reach for.
+`benchmarks/baselines.py` carries a dependency-free TPE and an Optuna
+`TPESampler` wrapper. `evaluate.py` also reports paired win-loss records and
+two-sided exact sign-test p-values; those records, not the medians, are what
+the claims here rest on.
+
+The suite is split in half. Defaults were selected by coordinate descent
+against the **tuned-on** problems only; the **held-out** problems were never
+consulted during that sweep, so they are the fairer test of whether the
+defaults generalise.
+
+### Tuned on
+
+| Problem | Budget | Random | TPE | BUTChC | vs TPE |
+|---|---|---|---|---|---|
+| 2D quadratic | 200 | -0.0425 | -0.0004 | **-0.0000** | 30-0, p=0.000 |
+| 5D sphere | 500 | -3.7393 | -0.1312 | **-0.0000** | 29-1, p=0.000 |
+| Rosenbrock | 500 | -0.1036 | **-0.0202** | -0.0368 | 13-17, p=0.585 |
+| Rastrigin 4D | 600 | -17.5005 | -8.6510 | **-5.1169** | 26-4, p=0.000 |
+| 10D sphere | 1000 | -19.0750 | -2.2176 | **-0.0036** | 30-0, p=0.000 |
+| Ackley 5D | 600 | -14.1027 | -4.3008 | **-0.0323** | 30-0, p=0.000 |
+| Log-scale target | 200 | -0.0085 | -0.0006 | **-0.0000** | 29-1, p=0.000 |
+| Branch trap | 300 | -0.6219 | -1.0004 | **-0.0189** | 25-5, p=0.000 |
+| Categorical mix | 300 | -0.3226 | -0.0680 | **-0.0004** | 28-2, p=0.000 |
+| Integer mix | 300 | -0.2727 | -0.0016 | **-0.0000** | 26-4, p=0.000 |
+| Plateau (ties) | 300 | -0.5000 | -0.5000 | -0.5000 | 0-0, p=1.000 |
+
+### Held out
+
+| Problem | Budget | Random | TPE | BUTChC | vs TPE |
+|---|---|---|---|---|---|
+| Griewank 6D | 800 | -1.0183 | -0.4915 | **-0.4383** | 18-12, p=0.362 |
+| Styblinski 4D | 600 | -20.0007 | -4.0751 | **+0.0007** | 30-0, p=0.000 |
+| Nested pipeline | 400 | -0.2970 | -0.0096 | **-0.0089** | 15-15, p=1.000 |
+| Optimiser choice | 300 | -0.0277 | -0.0007 | **-0.0001** | 25-5, p=0.000 |
+| Rastrigin 8D | 1000 | -62.4076 | -39.0921 | **-21.5439** | 28-2, p=0.000 |
+| 20D sphere | 1500 | -69.0434 | -18.8961 | **-0.1255** | 30-0, p=0.000 |
+
+Four of the six held-out problems are significant wins on defaults that never
+saw them.
+
+### Under observation noise
+
+Scoring the *reported* best on the noise-free function (5D sphere, N(0,1)
+noise, budget 400): random -4.8604, TPE -0.8099, BUTChC **-0.4490** — 19-11
+against TPE at p=0.200, better on the median but not separable at 30 seeds.
+
+### What the records show
+
+**13 significant wins, zero significant losses, 5 ties**, across all 18
+problems. The margins are large where they are large: `Styblinski 4D` reaches
+the optimum outright (+0.0007 against -4.0751, 30-0), `20D sphere` lands 150×
+nearer it, `Ackley 5D` 133× nearer. `Branch trap` is 25-5 on a problem where
+TPE does *worse than random*, because the trap is precisely the conditional
+structure a flat model cannot see.
+
+The five ties are genuine non-results rather than hidden losses, and each has a
+reason; they are set out in full in
+[limitations](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/limitations.md#the-non-results-in-full).
+
+### How long a result takes to arrive
+
+Final quality says where a method ends up, not when. `evaluate.py --anytime`
+reports the other axis. The clearest framing is **how much budget BUTChC needs
+to match TPE's final answer**:
+
+| Problem | Budget | Trials BUTChC needed | Fraction of budget |
+|---|---|---|---|
+| 20D sphere | 1500 | 148 | **1/10.1** |
+| 10D sphere | 1000 | 138 | **1/7.3** |
+| Styblinski 4D | 600 | 146 | 1/4.1 |
+| Ackley 5D | 600 | 155 | 1/3.9 |
+| 5D sphere | 500 | 128 | 1/3.9 |
+| Rastrigin 8D | 1000 | 325 | 1/3.1 |
+| Integer mix | 300 | 100 | 1/3.0 |
+| Categorical mix | 300 | 107 | 1/2.8 |
+| Branch trap | 300 | 111 | 1/2.7 |
+| Nested pipeline | 400 | 324 | 1/1.2 |
+
+On every problem it reaches, BUTChC matches TPE's *final* result partway
+through its own budget — median around a third of it.
+
+Against TPE's own arrival time the picture is split. On high-dimensional
+problems BUTChC is far quicker: `20D sphere` in 148 trials against TPE's 1133,
+on 30 of 30 seeds against TPE's 15. On conditional problems it arrives later
+even while matching or beating TPE's final quality — `Nested pipeline` 324
+against 188, `Optimiser choice` 206 against 115.
+
+Reliability is consistently BUTChC's. At 50% of the achievable range it reaches
+the target on 27–30 of 30 seeds where TPE manages 10–29; at 99% on `20D sphere`
+it arrives on 29 of 30 seeds while TPE never arrives at all. Full tables in
+[`dev/tools/results/`](https://github.com/AdventuresInDataScience/BUTChC/tree/main/dev/tools/results).
+
+### What the optimiser itself costs
+
+Sample efficiency is one axis; the wall clock the optimiser spends choosing is
+another. With the objective stubbed to a constant, so the number is all
+optimiser:
+
+| Optimiser | 1D | 5D | 20D |
+|---|---|---|---|
+| **BUTChC** (pure Python, 0 deps) | **14 µs** | **52 µs** | **193 µs** |
+| TPE (this repo, pure Python) | 1037 µs | 5219 µs | 21270 µs |
+| Optuna `TPESampler` (numpy-backed) | 1433 µs | 6789 µs | 27864 µs |
+
+BUTChC is **73–145× cheaper per trial** than either TPE. Zero dependencies is
+not costing speed here: the numpy-backed implementation pays ~145× more per
+suggestion, because TPE refits Parzen estimators over the whole history and
+scores candidates, while BUTChC draws a reservoir point and jitters it —
+`O(d·K)` with `K = 25`. The gap widens with width, since BUTChC only touches
+the parameters on the sampled path while a flat model touches all of them.
+
+Reproduce with `python benchmarks/overhead.py`.
+
+### What batching costs
+
+A batch of `k` leaves the model stale for `k-1` evaluations. Median extra regret against sequential, 18 problems × 20 seeds at matched budgets:
+
+| `batch` | 2 | 4 | 8 | 16 | 32 |
+|---|---|---|---|---|---|
+| Extra regret | −3.3% | −0.1% | +2.6% | +21.3% | +43.1% |
+
+```bash
+python benchmarks/batch_cost.py 20 --sizes 1,2,4,8,16,32
+```
+
+Up to `k=8` the cost sits inside seed noise, which makes an 8× wall-clock speedup close to free. Past `k=16` it is real. Batch sizes above your worker count pay the cost for nothing.
+
+### The spaces this is built for
+
+The 18 problems above are synthetic and were written in this repository. To
+measure the *shape* of real conditional spaces independently,
+[`dev/eval/pcs_stats.py`](https://github.com/AdventuresInDataScience/BUTChC/blob/main/dev/eval/pcs_stats.py)
+parses eleven configuration spaces published by other people, for other
+purposes, years before this library existed, and reports how much of each is
+inactive in a typical configuration. The five most conditional:
+
+| Space | Params | Median active | Inactive | Depth |
+|---|---|---|---|---|
+| AutoWEKA | 786 | 14 | **98.2%** | 4 |
+| auto-sklearn (2017) | 138 | 16 | **88.4%** | 2 |
+| SparrowToRiss | 222 | 67 | 69.8% | 4 |
+| SATenstein | 54 | 26 | 51.9% | 4 |
+| clasp 3.1.4 | 98 | 59 | 39.8% | 3 |
+
+```bash
+python dev/eval/pcs_stats.py --download
+```
+
+In AutoWEKA, 98.2% of the declared parameters are inactive in any given
+configuration — the share a flat optimiser searches and a conditional one
+skips. All 174 of its multi-parent conditions are chain-shaped, so the space is
+representable as a tree exactly rather than approximately.
+
+This measures the premise the library is built on, in spaces nobody here
+designed. It is not a head-to-head: no BUTChC-versus-TPE run has been executed
+on these spaces yet. Both halves of that are set out in
+[limitations](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/limitations.md#what-the-benchmarks-establish-and-what-they-do-not).
+
+---
+
+## Where it fits
+
+Reach for BUTChC when the search space is conditional, when it is wide, when
+the objective is noisy or on an awkward scale, or when adding a dependency is
+not an option. Those are the axes it is measured strongest on.
+
+Reach for something else when the budget is under ~50 trials on a smooth
+low-dimensional objective, where a Gaussian process will do better, or when
+your parameters interact strongly *within* a branch — sibling nodes are
+modelled independently, and `Rosenbrock` measures what that costs.
+
+The complete list of what the library does not model, does not express, and has
+not yet measured is in **[limitations](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/limitations.md)**. The one
+line worth carrying from it: the 18 benchmark problems are synthetic and were
+written alongside the optimiser, so benchmark your own space before committing
+to it.
+
+---
+
+## Versioning
+
+Defaults and internals change between minor versions, so **seeded runs do not reproduce across them**. See the [changelog](https://github.com/AdventuresInDataScience/BUTChC/blob/main/CHANGELOG.md).
+
+## Development
+
+```bash
+git clone https://github.com/AdventuresInDataScience/BUTChC
+cd BUTChC
+pip install -e ".[test]"
+pytest
+```
+
+The benchmark suite lives in `benchmarks/`, and a map of the source for
+contributors is in [docs/codemap.md](https://github.com/AdventuresInDataScience/BUTChC/blob/main/docs/codemap.md).
+
+## License
+
+MIT — see [LICENSE](https://github.com/AdventuresInDataScience/BUTChC/blob/main/LICENSE).