optimized-svm 0.1.0__tar.gz

optimized_svm-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,44 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install
+        run: |
+          uv venv
+          uv pip install -e . pytest cvxpy ruff
+      - run: uv run --no-sync ruff check src tests examples benchmarks
+      - run: uv run --no-sync pytest -q
+
+  docs:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - name: Build documentation
+        run: |
+          uv venv
+          uv pip install -e . mkdocs-material "mkdocstrings[python]"
+          uv run --no-sync mkdocs build --strict
+      - name: Deploy to GitHub Pages
+        if: github.ref == 'refs/heads/main'
+        uses: peaceiris/actions-gh-pages@v4
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./site

optimized_svm-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,32 @@
+name: Release
+
+on:
+  push:
+    tags: ["v*"]
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - run: uv build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist
+
+  publish:
+    # requires a PyPI "trusted publisher" configured for this repository
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist
+      - uses: pypa/gh-action-pypi-publish@release/v1

optimized_svm-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.coverage
+htmlcov/
+site/
+.ruff_cache/
+.mypy_cache/

optimized_svm-0.1.0/CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+## 0.1.0 – 2026-10-28
+
+Complete overhaul of the 2022 code (tag `legacy-2022`).
+
+### Changed
+- training backend: Rust solver `smo-solver` instead of the Nim package `nimsmo`
+- new package layout (`src/`), `pyproject.toml`, snake_case API, scikit-learn compatible
+  estimators `SmoothSVC`, `SmoothSVR`, `OptimizedSVC`, `OptimizedSVR`
+- adjoint formulation of the hypergradient (one linear system per fold)
+
+### Added
+- one-vs-rest multi-class classification, class and sample weights
+- warm starts across hyperparameter evaluations, rough SMO tolerance before Newton
+- `decision_gradient()` (replaces the unfinished `estimate_sensitivity`)
+- test suite (finite-difference checks of all gradients, scikit-learn estimator checks),
+  documentation, examples and a reproduction script for thesis Table 10.4.1
+
+### Fixed
+- see `LEGACY_BUGS.md` in the project workspace (P1–P13), notably: regression HPO with
+  `maxAsum` did not terminate, SMO and Newton used different kernel parameters, the 1-norm
+  constraint was always treated as active

optimized_svm-0.1.0/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2022-2026, Nico Strasdat
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

optimized_svm-0.1.0/PKG-INFO

@@ -0,0 +1,131 @@
+Metadata-Version: 2.5
+Name: optimized-svm
+Version: 0.1.0
+Summary: Support vector machines with hyperparameters optimized by exact gradients (implicit function theorem)
+Project-URL: Homepage, https://wotzlaff.github.io/svm-site/
+Project-URL: Documentation, https://wotzlaff.github.io/optimized-svm/
+Project-URL: Repository, https://github.com/wotzlaff/optimized-svm
+Author-email: Nico Strasdat <nstrasdat@gmail.com>
+License-Expression: BSD-3-Clause
+License-File: LICENSE
+Keywords: bilevel optimization,hyperparameter optimization,kernel methods,svm
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.23
+Requires-Dist: scikit-learn>=1.6
+Requires-Dist: scipy>=1.10
+Requires-Dist: smo-solver>=0.1
+Description-Content-Type: text/markdown
+
+# optimized-svm
+
+**Support vector machines whose hyperparameters are tuned by exact gradients instead of grid search.**
+
+`optimized-svm` treats hyperparameter selection as a bilevel optimization problem: the upper level
+minimizes a (smoothed) cross-validation error, the lower level is the SVM training problem. Smoothing
+the loss makes the training solution differentiable in the hyperparameters, so the gradient of the
+CV error follows from the implicit function theorem and a quasi-Newton method (L-BFGS-B) can be used.
+This works for the regularization parameter **and** for per-feature kernel scalings (automatic
+relevance determination), where grid search is hopeless.
+
+The methods are developed in
+
+> N. Strasdat, *Duality, Derivative-Based Training Methods and Hyperparameter Optimization for
+> Support Vector Machines*, Dissertation, Technische Universität Dresden, 2023.
+> [urn:nbn:de:bsz:14-qucosa2-874731](https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-874731)
+
+Training uses the Rust SMO solver [`smo-solver`](https://github.com/wotzlaff/smo-solver) as a warm start
+followed by a semismooth Newton method on the smoothed primal problem.
+
+## Features
+
+- scikit-learn estimators: `OptimizedSVC`, `OptimizedSVR` (tuned) and `SmoothSVC`, `SmoothSVR`
+  (fixed hyperparameters); binary and one-vs-rest classification
+- losses: smoothed hinge, smoothed ε-insensitive, squared (LS-SVM), Poisson; smoothing of degree
+  2, 3 or 6; sample/class weights; optional bound on `Σ|α|` (generalized ν-SVM)
+- exact hypergradients via one adjoint linear system per fold (verified against finite differences)
+- ARD kernels (`blocks="features"`) with two-stage optimization, warm starts, time limits,
+  full optimization history
+- `decision_gradient()` for local feature sensitivities
+
+## Installation
+
+```bash
+pip install optimized-svm
+```
+
+During development (sibling checkout of `smo-solver`, Rust toolchain required):
+
+```bash
+git clone https://github.com/wotzlaff/smo-solver
+git clone https://github.com/wotzlaff/optimized-svm && cd optimized-svm
+uv sync --group dev
+uv run pytest
+```
+
+## Quick start
+
+```python
+from sklearn.datasets import make_friedman1
+from optimized_svm import OptimizedSVR
+
+x, y = make_friedman1(600, noise=0.5, random_state=0)
+model = OptimizedSVR(blocks="features").fit(x, y)   # one kernel scaling per feature
+
+model.lam_            # optimized regularization parameter
+model.scaling_opt_    # optimized scalings: features 5..9 (pure noise) are switched off
+model.history_        # every evaluation of the CV error
+model.predict(x[:5])
+```
+
+On a 1000-sample classification task (`examples/quickstart.py`) the optimizer needs 16 CV
+evaluations (≈10 s) versus 143 grid points (≈20 s) for `GridSearchCV` with `sklearn.svm.SVC`,
+with comparable test accuracy (0.844 vs. 0.856).
+
+## How it works
+
+| step | method | thesis |
+|---|---|---|
+| training, warm start | SMO on the dual with quadratic smoothing (`smo-solver`) | Ch. 7 |
+| training, refinement | semismooth Newton on `a + φ'(Ka/λ + b + sc) = 0` | Ch. 8.4 |
+| CV error gradient | implicit function theorem, adjoint system per fold | Ch. 9.2 |
+| hyperparameter search | L-BFGS-B over `(log λ, log s₁, …, log s_K)` | Ch. 9.1, 10.4 |
+
+## Reproducing the thesis
+
+`benchmarks/thesis_regression.py` repeats the protocol of Table 10.4.1 (LIBSVM regression data,
+30 random 80/20 splits, ε-SVR with cubic smoothing and LS-SVM):
+
+```bash
+python benchmarks/thesis_regression.py --datasets pyrim triazines mpg housing --splits 30
+```
+
+## Documentation
+
+Full documentation including the mathematical background: <https://wotzlaff.github.io/optimized-svm/>.
+Build locally with `uv run --group docs mkdocs serve`.
+
+## Citation
+
+If you use this software, please cite the dissertation:
+
+```bibtex
+@phdthesis{strasdat2023,
+  author = {Nico Strasdat},
+  title  = {Duality, Derivative-Based Training Methods and
+            Hyperparameter Optimization for Support Vector Machines},
+  school = {Technische Universit{\"a}t Dresden},
+  type   = {Dissertation},
+  year   = {2023},
+  url    = {https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-874731},
+  note   = {urn:nbn:de:bsz:14-qucosa2-874731}
+}
+```
+
+## License
+
+BSD-3-Clause. The 2022 version (Nim backend) is preserved in the git tag `legacy-2022`.

optimized_svm-0.1.0/README.md

@@ -0,0 +1,108 @@
+# optimized-svm
+
+**Support vector machines whose hyperparameters are tuned by exact gradients instead of grid search.**
+
+`optimized-svm` treats hyperparameter selection as a bilevel optimization problem: the upper level
+minimizes a (smoothed) cross-validation error, the lower level is the SVM training problem. Smoothing
+the loss makes the training solution differentiable in the hyperparameters, so the gradient of the
+CV error follows from the implicit function theorem and a quasi-Newton method (L-BFGS-B) can be used.
+This works for the regularization parameter **and** for per-feature kernel scalings (automatic
+relevance determination), where grid search is hopeless.
+
+The methods are developed in
+
+> N. Strasdat, *Duality, Derivative-Based Training Methods and Hyperparameter Optimization for
+> Support Vector Machines*, Dissertation, Technische Universität Dresden, 2023.
+> [urn:nbn:de:bsz:14-qucosa2-874731](https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-874731)
+
+Training uses the Rust SMO solver [`smo-solver`](https://github.com/wotzlaff/smo-solver) as a warm start
+followed by a semismooth Newton method on the smoothed primal problem.
+
+## Features
+
+- scikit-learn estimators: `OptimizedSVC`, `OptimizedSVR` (tuned) and `SmoothSVC`, `SmoothSVR`
+  (fixed hyperparameters); binary and one-vs-rest classification
+- losses: smoothed hinge, smoothed ε-insensitive, squared (LS-SVM), Poisson; smoothing of degree
+  2, 3 or 6; sample/class weights; optional bound on `Σ|α|` (generalized ν-SVM)
+- exact hypergradients via one adjoint linear system per fold (verified against finite differences)
+- ARD kernels (`blocks="features"`) with two-stage optimization, warm starts, time limits,
+  full optimization history
+- `decision_gradient()` for local feature sensitivities
+
+## Installation
+
+```bash
+pip install optimized-svm
+```
+
+During development (sibling checkout of `smo-solver`, Rust toolchain required):
+
+```bash
+git clone https://github.com/wotzlaff/smo-solver
+git clone https://github.com/wotzlaff/optimized-svm && cd optimized-svm
+uv sync --group dev
+uv run pytest
+```
+
+## Quick start
+
+```python
+from sklearn.datasets import make_friedman1
+from optimized_svm import OptimizedSVR
+
+x, y = make_friedman1(600, noise=0.5, random_state=0)
+model = OptimizedSVR(blocks="features").fit(x, y)   # one kernel scaling per feature
+
+model.lam_            # optimized regularization parameter
+model.scaling_opt_    # optimized scalings: features 5..9 (pure noise) are switched off
+model.history_        # every evaluation of the CV error
+model.predict(x[:5])
+```
+
+On a 1000-sample classification task (`examples/quickstart.py`) the optimizer needs 16 CV
+evaluations (≈10 s) versus 143 grid points (≈20 s) for `GridSearchCV` with `sklearn.svm.SVC`,
+with comparable test accuracy (0.844 vs. 0.856).
+
+## How it works
+
+| step | method | thesis |
+|---|---|---|
+| training, warm start | SMO on the dual with quadratic smoothing (`smo-solver`) | Ch. 7 |
+| training, refinement | semismooth Newton on `a + φ'(Ka/λ + b + sc) = 0` | Ch. 8.4 |
+| CV error gradient | implicit function theorem, adjoint system per fold | Ch. 9.2 |
+| hyperparameter search | L-BFGS-B over `(log λ, log s₁, …, log s_K)` | Ch. 9.1, 10.4 |
+
+## Reproducing the thesis
+
+`benchmarks/thesis_regression.py` repeats the protocol of Table 10.4.1 (LIBSVM regression data,
+30 random 80/20 splits, ε-SVR with cubic smoothing and LS-SVM):
+
+```bash
+python benchmarks/thesis_regression.py --datasets pyrim triazines mpg housing --splits 30
+```
+
+## Documentation
+
+Full documentation including the mathematical background: <https://wotzlaff.github.io/optimized-svm/>.
+Build locally with `uv run --group docs mkdocs serve`.
+
+## Citation
+
+If you use this software, please cite the dissertation:
+
+```bibtex
+@phdthesis{strasdat2023,
+  author = {Nico Strasdat},
+  title  = {Duality, Derivative-Based Training Methods and
+            Hyperparameter Optimization for Support Vector Machines},
+  school = {Technische Universit{\"a}t Dresden},
+  type   = {Dissertation},
+  year   = {2023},
+  url    = {https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-874731},
+  note   = {urn:nbn:de:bsz:14-qucosa2-874731}
+}
+```
+
+## License
+
+BSD-3-Clause. The 2022 version (Nim backend) is preserved in the git tag `legacy-2022`.

optimized_svm-0.1.0/benchmarks/thesis_regression.py

@@ -0,0 +1,121 @@
+"""Reproduces the protocol of Table 10.4.1 (Strasdat, Sec. 10.4.2) on LIBSVM regression data.
+
+For every dataset and split (80 % for hyperparameter optimisation + training, 20 % for testing):
+features and targets are standardised on the training part, lambda and the RBF scaling are
+optimised by minimising the 5-fold CV error (smoothed absolute error, delta = 0.1) and the
+test MSE/MAE are reported in original units. Models: smoothed epsilon-SVR (cubic smoothing,
+delta = epsilon / 4) for epsilon in {0.1, 0.25, 0.5} and the LS-SVM.
+
+Usage::
+
+    python benchmarks/thesis_regression.py --datasets pyrim triazines --splits 30
+"""
+
+from __future__ import annotations
+
+import argparse
+import time
+import urllib.request
+from pathlib import Path
+
+import numpy as np
+from sklearn.datasets import load_svmlight_file
+from sklearn.model_selection import train_test_split
+from sklearn.preprocessing import StandardScaler
+
+from optimized_svm import OptimizedSVR
+
+URL = "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression/{}"
+FILES = {
+    "pyrim": "pyrim",
+    "triazines": "triazines",
+    "mpg": "mpg",
+    "housing": "housing",
+    "spacega": "space_ga",
+    "abalone": "abalone",
+    "cpusmall": "cpusmall",
+}
+#: values of Table 10.4.1 (validation loss 10.4.1): MSE / MAE for eps 0.1, 0.25, 0.5, LS-SVM
+THESIS = {
+    "pyrim": ([0.00656, 0.00625, 0.00645, 0.00597], [0.0532, 0.0512, 0.0526, 0.0491]),
+    "triazines": ([0.0244, 0.0239, 0.0237, 0.0232], [0.107, 0.109, 0.111, 0.110]),
+    "mpg": ([6.50, 6.58, 7.37, 6.63], [1.81, 1.84, 1.98, 1.82]),
+    "housing": ([10.8, 10.9, 11.9, 9.54], [2.10, 2.17, 2.43, 2.07]),
+    "spacega": ([0.0133, 0.0127, 0.0128, 0.0127], [0.0849, 0.0845, 0.0854, 0.0845]),
+    "abalone": ([4.72, 4.61, 4.56, 4.53], [1.50, 1.50, 1.52, 1.53]),
+    "cpusmall": ([19.1, 20.0, 32.6, 18.7], [2.70, 2.96, 3.66, 2.70]),
+}
+MODELS = [("eps=0.1", 0.1), ("eps=0.25", 0.25), ("eps=0.5", 0.5), ("LS-SVM", None)]
+
+
+def load(name: str, cache: Path) -> tuple[np.ndarray, np.ndarray]:
+    cache.mkdir(parents=True, exist_ok=True)
+    path = cache / FILES[name]
+    if not path.exists():
+        urllib.request.urlretrieve(URL.format(FILES[name]), path)
+    x, y = load_svmlight_file(str(path))
+    return np.asarray(x.todense()), np.asarray(y)
+
+
+def model(eps: float | None, n_features: int, time_limit: float) -> OptimizedSVR:
+    common = dict(
+        lam_init=1e-2,
+        scaling_init=float(np.sqrt(np.log(2.0) / n_features)),
+        lam_bounds=(1e-6, 1.0),
+        scaling_bounds=(1e-6, 1e2),
+        validation="absolute",
+        validation_smoothing=0.1,
+        normalize_y=False,  # targets are standardised by the protocol
+        time_limit=time_limit,
+    )
+    if eps is None:
+        return OptimizedSVR(loss="squared", **common)
+    return OptimizedSVR(epsilon=eps, smoothing=eps / 4, degree="3", **common)
+
+
+def run(name: str, splits: int, time_limit: float, cache: Path, max_size: int) -> None:
+    x, y = load(name, cache)
+    rng = np.random.default_rng(0)
+    mse = np.zeros((splits, len(MODELS)))
+    mae = np.zeros_like(mse)
+    t0 = time.perf_counter()
+    for s in range(splits):
+        if len(y) > max_size:
+            idx = rng.choice(len(y), max_size, replace=False)
+            xs, ys = x[idx], y[idx]
+        else:
+            xs, ys = x, y
+        xa, xb, ya, yb = train_test_split(xs, ys, test_size=0.2, random_state=s)
+        sx, sy = StandardScaler().fit(xa), StandardScaler().fit(ya[:, None])
+        xa_n, xb_n = sx.transform(xa), sx.transform(xb)
+        ya_n = sy.transform(ya[:, None]).ravel()
+        for m, (_, eps) in enumerate(MODELS):
+            est = model(eps, x.shape[1], time_limit).fit(xa_n, ya_n)
+            pred = sy.inverse_transform(est.predict(xb_n)[:, None]).ravel()
+            mse[s, m] = np.mean((pred - yb) ** 2)
+            mae[s, m] = np.mean(np.abs(pred - yb))
+    elapsed = time.perf_counter() - t0
+    ref_mse, ref_mae = THESIS[name]
+    print(f"\n{name} (n = {min(len(y), max_size)}, {splits} splits, {elapsed:.0f} s)")
+    print(f"{'model':<10}{'MSE':>11}{'thesis':>11}{'MAE':>11}{'thesis':>11}")
+    for m, (label, _) in enumerate(MODELS):
+        print(
+            f"{label:<10}{mse[:, m].mean():>11.4g}{ref_mse[m]:>11.4g}"
+            f"{mae[:, m].mean():>11.4g}{ref_mae[m]:>11.4g}"
+        )
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument("--datasets", nargs="+", default=list(FILES), choices=list(FILES))
+    parser.add_argument("--splits", type=int, default=30)
+    parser.add_argument("--time-limit", type=float, default=600.0)
+    parser.add_argument("--max-size", type=int, default=1000)
+    parser.add_argument("--cache", type=Path, default=Path.home() / ".cache" / "optimized-svm")
+    args = parser.parse_args()
+    for name in args.datasets:
+        run(name, args.splits, args.time_limit, args.cache, args.max_size)
+
+
+if __name__ == "__main__":
+    main()

optimized_svm-0.1.0/docs/api.md

@@ -0,0 +1,41 @@
+# API reference
+
+## Estimators
+
+::: optimized_svm.OptimizedSVC
+
+::: optimized_svm.OptimizedSVR
+
+::: optimized_svm.SmoothSVC
+
+::: optimized_svm.SmoothSVR
+
+## Training
+
+::: optimized_svm.TrainingProblem
+
+::: optimized_svm.train
+
+::: optimized_svm.newton
+
+::: optimized_svm.Solution
+
+## Losses
+
+::: optimized_svm.losses
+
+## Hypergradient
+
+::: optimized_svm.cv_error_and_gradient
+
+::: optimized_svm.fold_error_and_gradient
+
+## Validation losses
+
+::: optimized_svm.validation
+
+## Kernels and smoothing
+
+::: optimized_svm.kernels
+
+::: optimized_svm.smooth_max

optimized_svm-0.1.0/docs/changelog.md

@@ -0,0 +1,23 @@
+# Changelog
+
+## 0.1.0 – 2026-10-28
+
+Complete overhaul of the 2022 code (tag `legacy-2022`).
+
+### Changed
+- training backend: Rust solver `smo-solver` instead of the Nim package `nimsmo`
+- new package layout (`src/`), `pyproject.toml`, snake_case API, scikit-learn compatible
+  estimators `SmoothSVC`, `SmoothSVR`, `OptimizedSVC`, `OptimizedSVR`
+- adjoint formulation of the hypergradient (one linear system per fold)
+
+### Added
+- one-vs-rest multi-class classification, class and sample weights
+- warm starts across hyperparameter evaluations, rough SMO tolerance before Newton
+- `decision_gradient()` (replaces the unfinished `estimate_sensitivity`)
+- test suite (finite-difference checks of all gradients, scikit-learn estimator checks),
+  documentation, examples and a reproduction script for thesis Table 10.4.1
+
+### Fixed
+- see `LEGACY_BUGS.md` in the project workspace (P1–P13), notably: regression HPO with
+  `maxAsum` did not terminate, SMO and Newton used different kernel parameters, the 1-norm
+  constraint was always treated as active

optimized_svm-0.1.0/docs/getting-started.md

@@ -0,0 +1,71 @@
+# Getting started
+
+## Classification
+
+```python
+from sklearn.datasets import load_breast_cancer
+from sklearn.model_selection import train_test_split
+from sklearn.preprocessing import StandardScaler
+from optimized_svm import OptimizedSVC
+
+x, y = load_breast_cancer(return_X_y=True)
+x = StandardScaler().fit_transform(x)
+x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=0)
+
+clf = OptimizedSVC(cv=5, verbose=1).fit(x_train, y_train)
+print(clf.lam_, clf.scaling_opt_, clf.score(x_test, y_test))
+```
+
+`OptimizedSVC` minimizes the 5-fold cross-validation error of a smoothed hinge loss over
+$\log\lambda$ and the log kernel scaling $\log s$ (the RBF parameter is $\gamma = s^2$). After the
+search the model is refitted on all data with the best parameters.
+
+!!! tip "Feature scaling"
+    The kernel acts on the raw features. Standardize them (e.g. with `StandardScaler`) or use
+    `blocks="features"` so that every feature gets its own scaling.
+
+## Regression and feature relevance
+
+```python
+from sklearn.datasets import make_friedman1
+from optimized_svm import OptimizedSVR
+
+x, y = make_friedman1(600, noise=0.5, random_state=0)
+reg = OptimizedSVR(epsilon=0.1, blocks="features").fit(x, y)
+print(reg.scaling_opt_.round(3))   # small values = irrelevant features
+```
+
+With `blocks="features"` the kernel is $k(x, z) = \exp(-\sum_j s_j^2 (x_j - z_j)^2)$. With
+`two_stage=True` (default) a single common scaling is optimized first and used as starting point.
+
+## Choosing the model
+
+| estimator | loss | parameters |
+|---|---|---|
+| `SmoothSVC` / `OptimizedSVC` | smoothed hinge $w_i\, m_\delta(\text{shift} - y_i f(x_i))$ | `smoothing`, `degree`, `shift`, `class_weight`, `max_asum` |
+| `SmoothSVR` / `OptimizedSVR` (`loss="epsilon_insensitive"`) | $w_i (m_\delta(f_i - y_i - \varepsilon) + m_\delta(y_i - f_i - \varepsilon))$ | `epsilon`, `smoothing`, `degree` |
+| `loss="squared"` | $\tfrac{w_i}{2} (f_i - y_i)^2$ (LS-SVM) | – |
+| `loss="poisson"` | $w_i (e^{f_i} - y_i f_i)$, prediction $e^{f}$ | – |
+
+`smoothing` ($\delta$) controls how strongly the kinks of the loss are rounded; `degree` selects
+the smoothing polynomial ("2": $C^1$, exactly solvable by SMO; "3" and "6": $C^2$).
+
+## Useful attributes
+
+- `lam_`, `scaling_opt_`: optimized hyperparameters
+- `history_`: list of all CV evaluations (`lam`, `scaling`, `error`, `gradient_norm`, `time`, …)
+- `cv_error_`, `time_`, `n_iter_`
+- `support_`, `dual_coef_`, `intercept_`, `training_info_` of the final model
+- `decision_gradient(X)`: derivative of the decision function w.r.t. the inputs
+
+## Lower-level building blocks
+
+```python
+from optimized_svm import EpsilonInsensitiveLoss, TrainingProblem, train
+
+prob = TrainingProblem(x * 0.5, y, EpsilonInsensitiveLoss(0.1, 0.05, "3"), lam=1e-3)
+sol = train(prob)          # SMO warm start + Newton refinement
+sol.a, sol.b, sol.residual
+```
+
+See `examples/` in the repository for complete scripts.