scm2cpp-lasso 0.8.0__tar.gz → 0.9.0__tar.gz

{scm2cpp_lasso-0.8.0 → scm2cpp_lasso-0.9.0}/PKG-INFO

@@ -1,12 +1,12 @@
 Metadata-Version: 2.4
 Name: scm2cpp-lasso
-Version: 0.8.0
-Summary: Lasso by covariance-update coordinate descent, translated from Scheme to C++ by scm2cpp; GPU batch solver included
+Version: 0.9.0
+Summary: Lasso, elastic net, ridge, SVM, SVR, L1 logistic, NMF, graphical lasso, RANSAC, SGD, OMP and LARS translated from Scheme to C++ by scm2cpp; GPU batch solver included
 Author: Hirotaka Niitsuma
 License: MIT
 Project-URL: Homepage, https://github.com/niitsuma/scm2cpp
 Project-URL: Source, https://github.com/niitsuma/scm2cpp/tree/master/python/scm2cpp-lasso
-Keywords: lasso,elastic-net,ridge,coordinate-descent,cuda,compiler
+Keywords: lasso,elastic-net,ridge,svm,logistic-regression,coordinate-descent,cuda,compiler
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: C++
@@ -156,12 +156,58 @@ three larger shapes -- the figures of the repository's equal-work
 table, now reached from the package with nothing in front of the
 kernel: the constructor keeps X and y and the kernel takes the
 residual and the column norms itself (the norms folded into the Gram
-diagonal by the derivation).  Against a reference driven to
+diagonal by the derivation).  Threads change the picture less than one might expect: scikit-learn's
+sweeps are single-threaded Cython and only its Gram product scales,
+so with 20 BLAS threads `lasso_path` takes 0.051 / 0.227 / 0.450 s on
+the three larger shapes, against 0.023 / 0.050 / 0.113 s for the
+cuBLAS build (2.2x, 4.5x, 4.0x) and 0.025 / 0.036 / 0.123 s for the
+CBLAS build (2.0x, 6.3x, 3.7x), whose `dsyrk` scales with the
+threads; the GPU figures do not move.  Against a reference driven to
 tol=1e-12, `DerivedLasso` lands within 5e-8 on every backend (its
 tolerance is absolute, on one sweep's largest move), `CovLasso`
 within 2e-10 (it tests every 20 sweeps and overshoots), and
 scikit-learn at its default within 1e-3.
 
+### Deriving it yourself
+
+Everything `DerivedLasso` runs is reproducible from the Scheme with
+the translator; this is what `regenerate.sh` does for the package,
+spelled out for one kernel (Racket 8 and a C++17 compiler; a CBLAS
+for `--blas`, the CUDA toolkit for `--cublas`):
+
+```console
+$ git clone https://github.com/niitsuma/scm2cpp && cd scm2cpp
+$ raco link --user vendor/rkanren                 # once per machine
+$ mkdir /tmp/k && cp examples/kernel-only/lasso-design.scm examples/kernel-only/soft-threshold.scm /tmp/k/
+$ racket scm2cpp-file.scm -t scm2c.typ --derive --cublas -M -S /tmp/k/lasso-design.scm
+derive: lasso-design: raise differencing merge matmul guard
+```
+
+`derive:` on stderr names what fired: the loops raised to the array
+algebra, the residual differenced into the memo, the column norms
+merged into the Gram diagonal, the products folded to contractions,
+the `(if (< p n) ..)` guard.  The run leaves, beside the source,
+`lasso-design.hpp` (the derived kernel as C++: one `blas_syrk`, two
+`blas_gemv`, the sweep), `lasso-design_capi.cpp` and
+`lasso_design.py` (the `-M` C ABI and its ctypes loader -- what the
+package's `_loader_derived_*.py` are), and `lasso-design.expanded.scm`
+(`-S`: the derived program as Scheme, before C++ -- read this to see
+what the derivation did).  `--blas` instead of `--cublas` gives the
+CBLAS variant, neither the loop variant.  To call it from numpy
+without the package:
+
+```console
+$ g++ -O2 -std=c++17 -shared -fPIC -I. -I$CUDA_HOME/include -o /tmp/k/liblasso-design.so \
+      /tmp/k/lasso-design_capi.cpp -L$CUDA_HOME/lib64 -lcublas -lcudart   # -lopenblas for --blas
+$ cd /tmp/k && python3 -c 'import lasso_design'   # lasso_design.lasso_design(x, y, resid, xnorm, beta, lams, betas, iters, tol, n, p, nlam)
+```
+
+(`-M` names the Python module after the file with `-` read as `_`,
+and the loader looks for `liblasso-design.so` beside it.)
+`examples/kernel-only/README.md` walks through the same `-M` flow on
+the row-major kernel, and `python/PACKAGING.md` through turning such
+a kernel into a pip package.
+
 ## Against the other lassos pip can install
 
 `bench/lasso-compare.py` in the repository runs the same
@@ -235,6 +281,23 @@ the pick is exact at n=5,000 and one grid step apart on a near-tie
 elsewhere (mean-MSE relative gap at most 2.6e-4), and sklearn at
 tol=1e-10 picks our alpha at every size.
 
+The elastic net cross-validates the same way: `CovLassoCV(l1_ratio=0.5)`
+against `ElasticNetCV(l1_ratio=0.5)` on the same folds and alpha grid
+(`bench/lasso-compare.py --cv-only --l1-ratio 0.5`, same conditions,
+load under 1):
+
+| n       | p    | CPU    | CUDA   | sklearn `ElasticNetCV` |
+|---------|------|--------|--------|------------------------|
+| 1,800   | 200  | 0.35 s | 0.02 s | 0.11 s                 |
+| 5,000   | 1000 | 1.55 s | 0.26 s | 0.98 s                 |
+| 100,000 | 200  | 0.52 s | 0.35 s | 1.76 s                 |
+| 100,000 | 500  | 1.38 s | 1.15 s | 4.97 s                 |
+
+The picture is the lasso's: on the CPU, 3.4-3.6x at 100,000 rows and
+slower than sklearn at the two small shapes; with CUDA, 3.8-5.5x
+everywhere.  The chosen alpha is sklearn's at all four sizes, the
+coefficients within 8e-6 (sklearn's default tolerance).
+
 `CovMultiTaskLasso` and `CovMultiTaskLassoCV` are the multi-task
 family over the same machinery: scikit-learn's `MultiTaskLasso`,
 `MultiTaskElasticNet` (through `l1_ratio`), `MultiTaskLassoCV` and
@@ -319,6 +382,247 @@ ridge = CovRidge(X, y)
 betas = ridge.fit_path(ridge.alpha_grid())     # a whole ridge path
 ```
 
+Measured (`bench/ridge-enet-compare.py` in the repository: AR(1)
+designs, 20 penalties, whole calls with the Gram matrix wherever one
+is built, one core, best of three):
+
+| n x p                                        | 1,800 x 200 | 5,000 x 1000 | 100,000 x 200 | 100,000 x 500 |
+|----------------------------------------------|-------------|--------------|---------------|---------------|
+| elastic net, `CovLasso.fit_path(l1_ratio=0.5)`, tol 1e-8 | 0.007 s | 0.092 s | 0.11 s      | 0.40 s        |
+| elastic net, sklearn `enet_path(precompute=True)`, tol 1e-4 | 0.005 s | 0.099 s | 0.23 s   | 0.74 s        |
+| ridge, `CovRidge.fit_path(alphas)`           | 0.004 s     | 0.21 s       | 0.10 s        | 0.41 s        |
+| ridge, sklearn `Ridge(alpha).fit` per alpha  | 0.060 s     | 2.05 s       | 3.37 s        | 11.2 s        |
+| ridge, sklearn `RidgeCV(alphas)` (its LOO path) | 0.093 s  | 3.40 s       | 5.38 s        | 19.0 s        |
+
+The elastic net is the same coordinate descent on both sides, so the
+two rows differ by the Gram and the warm path, not the sweeps: level
+at small n, 1.8-2.2x at 100,000 rows, and ours lands within 2e-10 of
+a tight reference where scikit-learn's default tolerance lands within
+6e-4.  The ridge rows compare a whole path against what a user
+writes -- one `Ridge.fit` per alpha, each forming X'X and solving
+afresh, O(np^2 + p^3) every time -- so the gap is the path's: 10-33x,
+agreeing to 1e-14.  `RidgeCV` is given for scale (it picks one alpha
+by leave-one-out rather than returning the path).  Neither row runs
+on the GPU: the ridge is one eigendecomposition and the elastic net's
+sweeps are the lasso's; `fit_path_batch(l1_ratio=..)` is the GPU
+grid for the elastic net.
+
+## Bayesian ridge
+
+`CovBayesianRidge` is scikit-learn's `BayesianRidge` -- the same model,
+the same defaults, the same iteration (MacKay's updates of the noise
+precision `alpha_` and the weight precision `lambda_`, stopping when
+the posterior mean moves by less than `tol`) -- with the linear
+algebra moved into the eigenbasis of the Gram matrix.  scikit-learn
+takes the SVD of X once and multiplies by X again in every iteration
+for the squared error.  Here X enters once, through X'X (X X' when
+n <= p): one eigendecomposition, after which the posterior mean, its
+norm and the squared error are sums over the eigenvalues, O(p) per
+iteration.  `sigma_` (the posterior covariance) and
+`predict(return_std=True)` are there as in scikit-learn.
+
+```python
+from scm2cpp_lasso import CovBayesianRidge
+m = CovBayesianRidge().fit(X, y)          # fit_intercept=True, as sklearn
+mean, std = m.predict(X_new, return_std=True)
+```
+
+Measured (`bench/bayesian-ridge-compare.py` in the repository:
+Gaussian designs, a sparse truth plus noise, whole fits including
+`sigma_`, one core, best of three, sklearn 1.9.1):
+
+| n x p          | `CovBayesianRidge` | sklearn `BayesianRidge` | ratio | iterations (both) | max coefficient difference |
+|----------------|--------------------|-------------------------|-------|-------------------|----------------------------|
+| 1,800 x 200    | 0.005 s            | 0.026 s                 | 5.2x  | 3                 | 3.6e-15                    |
+| 5,000 x 1000   | 0.257 s            | 1.231 s                 | 4.8x  | 4                 | 6.7e-15                    |
+| 100,000 x 200  | 0.126 s            | 3.495 s                 | 27.8x | 3                 | 2.8e-14                    |
+| 100,000 x 500  | 0.475 s            | 8.909 s                 | 18.8x | 3                 | 1.6e-14                    |
+| 500 x 5000     | 0.662 s            | 8.689 s                 | 13.1x | 71                | 1.7e-15                    |
+
+The iteration counts are identical and the coefficients agree to
+rounding, so the gap is the linear algebra alone: X'X by one BLAS
+product against a full SVD of the tall X, and no pass over X inside
+the loop.  At 500 x 5000 scikit-learn takes the full 5000 x 5000 SVD
+for `sigma_`; here `sigma_` comes from the 500 x 500 kernel side by
+the Woodbury identity.  This estimator is numpy, not translated code:
+the eigendecomposition is LAPACK's either way.
+
+## Classifiers and SVRs: `LinearSVM`, `KernelSVM`, `LinearSVR`, `KernelSVR`, `L1LogisticRegression`
+
+Three classifiers and two regressors, each the algorithm scikit-learn runs, written out
+in the Scheme subset and translated plainly (no derivation; the C++
+is committed under `_generated/classifiers*`):
+
+| class | algorithm | scikit-learn counterpart |
+|---|---|---|
+| `LinearSVM` | liblinear's dual coordinate descent for the hinge loss (random order, shrinking) | `LinearSVC(loss="hinge", dual=True, fit_intercept=False)` |
+| `KernelSVM` | libsvm's SMO (WSS3, shrinking, the bias) over the whole kernel matrix | `SVC` (`kernel="rbf"`, `"linear"` or `"precomputed"`) |
+| `L1LogisticRegression` | liblinear's newGLMNET (Newton over coordinate descent, line search) | `LogisticRegression(l1_ratio=1, solver="liblinear", fit_intercept=False)` |
+| `LinearSVR` | liblinear's dual coordinate descent for the epsilon-insensitive loss (random order, shrinking) | `LinearSVR(loss="epsilon_insensitive", dual=True, fit_intercept=False)` |
+| `KernelSVR` | the same SMO as `KernelSVM`, set up as libsvm's epsilon-SVR (2n variables) | `SVR` (`kernel="rbf"`, `"linear"` or `"precomputed"`) |
+
+```python
+from scm2cpp_lasso import LinearSVM, KernelSVM, L1LogisticRegression
+LinearSVM(C=1.0).fit(X, y).predict(X_new)
+KernelSVM(C=1.0, gamma="scale").fit(X, y).decision_function(X_new)
+L1LogisticRegression(C=0.1).fit(X, y).predict_proba(X_new)
+```
+
+The classifiers take two classes (the second of `classes_` is the
+positive one), and `tol` / `max_iter` mean what they mean in scikit-learn.
+`KernelSVM` and `KernelSVR` build the kernel matrix in numpy and keep all of it,
+n x n doubles (800 MB at n = 10,000), where `SVC` computes rows on
+demand through a cache; that is its price.
+
+Measured with `check-classifiers.py` (in this directory): the same
+problem and stopping rule on both sides, one core, best of three,
+sklearn 1.9.1, load average under 2. Ours includes building the kernel
+matrix (`KernelSVM`) or transposing X into columns
+(`L1LogisticRegression`):
+
+| problem | ours | scikit-learn | ratio | objective (ours / sklearn) | same predictions |
+|---|---|---|---|---|---|
+| `LinearSVM`, 5,000 x 200 | 0.379 s | 0.598 s | 1.6x | 1541.06 / 1541.29 | 99.5% |
+| `LinearSVM`, 100,000 x 200 | 23.0 s | 41.7 s | 1.8x | 34503.5 / 34500.7 | 98.9% |
+| `LinearSVM`, 2,000 x 20,000 | 0.975 s | 1.894 s | 1.9x | 0.052484 / 0.052428 | 100% |
+| `KernelSVM` rbf, 2,000 x 50 | 0.068 s | 0.139 s (precomputed 0.081 s) | 2.1x (1.2x) | -1065.5026 / -1065.5026 | 100% |
+| `KernelSVM` rbf, 5,000 x 50 | 0.501 s | 1.032 s (precomputed 0.588 s) | 2.1x (1.2x) | -2628.1422 / -2628.1422 | 100% |
+| `KernelSVM` rbf, 10,000 x 50 | 2.12 s | 4.15 s (precomputed 2.61 s) | 2.0x (1.2x) | -4994.9982 / -4994.9982 | 100% |
+| `L1LogisticRegression`, 5,000 x 200, C=0.1 | 0.064 s | 0.103 s | 1.6x | 61.676698 / 61.676708 | 100% |
+| `L1LogisticRegression`, 20,000 x 500, C=0.05 | 1.20 s | 2.23 s | 1.9x | 106.77913 / 106.77915 | 100% |
+| `L1LogisticRegression`, 100,000 x 200, C=0.05 | 9.06 s | 18.7 s | 2.1x | 349.61154 / 349.61216 | 100% |
+
+The regressors, from the same script (C = 1, epsilon 0.1):
+
+| problem | ours | scikit-learn | ratio | agreement |
+|---|---|---|---|---|
+| `LinearSVR`, 5,000 x 200 | 1.32 s | 2.53 s | 1.9x | both at the 1,000-sweep cap; objective 779.99 / 790.72 |
+| `LinearSVR`, 20,000 x 500 | 22.1 s | 36.6 s | 1.7x | both at the cap; objective 3741.4 / 3712.2 |
+| `KernelSVR` rbf, 1,000 x 10 | 0.023 s | 0.043 s | 1.9x | iterations 2,172 / 2,177; predictions within 4e-4 |
+| `KernelSVR` rbf, 3,000 x 10 | 0.246 s | 0.332 s | 1.3x | iterations 7,471 / 7,581; within 5e-4 |
+| `KernelSVR` rbf, 6,000 x 10 | 1.18 s | 1.38 s | 1.2x | iterations 16,805 / 16,630; within 5e-4 |
+
+`LinearSVR` on these data does not reach its tolerance in 1,000 sweeps on
+either side, and at the cap the random order alone moves the objective
+by about 1% (scikit-learn's own seeds give 776-789 at 5,000 x 200);
+with 20,000 sweeps both land on 728.58.  `KernelSVR`'s gain shrinks
+with n here because the 10 features make each kernel row cheap for
+libsvm to compute, while ours reads the rows of a matrix of n^2
+doubles; the predictions agree to within scikit-learn's tolerance.
+
+The objectives agree to the stopping tolerance on both sides, not to
+rounding. The linear SVM on tall data stops at the 1,000-sweep cap on
+both sides, so there the two land a hair apart and 1% of the
+predictions differ. The gain is the same in all three: the same
+algorithm, run as dense loops that vectorise, against libsvm's and
+liblinear's sparse rows. The classifiers are compiled at `-O3` with
+`-fstrict-overflow`: Python's build flags carry
+`-fno-strict-overflow`, which keeps the compiler from treating the
+flat index `i*p + j` as an induction variable and makes the linear
+SVM 25% slower.
+
+## NMF: `CDNMF`
+
+`CDNMF` is scikit-learn's `NMF(solver="cd")` -- Frobenius loss, no
+regularization, no shuffling -- as the translator derives it.
+`examples/kernel-only/nmf-cd.scm` writes the half step with the
+residual R = X - W H and no Gram matrix. `--derive` memoises R H' over
+the hoisted H H' and distributes the residual through the memo, so the
+compiled half step is scikit-learn's `XHt - W HHt` with R never formed.
+Like `DerivedLasso`, it is built in three variants (`nmf_backend()`
+says which is taken: cublas when a device answers, else blas, else
+loops).
+
+```python
+from scm2cpp_lasso import CDNMF
+model = CDNMF(n_components=20, random_state=0)   # init="random", as sklearn's
+W = model.fit_transform(X)                       # H in model.components_
+```
+
+`init="random"` draws the start as scikit-learn does, so the same seed
+gives the same W and H. scikit-learn's default `"nndsvda"` is not
+offered; pass `init="custom"` with its W and H for that start.
+Measured with `check-classifiers.py --only nmf` against
+`NMF(solver="cd", init="random")` from the same seed, tol 1e-4, at most
+200 iterations, one core:
+
+| n x p, k | `CDNMF` cuBLAS | `CDNMF` BLAS | sklearn | iterations | max \|H - H_sklearn\| |
+|---|---|---|---|---|---|
+| 1,000 x 500, 10 | 0.441 s (0.6x) | 0.212 s (1.15x) | 0.244 s | 200 / 200 | 1.3e-13 |
+| 5,000 x 2,000, 20 | 7.01 s (1.6x) | 8.27 s (1.36x) | 11.25 s | 200 / 200 | 4.1e-12 |
+| 20,000 x 1,000, 30 | 16.9 s (1.8x) | 20.6 s (1.46x) | 30.1 s | 200 / 200 | 2.4e-12 |
+
+The iterates are scikit-learn's to rounding. The cuBLAS variant pays
+from the middle size on; at 1,000 x 500 its transfers cost more than
+the products save, and `backend="blas"` is the faster choice there.
+
+## Graphical lasso and RANSAC: `CDGraphicalLasso`, `LinearRANSAC`
+
+Two more of scikit-learn's algorithms, written out in the Scheme subset
+and translated plainly.
+
+- `CDGraphicalLasso` is `GraphicalLasso(mode="cd")`: `_graphical_lasso`
+  with `enet_coordinate_descent_gram` inside it, its two matrix-vector
+  products as dgemv. `alpha`, `tol`, `enet_tol`, `max_iter` and
+  `assume_centered` mean what they mean in scikit-learn. scikit-learn's
+  `costs_` (a log-determinant every iteration) is not computed.
+- `LinearRANSAC` is `RANSACRegressor` around `LinearRegression`, with
+  the same defaults. With scikit-learn installed, the subsets are drawn
+  exactly as it draws them, so `random_state` gives scikit-learn's fit.
+
+```python
+from scm2cpp_lasso import CDGraphicalLasso, LinearRANSAC
+prec = CDGraphicalLasso(alpha=0.1).fit(X).precision_
+fit = LinearRANSAC(random_state=0).fit(X, y)       # fit.coef_, fit.inlier_mask_
+```
+
+Measured through the package, one core, best of three, against
+scikit-learn on the same data (an AR(1) design with n = 20p for the
+graphical lasso, 100 iterations on both sides; a linear model with
+outliers for RANSAC, max_trials 1000):
+
+| problem | ours | scikit-learn | ratio | agreement |
+|---|---|---|---|---|
+| `CDGraphicalLasso`, p = 50 | 0.0069 s | 0.306 s | 44.5x | 100 / 100 iterations, precision to 1.8e-14 |
+| `CDGraphicalLasso`, p = 200 | 0.286 s | 1.62 s | 5.7x | precision to 2.6e-14 |
+| `CDGraphicalLasso`, p = 800 | 33.9 s | 43.8 s | 1.3x | precision to 5.0e-14 |
+| `LinearRANSAC`, 10,000 x 5 | 0.0254 s | 0.0916 s | 3.6x | 98 / 98 trials, same inliers |
+| `LinearRANSAC`, 100,000 x 10 | 0.153 s | 0.241 s | 1.6x | 97 / 97 trials, same inliers |
+| `LinearRANSAC`, 20,000 x 20 | 0.470 s | 1.19 s | 2.5x | 1,000 / 1,000 trials, same inliers |
+
+Both gains are mostly scikit-learn's Python between its compiled steps:
+the per-column bookkeeping of the graphical lasso, and the per-trial
+validation and fitting of RANSAC. They shrink as the compiled part
+grows (the graphical lasso at large p).
+
+## SGD, OMP and LARS: `SGDLinearClassifier`, `SGDLinearRegressor`, `GramOMP`, `GramLars`, `GramLassoLars`
+
+scikit-learn's own algorithms, translated:
+
+- `SGDLinearClassifier` and `SGDLinearRegressor` are
+  `SGDClassifier` / `SGDRegressor` at their defaults, run by
+  `_plain_sgd` with the points in order. Against
+  `shuffle=False` the weights are bit-identical. scikit-learn's
+  shuffling generator is not reproduced.
+- `GramOMP` is `OrthogonalMatchingPursuit` over the Gram matrix (its
+  route for n > p, taken here always; X'X and X'y by dsyrk and dgemv).
+- `GramLars` and `GramLassoLars` are `Lars` and `LassoLars` over the
+  Gram (their route for n > p, taken here always; the Gram formed in
+  numpy).
+
+```python
+from scm2cpp_lasso import SGDLinearClassifier, GramOMP, GramLassoLars
+SGDLinearClassifier().fit(X, y).predict(X_new)
+GramOMP(n_nonzero_coefs=10).fit(X, y).coef_
+GramLassoLars(alpha=0.01).fit(X, y).coef_
+```
+
+Each gives scikit-learn's coefficients, to the bit for SGD and to 1e-15
+for the rest, with the same number of epochs or steps. The speed is the
+repository README's table: 1.0-3.3x for SGD, 1.1-2.8x for OMP and
+1.2-5.0x for LARS, largest on small problems, where scikit-learn's
+Python between steps dominates.
+
 ## Logistic regression with L1
 
 `CovLogistic` solves L1-penalized logistic regression by