scm2cpp-lasso 0.15.0__tar.gz → 0.16.0__tar.gz

{scm2cpp_lasso-0.15.0/src/scm2cpp_lasso.egg-info → scm2cpp_lasso-0.16.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: scm2cpp-lasso
-Version: 0.15.0
-Summary: scikit-learn estimators (lasso, elastic net, ridge, ARD, SVM, SVR, logistic regression, GLMs, Huber, Theil-Sen, F-score selection, RFE, NMF, Gaussian mixture, FastICA, PCA, graphical lasso, RANSAC, SGD, OMP, LARS) translated from Scheme to C++ by scm2cpp; GPU batch solver and CUDA kernels included
+Version: 0.16.0
+Summary: scikit-learn estimators (lasso, elastic net, ridge, RidgeCV, ARD, SVM, SVR, logistic regression, GLMs, Huber, Theil-Sen, F-score selection, RFE, NMF, Gaussian mixture, FastICA, PCA, KernelPCA, graphical lasso, RANSAC, SGD, OMP, LARS) translated from Scheme to C++ by scm2cpp; GPU batch solver and CUDA kernels included
 Author: Hirotaka Niitsuma
 License: MIT
 Project-URL: Homepage, https://github.com/niitsuma/scm2cpp
@@ -41,6 +41,16 @@ from the sdist on a machine with `nvcc` on the path:
 $ pip install --no-binary scm2cpp-lasso scm2cpp-lasso
 ```
 
+Built from the sdist, the products and the eigen- and singular value
+problems go to Intel MKL where pkg-config knows it
+(`mkl-dynamic-lp64-gomp`, GNU OpenMP threading, as Gentoo's
+`sci-libs/mkl` installs it), else to an OpenBLAS with LAPACKE, else to
+the translated loops; `SCM2CPP_NO_MKL=1` skips MKL.  The translated
+loops that the dependence test proves independent run on OpenMP where
+the compiler has it (`SCM2CPP_NO_OPENMP=1` builds them serial).  Both
+use every core unless told otherwise: `OMP_NUM_THREADS` (and
+`MKL_NUM_THREADS` / `OPENBLAS_NUM_THREADS`) set the threads.
+
 The batched GPU solver, the cuBLAS variants and the `--cuda`
 estimators are then built as well; without `nvcc` the package installs
 and works exactly the same, minus those.
@@ -109,9 +119,7 @@ from scikit-learn's `BaseEstimator` and the matching mixin, so `clone`,
 `get_params` / `set_params`, `GridSearchCV`, `Pipeline` and
 `cross_val_score` take them, and all 44 pass
 `sklearn.utils.estimator_checks.check_estimator` (`check-sklearn.py`
-runs it; the expected failures are `DesignLasso`'s missing `n_iter_`,
-since the derived kernel does not count its sweeps, and
-`LBFGSLogisticRegressionCV`'s
+runs it; the one expected failure is `LBFGSLogisticRegressionCV`'s
 `n_iter_` of 0 for a C whose warm start is already converged, which
 scikit-learn's own `LogisticRegressionCV` reports on the same data).
 scikit-learn stays optional: without it a numpy-only stand-in gives
@@ -193,7 +201,8 @@ package picks the fastest one present.
 |---|---|---|---|---|
 | `GramLasso` | Lasso(precompute=True), fit(X, y) with the intercept | `lasso-cov.scm` | CPU | no |
 | `GramElasticNet` | ElasticNet(precompute=True), fit(X, y) with the intercept | `lasso-cov.scm` | CPU | no |
-| `GramRidge` | Ridge, fit(X, y) with the intercept, one or many targets | numpy (Cholesky) | CPU | Ridge |
+| `GramRidge` | Ridge, fit(X, y) with the intercept, one or many targets | `rfe.scm` (Cholesky) | loops / BLAS | Ridge |
+| `GramRidgeCV` | RidgeCV (cv=None), leave-one-out over every alpha from one eigenproblem | `ridge-gcv.scm` | loops / BLAS | no |
 | `GramMultiTaskLasso` | MultiTaskLasso, fit(X, Y) with the intercept | `lasso-cov.scm` | CPU | no |
 | `GramMultiTaskElasticNet` | MultiTaskElasticNet, fit(X, Y) with the intercept | `lasso-cov.scm` | CPU | no |
 | `GramGroupLasso` | (none; skglm GroupLasso), fit(X, y) with the intercept | `lasso-cov.scm` | CPU | - |
@@ -238,6 +247,7 @@ package picks the fastest one present.
 | `EMGaussianMixture` | GaussianMixture(covariance_type='full') | `gmm.scm` | loops / BLAS / --cuda | no |
 | `ParallelFastICA` | FastICA | `fastica.scm` | loops / BLAS / --cuda | no |
 | `CovPCA` | PCA(svd_solver='covariance_eigh') | `pca.scm` | loops / BLAS / cuBLAS | PCA |
+| `GramKernelPCA` | KernelPCA (no inverse_transform) | `kernel-pca.scm` | loops / BLAS | no |
 | `f_regression, f_classif` | f_regression, f_classif | `fscore.scm` | loops / BLAS / cuBLAS | no |
 | `SelectKBest, SelectPercentile` | SelectKBest, SelectPercentile | `fscore.scm` | loops / BLAS / cuBLAS | no |
 | `GramRFE` | RFE(Ridge), RFE(LinearRegression) | `rfe.scm` | loops / BLAS / cuBLAS | inside: Ridge |
@@ -560,6 +570,79 @@ on the GPU: the ridge is one eigendecomposition and the elastic net's
 sweeps are the lasso's; `fit_path_batch(l1_ratio=..)` is the GPU
 grid for the elastic net.
 
+## Leave-one-out ridge: `GramRidgeCV`
+
+`GramRidgeCV` is scikit-learn's `RidgeCV` with `cv=None`: the alpha
+whose leave-one-out mean squared error is least, without a refit per
+row. With the centred design Xc and H = Xc (Xc'Xc + alpha I)^-1 Xc'
+(plus 1/n for the intercept), the leave-one-out residual is
+(y - yhat) / (1 - diag H). One eigenproblem gives H = Z diag(1 /
+(lam + alpha)) Z' for every alpha: Z = Xc V from Xc'Xc = V diag(lam) V'
+when n > p (X'X by dsyrk), Z = Q diag(sqrt mu) from Xc Xc' = Q diag(mu)
+Q' otherwise. Then diag H and yhat at all the alphas are two products,
+(Z o Z) W and Z (R o W) with W[k, a] = 1 / (lam_k + alpha_a): the
+alphas cost one dgemm, where scikit-learn loops over them in numpy
+(`examples/kernel-only/ridge-gcv.scm`; the eigenproblem is LAPACK's
+dsyevd in the BLAS build).
+
+```python
+from scm2cpp_lasso import GramRidgeCV
+m = GramRidgeCV(alphas=np.logspace(-2, 4, 50)).fit(X, y)
+m.alpha_, m.best_score_, m.coef_, m.intercept_
+```
+
+It takes `fit_intercept`, `gcv_mode` (`"svd"` / `"eigen"` choose the
+two Zs), `store_cv_results` and `alpha_per_target`, y one target or
+several; `cv` and `scoring` must stay None. The chosen alpha is
+scikit-learn's, the coefficients agree to 1e-12 and `cv_results_` to
+1e-7 relative (the wide case near interpolation). Measured
+(`bench/eigen-compare.py`, the MKL build, best of three):
+
+| case | scikit-learn `RidgeCV` | `GramRidgeCV` | ratio, one core | ratio, 20 cores |
+|---|---|---|---|---|
+| 200,000 x 200, 3 alphas | 2.04 s | 0.80 s | 2.6x | 3.6x |
+| 200,000 x 200, 50 alphas | 26.6 s | 0.96 s | 27.7x | 29.0x |
+| 20,000 x 500, 50 alphas | 9.64 s | 0.33 s | 28.9x | 29.2x |
+| 2,000 x 5,000, 50 alphas | 3.16 s | 1.47 s | 2.1x | 4.5x |
+
+## Kernel PCA: `GramKernelPCA`
+
+`GramKernelPCA` is scikit-learn's `KernelPCA`: the kernel matrix
+(linear, rbf, poly, sigmoid, cosine or precomputed), centred in feature
+space, its largest eigenpairs with svd_flip's signs
+(`examples/kernel-only/kernel-pca.scm`). Where the eigenpairs come from
+is what differs:
+
+- fewer than 10 components of more than 200 rows, where scikit-learn
+  takes ARPACK (Lanczos, one matrix-vector product at a time): subspace
+  iteration on c + 25 columns, so each step is one dgemm, stopped when
+  every residual |K u - lam u| is under 1e-12 of the largest value
+  (falling back to the dense route if it is not reached);
+- otherwise LAPACK's dsyevr over the wanted index range, which is what
+  scipy's `eigh(subset_by_index=..)` runs for scikit-learn;
+- the linear kernel with no more components than features, n > p:
+  through Xc'Xc, p x p (the eigenvectors of Xc Xc' are Xc v / sqrt(lam)),
+  where scikit-learn forms the n x n kernel.
+
+```python
+from scm2cpp_lasso import GramKernelPCA
+Z = GramKernelPCA(5, kernel="rbf", gamma=0.05).fit_transform(X)
+```
+
+Eigenvalues, eigenvectors and transforms agree with scikit-learn's to
+1e-12 on every kernel; no `inverse_transform`. Measured
+(`bench/eigen-compare.py`, the MKL build, best of three):
+
+| case | scikit-learn `KernelPCA` | `GramKernelPCA` | ratio, one core | ratio, 20 cores |
+|---|---|---|---|---|
+| rbf, 3,000 x 20, 5 components | 1.11 s (ARPACK) | 0.35 s (subspace) | 3.2x | 3.0x |
+| rbf, 6,000 x 20, 5 components | 4.63 s (ARPACK) | 1.33 s (subspace) | 3.5x | 4.2x |
+| rbf, 3,000 x 20, 50 components | 3.48 s (dense) | 3.63 s (dense) | 1.0x | 1.2x |
+| linear, 5,000 x 50, 10 components | 19.8 s (ARPACK on 5,000 x 5,000) | 0.002 s (X'X) | 10,000x | 2,100x |
+
+The linear row is the algorithm (an n x n problem against a p x p one),
+not the code.
+
 ## Bayesian ridge
 
 `CovBayesianRidge` is scikit-learn's `BayesianRidge` -- the same model,
@@ -645,6 +728,15 @@ matrix (`KernelSVM`) or transposing X into columns
 | `L1LogisticRegression`, 20,000 x 500, C=0.05 | 1.20 s | 2.23 s | 1.9x | 106.77913 / 106.77915 | 100% |
 | `L1LogisticRegression`, 100,000 x 200, C=0.05 | 9.06 s | 18.7 s | 2.1x | 349.61154 / 349.61216 | 100% |
 
+On 2,000 x 20,000 neither side reaches the optimum at tol 1e-4 (the
+objective at tol 1e-10 is 0.0519588), and where each stops depends on
+the coordinate order: over eight seeds ours stops at 0.05240 to
+0.05251 (0.9 to 1.1% above the optimum), `LinearSVC`, whose order is
+random unless `random_state` is given, at 0.05242 to 0.05319 (0.9 to
+2.4%). When `LinearSVC` draws one of its worse orders the gap passes
+the script's 0.5% bound and the row prints `<-- disagrees`; ours is
+then the better solution of the two, so the flag there is not a fault.
+
 The regressors, from the same script (C = 1, epsilon 0.1):
 
 | problem | ours | scikit-learn | ratio | agreement |
@@ -894,25 +986,30 @@ times the ones with a counterpart here, sklearnex's estimators
 imported directly (not patched in) so that scikit-learn itself is
 timed in the same process. The ratios are scikit-learn's or
 sklearnex's time over the package's best (its CPU build, or the device
-build where one answers); the load average was 8 to 11, so the times
-move by some tens of percent.
+build where one answers). The package was built against Intel MKL
+(see the build section), its loops under OpenMP; the load average was
+about 2.3, so the times move by some percent.
 
 | estimator, n x p | iterations (sk / sklearnex / ours) | vs scikit-learn, one core | vs sklearnex, one core | vs sklearnex, 20 cores |
 |---|---|---|---|---|
-| LogisticRegression, 200,000 x 100 | 6 / 12 / 6 | 10.6x (device) | 8.4x (device) | 4.4x (device); on the CPU 0.5x |
-| Ridge (alpha 1), 200,000 x 200 | - | 2.1x | 1.2x | 0.3x |
-| PCA (10 components), 100,000 x 50 | - | 1.5x | 2.4x | 1.7x |
-| PCA (10 components), 50,000 x 500 | - | 1.1x | 1.4x | 0.5x |
-| SVC (rbf), 5,000 x 20 | 4,858 / 7 / 4,882 | 1.2x | 1.4x | 0.5x |
-| SVR (rbf), 5,000 x 20 | 12,821 / 10 / 12,824 | 1.3x | 1.4x | 0.4x |
-| RFE(Ridge), 20 of 200 kept, 20,000 x 200 | - | 216x | 128x | 78x |
-
-On one core the package is ahead everywhere. sklearnex uses every
-core by default, and the package's kernels are single-threaded, so
-with 20 cores sklearnex is faster on Ridge, the larger PCA and the
-SVMs; what stays ahead is the device build (logistic regression) and
-an algorithm that does less work (RFE: the Gram once, the Cholesky
-factor downdated per drop, where scikit-learn refits). sklearnex is
+| LogisticRegression, 200,000 x 100 | 6 / 12 / 6 | 6.8x (device) | 6.4x (device) | 1.8x (device); on the CPU 0.9x |
+| Ridge (alpha 1, no intercept), 200,000 x 200 | - | 1.7x | 0.8x | 2.3x |
+| PCA (10 components), 100,000 x 50 | - | 1.4x | 2.2x | 1.4x |
+| PCA (10 components), 50,000 x 500 | - | 1.1x | 1.4x | 1.6x (device); on the CPU 1.2x |
+| SVC (rbf), 5,000 x 20 | 4,858 / 7 / 4,882 | 1.4x | 1.6x | 0.5x |
+| SVR (rbf), 5,000 x 20 | 12,821 / 10 / 12,824 | 1.4x | 1.4x | 0.4x |
+| RFE(Ridge), 20 of 200 kept, 20,000 x 200 | - | 138x | 106x | 116x |
+
+On one core the package is ahead everywhere but Ridge, where both
+sides are one symmetric rank-k update of the same size and sklearnex
+is 0.180 s to 0.219 s. With 20 cores both sides use every core: the
+package's loops run under OpenMP and its products in MKL's threads,
+so it is ahead on Ridge (the Gram by dsyrk and a Cholesky solve), PCA,
+the device build of logistic regression and RFE (an algorithm that
+does less work: the Gram once, the Cholesky factor downdated per drop,
+where scikit-learn refits), level on logistic regression on the CPU,
+and behind on the SVMs: SMO takes thousands of short steps, each
+waiting on the last, where sklearnex's solver takes 7 to 10 long ones. sklearnex is
 not scikit-learn's arithmetic: its logistic regression takes 12
 iterations to scikit-learn's 6 and its coefficients differ from
 scikit-learn's by 6e-3, where the package's agree to 4e-15; its SVMs

scm2cpp_lasso-0.15.0/PKG-INFO → scm2cpp_lasso-0.16.0/README.md

@@ -1,21 +1,3 @@
-Metadata-Version: 2.4
-Name: scm2cpp-lasso
-Version: 0.15.0
-Summary: scikit-learn estimators (lasso, elastic net, ridge, ARD, SVM, SVR, logistic regression, GLMs, Huber, Theil-Sen, F-score selection, RFE, NMF, Gaussian mixture, FastICA, PCA, graphical lasso, RANSAC, SGD, OMP, LARS) translated from Scheme to C++ by scm2cpp; GPU batch solver and CUDA kernels included
-Author: Hirotaka Niitsuma
-License: MIT
-Project-URL: Homepage, https://github.com/niitsuma/scm2cpp
-Project-URL: Source, https://github.com/niitsuma/scm2cpp/tree/master/python/scm2cpp-lasso
-Keywords: lasso,elastic-net,ridge,svm,logistic-regression,coordinate-descent,cuda,compiler
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: C++
-Classifier: Topic :: Scientific/Engineering :: Mathematics
-Classifier: Intended Audience :: Science/Research
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: numpy>=1.17
-
 # scm2cpp-lasso
 
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/niitsuma/scm2cpp/blob/master/notebooks/compare-sklearn.ipynb)
@@ -41,6 +23,16 @@ from the sdist on a machine with `nvcc` on the path:
 $ pip install --no-binary scm2cpp-lasso scm2cpp-lasso
 ```
 
+Built from the sdist, the products and the eigen- and singular value
+problems go to Intel MKL where pkg-config knows it
+(`mkl-dynamic-lp64-gomp`, GNU OpenMP threading, as Gentoo's
+`sci-libs/mkl` installs it), else to an OpenBLAS with LAPACKE, else to
+the translated loops; `SCM2CPP_NO_MKL=1` skips MKL.  The translated
+loops that the dependence test proves independent run on OpenMP where
+the compiler has it (`SCM2CPP_NO_OPENMP=1` builds them serial).  Both
+use every core unless told otherwise: `OMP_NUM_THREADS` (and
+`MKL_NUM_THREADS` / `OPENBLAS_NUM_THREADS`) set the threads.
+
 The batched GPU solver, the cuBLAS variants and the `--cuda`
 estimators are then built as well; without `nvcc` the package installs
 and works exactly the same, minus those.
@@ -109,9 +101,7 @@ from scikit-learn's `BaseEstimator` and the matching mixin, so `clone`,
 `get_params` / `set_params`, `GridSearchCV`, `Pipeline` and
 `cross_val_score` take them, and all 44 pass
 `sklearn.utils.estimator_checks.check_estimator` (`check-sklearn.py`
-runs it; the expected failures are `DesignLasso`'s missing `n_iter_`,
-since the derived kernel does not count its sweeps, and
-`LBFGSLogisticRegressionCV`'s
+runs it; the one expected failure is `LBFGSLogisticRegressionCV`'s
 `n_iter_` of 0 for a C whose warm start is already converged, which
 scikit-learn's own `LogisticRegressionCV` reports on the same data).
 scikit-learn stays optional: without it a numpy-only stand-in gives
@@ -193,7 +183,8 @@ package picks the fastest one present.
 |---|---|---|---|---|
 | `GramLasso` | Lasso(precompute=True), fit(X, y) with the intercept | `lasso-cov.scm` | CPU | no |
 | `GramElasticNet` | ElasticNet(precompute=True), fit(X, y) with the intercept | `lasso-cov.scm` | CPU | no |
-| `GramRidge` | Ridge, fit(X, y) with the intercept, one or many targets | numpy (Cholesky) | CPU | Ridge |
+| `GramRidge` | Ridge, fit(X, y) with the intercept, one or many targets | `rfe.scm` (Cholesky) | loops / BLAS | Ridge |
+| `GramRidgeCV` | RidgeCV (cv=None), leave-one-out over every alpha from one eigenproblem | `ridge-gcv.scm` | loops / BLAS | no |
 | `GramMultiTaskLasso` | MultiTaskLasso, fit(X, Y) with the intercept | `lasso-cov.scm` | CPU | no |
 | `GramMultiTaskElasticNet` | MultiTaskElasticNet, fit(X, Y) with the intercept | `lasso-cov.scm` | CPU | no |
 | `GramGroupLasso` | (none; skglm GroupLasso), fit(X, y) with the intercept | `lasso-cov.scm` | CPU | - |
@@ -238,6 +229,7 @@ package picks the fastest one present.
 | `EMGaussianMixture` | GaussianMixture(covariance_type='full') | `gmm.scm` | loops / BLAS / --cuda | no |
 | `ParallelFastICA` | FastICA | `fastica.scm` | loops / BLAS / --cuda | no |
 | `CovPCA` | PCA(svd_solver='covariance_eigh') | `pca.scm` | loops / BLAS / cuBLAS | PCA |
+| `GramKernelPCA` | KernelPCA (no inverse_transform) | `kernel-pca.scm` | loops / BLAS | no |
 | `f_regression, f_classif` | f_regression, f_classif | `fscore.scm` | loops / BLAS / cuBLAS | no |
 | `SelectKBest, SelectPercentile` | SelectKBest, SelectPercentile | `fscore.scm` | loops / BLAS / cuBLAS | no |
 | `GramRFE` | RFE(Ridge), RFE(LinearRegression) | `rfe.scm` | loops / BLAS / cuBLAS | inside: Ridge |
@@ -560,6 +552,79 @@ on the GPU: the ridge is one eigendecomposition and the elastic net's
 sweeps are the lasso's; `fit_path_batch(l1_ratio=..)` is the GPU
 grid for the elastic net.
 
+## Leave-one-out ridge: `GramRidgeCV`
+
+`GramRidgeCV` is scikit-learn's `RidgeCV` with `cv=None`: the alpha
+whose leave-one-out mean squared error is least, without a refit per
+row. With the centred design Xc and H = Xc (Xc'Xc + alpha I)^-1 Xc'
+(plus 1/n for the intercept), the leave-one-out residual is
+(y - yhat) / (1 - diag H). One eigenproblem gives H = Z diag(1 /
+(lam + alpha)) Z' for every alpha: Z = Xc V from Xc'Xc = V diag(lam) V'
+when n > p (X'X by dsyrk), Z = Q diag(sqrt mu) from Xc Xc' = Q diag(mu)
+Q' otherwise. Then diag H and yhat at all the alphas are two products,
+(Z o Z) W and Z (R o W) with W[k, a] = 1 / (lam_k + alpha_a): the
+alphas cost one dgemm, where scikit-learn loops over them in numpy
+(`examples/kernel-only/ridge-gcv.scm`; the eigenproblem is LAPACK's
+dsyevd in the BLAS build).
+
+```python
+from scm2cpp_lasso import GramRidgeCV
+m = GramRidgeCV(alphas=np.logspace(-2, 4, 50)).fit(X, y)
+m.alpha_, m.best_score_, m.coef_, m.intercept_
+```
+
+It takes `fit_intercept`, `gcv_mode` (`"svd"` / `"eigen"` choose the
+two Zs), `store_cv_results` and `alpha_per_target`, y one target or
+several; `cv` and `scoring` must stay None. The chosen alpha is
+scikit-learn's, the coefficients agree to 1e-12 and `cv_results_` to
+1e-7 relative (the wide case near interpolation). Measured
+(`bench/eigen-compare.py`, the MKL build, best of three):
+
+| case | scikit-learn `RidgeCV` | `GramRidgeCV` | ratio, one core | ratio, 20 cores |
+|---|---|---|---|---|
+| 200,000 x 200, 3 alphas | 2.04 s | 0.80 s | 2.6x | 3.6x |
+| 200,000 x 200, 50 alphas | 26.6 s | 0.96 s | 27.7x | 29.0x |
+| 20,000 x 500, 50 alphas | 9.64 s | 0.33 s | 28.9x | 29.2x |
+| 2,000 x 5,000, 50 alphas | 3.16 s | 1.47 s | 2.1x | 4.5x |
+
+## Kernel PCA: `GramKernelPCA`
+
+`GramKernelPCA` is scikit-learn's `KernelPCA`: the kernel matrix
+(linear, rbf, poly, sigmoid, cosine or precomputed), centred in feature
+space, its largest eigenpairs with svd_flip's signs
+(`examples/kernel-only/kernel-pca.scm`). Where the eigenpairs come from
+is what differs:
+
+- fewer than 10 components of more than 200 rows, where scikit-learn
+  takes ARPACK (Lanczos, one matrix-vector product at a time): subspace
+  iteration on c + 25 columns, so each step is one dgemm, stopped when
+  every residual |K u - lam u| is under 1e-12 of the largest value
+  (falling back to the dense route if it is not reached);
+- otherwise LAPACK's dsyevr over the wanted index range, which is what
+  scipy's `eigh(subset_by_index=..)` runs for scikit-learn;
+- the linear kernel with no more components than features, n > p:
+  through Xc'Xc, p x p (the eigenvectors of Xc Xc' are Xc v / sqrt(lam)),
+  where scikit-learn forms the n x n kernel.
+
+```python
+from scm2cpp_lasso import GramKernelPCA
+Z = GramKernelPCA(5, kernel="rbf", gamma=0.05).fit_transform(X)
+```
+
+Eigenvalues, eigenvectors and transforms agree with scikit-learn's to
+1e-12 on every kernel; no `inverse_transform`. Measured
+(`bench/eigen-compare.py`, the MKL build, best of three):
+
+| case | scikit-learn `KernelPCA` | `GramKernelPCA` | ratio, one core | ratio, 20 cores |
+|---|---|---|---|---|
+| rbf, 3,000 x 20, 5 components | 1.11 s (ARPACK) | 0.35 s (subspace) | 3.2x | 3.0x |
+| rbf, 6,000 x 20, 5 components | 4.63 s (ARPACK) | 1.33 s (subspace) | 3.5x | 4.2x |
+| rbf, 3,000 x 20, 50 components | 3.48 s (dense) | 3.63 s (dense) | 1.0x | 1.2x |
+| linear, 5,000 x 50, 10 components | 19.8 s (ARPACK on 5,000 x 5,000) | 0.002 s (X'X) | 10,000x | 2,100x |
+
+The linear row is the algorithm (an n x n problem against a p x p one),
+not the code.
+
 ## Bayesian ridge
 
 `CovBayesianRidge` is scikit-learn's `BayesianRidge` -- the same model,
@@ -645,6 +710,15 @@ matrix (`KernelSVM`) or transposing X into columns
 | `L1LogisticRegression`, 20,000 x 500, C=0.05 | 1.20 s | 2.23 s | 1.9x | 106.77913 / 106.77915 | 100% |
 | `L1LogisticRegression`, 100,000 x 200, C=0.05 | 9.06 s | 18.7 s | 2.1x | 349.61154 / 349.61216 | 100% |
 
+On 2,000 x 20,000 neither side reaches the optimum at tol 1e-4 (the
+objective at tol 1e-10 is 0.0519588), and where each stops depends on
+the coordinate order: over eight seeds ours stops at 0.05240 to
+0.05251 (0.9 to 1.1% above the optimum), `LinearSVC`, whose order is
+random unless `random_state` is given, at 0.05242 to 0.05319 (0.9 to
+2.4%). When `LinearSVC` draws one of its worse orders the gap passes
+the script's 0.5% bound and the row prints `<-- disagrees`; ours is
+then the better solution of the two, so the flag there is not a fault.
+
 The regressors, from the same script (C = 1, epsilon 0.1):
 
 | problem | ours | scikit-learn | ratio | agreement |
@@ -894,25 +968,30 @@ times the ones with a counterpart here, sklearnex's estimators
 imported directly (not patched in) so that scikit-learn itself is
 timed in the same process. The ratios are scikit-learn's or
 sklearnex's time over the package's best (its CPU build, or the device
-build where one answers); the load average was 8 to 11, so the times
-move by some tens of percent.
+build where one answers). The package was built against Intel MKL
+(see the build section), its loops under OpenMP; the load average was
+about 2.3, so the times move by some percent.
 
 | estimator, n x p | iterations (sk / sklearnex / ours) | vs scikit-learn, one core | vs sklearnex, one core | vs sklearnex, 20 cores |
 |---|---|---|---|---|
-| LogisticRegression, 200,000 x 100 | 6 / 12 / 6 | 10.6x (device) | 8.4x (device) | 4.4x (device); on the CPU 0.5x |
-| Ridge (alpha 1), 200,000 x 200 | - | 2.1x | 1.2x | 0.3x |
-| PCA (10 components), 100,000 x 50 | - | 1.5x | 2.4x | 1.7x |
-| PCA (10 components), 50,000 x 500 | - | 1.1x | 1.4x | 0.5x |
-| SVC (rbf), 5,000 x 20 | 4,858 / 7 / 4,882 | 1.2x | 1.4x | 0.5x |
-| SVR (rbf), 5,000 x 20 | 12,821 / 10 / 12,824 | 1.3x | 1.4x | 0.4x |
-| RFE(Ridge), 20 of 200 kept, 20,000 x 200 | - | 216x | 128x | 78x |
-
-On one core the package is ahead everywhere. sklearnex uses every
-core by default, and the package's kernels are single-threaded, so
-with 20 cores sklearnex is faster on Ridge, the larger PCA and the
-SVMs; what stays ahead is the device build (logistic regression) and
-an algorithm that does less work (RFE: the Gram once, the Cholesky
-factor downdated per drop, where scikit-learn refits). sklearnex is
+| LogisticRegression, 200,000 x 100 | 6 / 12 / 6 | 6.8x (device) | 6.4x (device) | 1.8x (device); on the CPU 0.9x |
+| Ridge (alpha 1, no intercept), 200,000 x 200 | - | 1.7x | 0.8x | 2.3x |
+| PCA (10 components), 100,000 x 50 | - | 1.4x | 2.2x | 1.4x |
+| PCA (10 components), 50,000 x 500 | - | 1.1x | 1.4x | 1.6x (device); on the CPU 1.2x |
+| SVC (rbf), 5,000 x 20 | 4,858 / 7 / 4,882 | 1.4x | 1.6x | 0.5x |
+| SVR (rbf), 5,000 x 20 | 12,821 / 10 / 12,824 | 1.4x | 1.4x | 0.4x |
+| RFE(Ridge), 20 of 200 kept, 20,000 x 200 | - | 138x | 106x | 116x |
+
+On one core the package is ahead everywhere but Ridge, where both
+sides are one symmetric rank-k update of the same size and sklearnex
+is 0.180 s to 0.219 s. With 20 cores both sides use every core: the
+package's loops run under OpenMP and its products in MKL's threads,
+so it is ahead on Ridge (the Gram by dsyrk and a Cholesky solve), PCA,
+the device build of logistic regression and RFE (an algorithm that
+does less work: the Gram once, the Cholesky factor downdated per drop,
+where scikit-learn refits), level on logistic regression on the CPU,
+and behind on the SVMs: SMO takes thousands of short steps, each
+waiting on the last, where sklearnex's solver takes 7 to 10 long ones. sklearnex is
 not scikit-learn's arithmetic: its logistic regression takes 12
 iterations to scikit-learn's 6 and its coefficients differ from
 scikit-learn's by 6e-3, where the package's agree to 4e-15; its SVMs

{scm2cpp_lasso-0.15.0 → scm2cpp_lasso-0.16.0}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scm2cpp-lasso"
-version = "0.15.0"
-description = "scikit-learn estimators (lasso, elastic net, ridge, ARD, SVM, SVR, logistic regression, GLMs, Huber, Theil-Sen, F-score selection, RFE, NMF, Gaussian mixture, FastICA, PCA, graphical lasso, RANSAC, SGD, OMP, LARS) translated from Scheme to C++ by scm2cpp; GPU batch solver and CUDA kernels included"
+version = "0.16.0"
+description = "scikit-learn estimators (lasso, elastic net, ridge, RidgeCV, ARD, SVM, SVR, logistic regression, GLMs, Huber, Theil-Sen, F-score selection, RFE, NMF, Gaussian mixture, FastICA, PCA, KernelPCA, graphical lasso, RANSAC, SGD, OMP, LARS) translated from Scheme to C++ by scm2cpp; GPU batch solver and CUDA kernels included"
 readme = "README.md"
 requires-python = ">=3.8"
 license = {text = "MIT"}

{scm2cpp_lasso-0.15.0 → scm2cpp_lasso-0.16.0}/setup.py

@@ -41,6 +41,32 @@ kernel = Extension(
 )
 
 
+def mkl_flags():
+    """Intel MKL where pkg-config knows it (mkl-dynamic-lp64-gomp: the GNU
+    OpenMP threading layer, so MKL's threads and the -P omp loops share
+    one libgomp) and a test program links, with SCM2CPP_MKL defined for
+    scm2cpp-blas.hpp's headers; else None.  SCM2CPP_NO_MKL=1 skips it."""
+    if os.name == "nt" or os.environ.get("SCM2CPP_NO_MKL"):
+        return None
+    try:
+        pc = subprocess.run(["pkg-config", "--cflags", "--libs", "mkl-dynamic-lp64-gomp"],
+                            capture_output=True, text=True, check=True)
+    except (subprocess.CalledProcessError, OSError):
+        return None
+    flags = pc.stdout.split() + ["-DSCM2CPP_MKL"]
+    probe = "#include <mkl_cblas.h>\n#include <mkl_lapacke.h>\nint main(){double a=1,b=0,w=0;" \
+            "cblas_dgemv(CblasRowMajor,CblasNoTrans,1,1,1.0,&a,1,&a,1,0.0,&b,1);" \
+            "LAPACKE_dsyevd(LAPACK_ROW_MAJOR,'V','U',1,&a,1,&w);" \
+            "return b==1?0:1;}\n"
+    cxx = os.environ.get("CXX", "c++")
+    try:
+        r = subprocess.run([cxx, "-x", "c++", "-", "-o", os.devnull] + flags,
+                           input=probe, capture_output=True, text=True)
+    except OSError:
+        return None
+    return flags if r.returncode == 0 else None
+
+
 def cblas_flags():
     """How to compile and link against a CBLAS here, or None.
 
@@ -49,9 +75,25 @@ def cblas_flags():
     kernel's BLAS variant needs cblas.h and a library; without them the
     loop variant is built instead (same answers, the Gram as a scalar
     loop nest).
+
+    On Windows there is no pkg-config and no system OpenBLAS: the
+    directory of an unpacked OpenBLAS release (include/cblas.h,
+    lib/libopenblas.lib) is named by SCM2CPP_OPENBLAS_DIR, and the
+    wheel build bundles its bin/libopenblas.dll (delvewheel).
     """
-    if os.name == "nt" or os.environ.get("SCM2CPP_NO_BLAS"):
+    if os.environ.get("SCM2CPP_NO_BLAS"):
         return None
+    mkl = mkl_flags()
+    if mkl is not None:
+        return mkl
+    if os.name == "nt":
+        root = os.environ.get("SCM2CPP_OPENBLAS_DIR")
+        if not root:
+            return None
+        inc, lib = Path(root) / "include", Path(root) / "lib" / "libopenblas.lib"
+        if not ((inc / "cblas.h").exists() and lib.exists()):
+            return None
+        return ["-I" + str(inc), str(lib)]
     cxx = os.environ.get("CXX", "c++")
     candidates = []
     try:
@@ -61,8 +103,11 @@ def cblas_flags():
     except (subprocess.CalledProcessError, OSError):
         pass
     candidates.append(["-lopenblas"])
-    probe = "#include <cblas.h>\nint main(){double a=1,b=0;" \
+    # the BLAS variants call LAPACKE too (lapack-syevd! for sym-eigen),
+    # so a CBLAS without LAPACKE counts as none
+    probe = "#include <cblas.h>\n#include <lapacke.h>\nint main(){double a=1,b=0,w=0;" \
             "cblas_dgemv(CblasRowMajor,CblasNoTrans,1,1,1.0,&a,1,&a,1,0.0,&b,1);" \
+            "LAPACKE_dsyevd(LAPACK_ROW_MAJOR,'V','U',1,&a,1,&w);" \
             "return b==1?0:1;}\n"
     for flags in candidates:
         try:
@@ -75,6 +120,32 @@ def cblas_flags():
     return None
 
 
+def openmp_flags():
+    """-fopenmp where the compiler takes the directives the translator
+    writes (-P omp: parallel for, reduction, an if clause on the work),
+    else nothing -- the same sources then compile serial.  Not on
+    Windows (MSVC's OpenMP needs vcomp140.dll beside the wheel) nor
+    with SCM2CPP_NO_OPENMP set.  OMP_NUM_THREADS sets the threads at
+    run time, all cores by default.
+    """
+    if os.name == "nt" or os.environ.get("SCM2CPP_NO_OPENMP"):
+        return []
+    cxx = os.environ.get("CXX", "c++")
+    probe = "int main(){double s=0;int n=4;\n" \
+            "#pragma omp parallel for reduction(+:s) if(n > 1)\n" \
+            "for(int i = 0 ; i < n ; i = (i+1)) s = (s+i);\n" \
+            "return s==6?0:1;}\n"
+    try:
+        r = subprocess.run([cxx, "-fopenmp", "-x", "c++", "-", "-o", os.devnull],
+                           input=probe, capture_output=True, text=True)
+    except OSError:
+        return []
+    return ["-fopenmp"] if r.returncode == 0 else []
+
+
+_omp = openmp_flags()
+
+
 # The derived kernel: lasso-design.scm, the plain residual-form
 # coordinate descent over X as numpy holds it, turned into the Gram form by the translator
 # (--derive) with its products as CBLAS calls (--blas).  Which variant
@@ -185,7 +256,8 @@ classifiers = Extension(
     "scm2cpp_lasso._classifiers",
     sources=[GEN_REL + "/classifiers_capi.cpp"],
     include_dirs=[GEN_REL],
-    extra_compile_args=_cls_flags,
+    extra_compile_args=_cls_flags + _omp,
+    extra_link_args=_omp,
 )
 
 # The derived NMF, in the same variants as the derived lasso.
@@ -194,7 +266,7 @@ if _blas is not None:
         "scm2cpp_lasso._nmf_blas",
         sources=[GEN_REL + "/nmf_blas/nmf_cd_capi.cpp"],
         include_dirs=[GEN_REL + "/nmf_blas", GEN_REL],
-        extra_compile_args=CXXFLAGS + ["-O3", "-fstrict-overflow"]
+        extra_compile_args=_cls_flags
         + [f for f in _blas if f.startswith("-I") or f.startswith("-D")],
         extra_link_args=[f for f in _blas if not f.startswith("-I")
                          and not f.startswith("-D")],
@@ -215,17 +287,18 @@ if _blas is not None:
         "scm2cpp_lasso._gramops_blas",
         sources=[GEN_REL + "/gramops_blas/gramops_capi.cpp"],
         include_dirs=[GEN_REL + "/gramops_blas", GEN_REL],
-        extra_compile_args=CXXFLAGS + ["-O3", "-fstrict-overflow"]
+        extra_compile_args=_cls_flags + _omp
         + [f for f in _blas if f.startswith("-I") or f.startswith("-D")],
-        extra_link_args=[f for f in _blas if not f.startswith("-I")
-                         and not f.startswith("-D")],
+        extra_link_args=_omp + [f for f in _blas if not f.startswith("-I")
+                                and not f.startswith("-D")],
     )
 else:
     gramops = Extension(
         "scm2cpp_lasso._gramops_loops",
         sources=[GEN_REL + "/gramops_loops/gramops_capi.cpp"],
         include_dirs=[GEN_REL + "/gramops_loops", GEN_REL],
-        extra_compile_args=_cls_flags,
+        extra_compile_args=_cls_flags + _omp,
+        extra_link_args=_omp,
     )
 
 _cuda = cuda_home()
@@ -245,7 +318,7 @@ if _cuda is not None:
         include_dirs=[GEN_REL + "/nmf_cublas", GEN_REL, str(_cuda / "include")],
         library_dirs=[str(_cuda / "lib64"), str(_cuda / "lib")],
         libraries=["cublas", "cudart"],
-        extra_compile_args=CXXFLAGS + ["-O3", "-fstrict-overflow"],
+        extra_compile_args=_cls_flags,
     ))
 
     extensions.append(Extension(
@@ -254,7 +327,7 @@ if _cuda is not None:
         include_dirs=[GEN_REL + "/accel_cublas", GEN_REL, str(_cuda / "include")],
         library_dirs=[str(_cuda / "lib64"), str(_cuda / "lib")],
         libraries=["cublas", "cudart"],
-        extra_compile_args=CXXFLAGS + ["-O3", "-fstrict-overflow"],
+        extra_compile_args=_cls_flags,
     ))
 
 # A wheel is tagged py3-none-<platform>: the extensions are shared