scm2cpp-lasso 0.5.3.1__tar.gz → 0.7.0__tar.gz

{scm2cpp_lasso-0.5.3.1 → scm2cpp_lasso-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scm2cpp-lasso
-Version: 0.5.3.1
+Version: 0.7.0
 Summary: Lasso by covariance-update coordinate descent, translated from Scheme to C++ by scm2cpp; GPU batch solver included
 Author: Hirotaka Niitsuma
 License: MIT
@@ -114,8 +114,8 @@ celer and skglm are built for very large sparse designs, where their
 screening rules dominate; at this size they pay their setup per fit.
 cuML parallelises *within* one fit, which wins when a fit is large; at
 this size its launch overhead dominates, while our GPU row
-parallelises *across* lambdas -- one CUDA thread per penalty, the axis
-a cross-validation grid actually offers.
+parallelises *across* lambdas -- one block of CUDA threads per
+penalty, the axis a cross-validation grid actually offers.
 
 `CovLassoCV` is scikit-learn's `LassoCV` over this machinery -- same
 grid construction, same contiguous folds, same minimum-mean-MSE choice
@@ -123,23 +123,80 @@ grid construction, same contiguous folds, same minimum-mean-MSE choice
 *subtraction* (the Gram is additive over rows, so G - Xf'Xf costs one
 fold's work where rebuilding costs four folds'), and once the Grams
 exist nothing downstream ever touches the n rows again -- which is why
-the gap grows with n.  On the GPU every fold and every alpha can run
-as one thread of a single launch; that launch replicates each fold's
-Gram per alpha, so it is chosen only while the replication stays under
-512 MB, the CPU warm path taking over at large p.  Measured at cv=5,
-100 alphas, one CPU core:
-
-| n       | p    | `CovLassoCV` | sklearn `LassoCV` |
-|---------|------|--------------|-------------------|
-| 1,800   | 200  | 0.23 s       | 0.20 s            |
-| 5,000   | 1000 | 1.1 s        | 3.4 s             |
-| 100,000 | 200  | 0.44 s       | 5.7 s             |
-| 100,000 | 500  | 0.90 s       | 8.0 s             |
-
-At n=100,000 the chosen alpha is identical and the coefficients agree
-to 7e-7; at small n a near-tie can leave the picks one grid step
-apart, which sklearn resolves our way once its tolerance is
-tightened.  GPU and CPU agree to 1e-14 wherever both run.
+the gap grows with n.  With a GPU the whole grid is one launch: one
+*block* of threads per (fold, alpha) cell, the fold's Gram read in
+place (cv Gram matrices on the device, not cv x 100), thread 0 of the
+block doing the coordinate step and the block spreading the O(p)
+update of the correlations.  cv x 100 = 500 cells are too few
+problems to fill a device one thread each, and a block per problem is
+what turns that count into enough parallel work; `bench/cv-grid-designs.py`
+times the alternatives (one thread per cell, warm runs of several
+alphas per thread or block) side by side, and the block per cell wins
+at every size tried.  The GPU is the default when a device answers
+(`force_cpu` takes the warm CPU path).  Measured at cv=5, 100 alphas,
+one CPU core, sklearn 1.9.0, best of three runs, whole estimator
+including the Grams:
+
+| n       | p    | CPU    | CUDA   | sklearn `LassoCV` |
+|---------|------|--------|--------|-------------------|
+| 1,800   | 200  | 0.36 s | 0.03 s | 0.23 s            |
+| 5,000   | 1000 | 3.9 s  | 0.39 s | 2.7 s             |
+| 100,000 | 200  | 0.75 s | 0.60 s | 3.9 s             |
+| 100,000 | 500  | 2.0 s  | 1.8 s  | 12.6 s            |
+
+(This table was taken on a machine sharing its cores with other jobs,
+load 14 on 20 threads, so its absolute times run some 2-3x above the
+tables earlier in this README; the ratios are the point.)  The grid
+launch itself takes 9 ms at n=1,800 and 64 ms at n=5,000, p=1000; at
+n=100,000 the CUDA column is nearly all Gram products, which the CPU
+path pays as well.  Large n is where the machinery pays either way:
+nothing downstream of the Grams touches the 100,000 rows again.  At
+n=5,000, p=1000 sklearn's raw-X descent beats our Gram-heavy CPU path,
+and the GPU makes up the difference seven times over.  CPU and CUDA
+choose the same alpha everywhere and agree to 1e-14; against sklearn
+the pick is exact at n=5,000 and one grid step apart on a near-tie
+elsewhere (mean-MSE relative gap at most 2.6e-4), and sklearn at
+tol=1e-10 picks our alpha at every size.
+
+`CovMultiTaskLasso` and `CovMultiTaskLassoCV` are the multi-task
+family over the same machinery: scikit-learn's `MultiTaskLasso`,
+`MultiTaskElasticNet` (through `l1_ratio`), `MultiTaskLassoCV` and
+`MultiTaskElasticNetCV`.  The penalty ties each feature's row of W
+together across tasks, so the coordinate update becomes a block soft
+threshold on the row's L2 norm; C = X'Y - GW is maintained per task
+exactly as the single-task C is, and nothing downstream of the Grams
+touches the n rows -- the kernel is the same translated Scheme
+(`mt-descend`).  Folds are independent and both the descent (ctypes)
+and the products (BLAS) release the GIL, so `n_jobs` threads scale the
+CV across folds; the default is sequential, as scikit-learn's
+`n_jobs=None` is, and the benchmark pins BLAS to one thread on both
+sides either way.  With a GPU the CV grid is one launch exactly as
+`CovLassoCV`'s is, a block per (fold, alpha) cell with the p x T
+coefficient block in shared memory (`force_cpu` opts out).  Same
+protocol as above, 8 tasks; single fits at alpha = 0.1 lambda_max;
+the CV pairs share the 100-alpha grid:
+
+| estimator             | n       | p   | ours   | ours `n_jobs=5` | ours CUDA | sklearn |
+|-----------------------|---------|-----|--------|-----------------|-----------|---------|
+| MultiTaskLasso        | 100,000 | 200 | 0.15 s | --              | --        | 0.80 s  |
+| MultiTaskLasso        | 100,000 | 500 | 0.53 s | --              | --        | 1.9 s   |
+| MultiTaskLassoCV      | 1,800   | 200 | 1.5 s  | 0.33 s          | 0.08 s    | 0.51 s  |
+| MultiTaskLassoCV      | 100,000 | 200 | 1.6 s  | 0.62 s          | 1.7 s     | 76 s    |
+| MultiTaskElasticNetCV | 1,800   | 200 | 1.6 s  | 0.51 s          | 0.10 s    | 1.0 s   |
+| MultiTaskElasticNetCV | 100,000 | 200 | 1.6 s  | 0.68 s          | 1.9 s     | 164 s   |
+
+(`MultiTaskElasticNet` single fits time the same as `MultiTaskLasso`'s.
+The CUDA column was timed on the busier day of the `CovLassoCV` table
+above, when the sequential CPU rows read 1.5 / 2.0 / 1.6 / 2.1 s and
+sklearn's 0.56 / 114 / 0.99 / 241 s.)  At n=1,800 our sequential CV
+is slower than sklearn's: the default tol=1e-8 runs more sweeps than
+sklearn's 1e-4 dual-gap stop, and at that size there are no rows to
+save -- and the GPU grid takes the whole thing in a tenth of a second.
+At n=100,000 the Gram route is 48x and 100x ahead, essentially
+unchanged from n=1,800 in absolute time; the CUDA column there is
+the Gram products, the same ones the CPU path forms.  Coefficients agree with scikit-learn's to 1e-15 at tight
+tolerance, and the CV pair picks the same alpha with coefficients to
+1e-13.
 
 ## Which method
 
@@ -206,9 +263,14 @@ groups it reduces exactly to the lasso -- verified against sklearn to
 
 `bootstrap` draws pairs-bootstrap resamples and refits them all at
 one lambda.  Each resample's Gram matrix is `X' diag(m) X` for its
-multiplicity counts `m` -- one BLAS product -- and because the
-problems are independent, the descents run as one batch: on the GPU,
-one thread per resample, each reading its own Gram matrix.
+multiplicity counts `m`, written as `Xs' Xs` with `Xs = diag(sqrt m) X`
+so that numpy hands it to the symmetric rank-k product (`dsyrk`,
+half the flops of `X' (diag(m) X)`) -- one BLAS call -- and because
+the problems are independent, the descents run as one batch: on the
+GPU, one block of threads per resample, each reading its own Gram
+matrix.  The products are the cost: at n=5000, p=1000 a hundred
+resamples took 18.1 s as general products and 11.2 s as `dsyrk`,
+descents included.
 
 ```python
 betas = model.bootstrap(lam, n_boot=500, seed=0)   # (500, p)

{scm2cpp_lasso-0.5.3.1 → scm2cpp_lasso-0.7.0}/README.md

@@ -96,8 +96,8 @@ celer and skglm are built for very large sparse designs, where their
 screening rules dominate; at this size they pay their setup per fit.
 cuML parallelises *within* one fit, which wins when a fit is large; at
 this size its launch overhead dominates, while our GPU row
-parallelises *across* lambdas -- one CUDA thread per penalty, the axis
-a cross-validation grid actually offers.
+parallelises *across* lambdas -- one block of CUDA threads per
+penalty, the axis a cross-validation grid actually offers.
 
 `CovLassoCV` is scikit-learn's `LassoCV` over this machinery -- same
 grid construction, same contiguous folds, same minimum-mean-MSE choice
@@ -105,23 +105,80 @@ grid construction, same contiguous folds, same minimum-mean-MSE choice
 *subtraction* (the Gram is additive over rows, so G - Xf'Xf costs one
 fold's work where rebuilding costs four folds'), and once the Grams
 exist nothing downstream ever touches the n rows again -- which is why
-the gap grows with n.  On the GPU every fold and every alpha can run
-as one thread of a single launch; that launch replicates each fold's
-Gram per alpha, so it is chosen only while the replication stays under
-512 MB, the CPU warm path taking over at large p.  Measured at cv=5,
-100 alphas, one CPU core:
-
-| n       | p    | `CovLassoCV` | sklearn `LassoCV` |
-|---------|------|--------------|-------------------|
-| 1,800   | 200  | 0.23 s       | 0.20 s            |
-| 5,000   | 1000 | 1.1 s        | 3.4 s             |
-| 100,000 | 200  | 0.44 s       | 5.7 s             |
-| 100,000 | 500  | 0.90 s       | 8.0 s             |
-
-At n=100,000 the chosen alpha is identical and the coefficients agree
-to 7e-7; at small n a near-tie can leave the picks one grid step
-apart, which sklearn resolves our way once its tolerance is
-tightened.  GPU and CPU agree to 1e-14 wherever both run.
+the gap grows with n.  With a GPU the whole grid is one launch: one
+*block* of threads per (fold, alpha) cell, the fold's Gram read in
+place (cv Gram matrices on the device, not cv x 100), thread 0 of the
+block doing the coordinate step and the block spreading the O(p)
+update of the correlations.  cv x 100 = 500 cells are too few
+problems to fill a device one thread each, and a block per problem is
+what turns that count into enough parallel work; `bench/cv-grid-designs.py`
+times the alternatives (one thread per cell, warm runs of several
+alphas per thread or block) side by side, and the block per cell wins
+at every size tried.  The GPU is the default when a device answers
+(`force_cpu` takes the warm CPU path).  Measured at cv=5, 100 alphas,
+one CPU core, sklearn 1.9.0, best of three runs, whole estimator
+including the Grams:
+
+| n       | p    | CPU    | CUDA   | sklearn `LassoCV` |
+|---------|------|--------|--------|-------------------|
+| 1,800   | 200  | 0.36 s | 0.03 s | 0.23 s            |
+| 5,000   | 1000 | 3.9 s  | 0.39 s | 2.7 s             |
+| 100,000 | 200  | 0.75 s | 0.60 s | 3.9 s             |
+| 100,000 | 500  | 2.0 s  | 1.8 s  | 12.6 s            |
+
+(This table was taken on a machine sharing its cores with other jobs,
+load 14 on 20 threads, so its absolute times run some 2-3x above the
+tables earlier in this README; the ratios are the point.)  The grid
+launch itself takes 9 ms at n=1,800 and 64 ms at n=5,000, p=1000; at
+n=100,000 the CUDA column is nearly all Gram products, which the CPU
+path pays as well.  Large n is where the machinery pays either way:
+nothing downstream of the Grams touches the 100,000 rows again.  At
+n=5,000, p=1000 sklearn's raw-X descent beats our Gram-heavy CPU path,
+and the GPU makes up the difference seven times over.  CPU and CUDA
+choose the same alpha everywhere and agree to 1e-14; against sklearn
+the pick is exact at n=5,000 and one grid step apart on a near-tie
+elsewhere (mean-MSE relative gap at most 2.6e-4), and sklearn at
+tol=1e-10 picks our alpha at every size.
+
+`CovMultiTaskLasso` and `CovMultiTaskLassoCV` are the multi-task
+family over the same machinery: scikit-learn's `MultiTaskLasso`,
+`MultiTaskElasticNet` (through `l1_ratio`), `MultiTaskLassoCV` and
+`MultiTaskElasticNetCV`.  The penalty ties each feature's row of W
+together across tasks, so the coordinate update becomes a block soft
+threshold on the row's L2 norm; C = X'Y - GW is maintained per task
+exactly as the single-task C is, and nothing downstream of the Grams
+touches the n rows -- the kernel is the same translated Scheme
+(`mt-descend`).  Folds are independent and both the descent (ctypes)
+and the products (BLAS) release the GIL, so `n_jobs` threads scale the
+CV across folds; the default is sequential, as scikit-learn's
+`n_jobs=None` is, and the benchmark pins BLAS to one thread on both
+sides either way.  With a GPU the CV grid is one launch exactly as
+`CovLassoCV`'s is, a block per (fold, alpha) cell with the p x T
+coefficient block in shared memory (`force_cpu` opts out).  Same
+protocol as above, 8 tasks; single fits at alpha = 0.1 lambda_max;
+the CV pairs share the 100-alpha grid:
+
+| estimator             | n       | p   | ours   | ours `n_jobs=5` | ours CUDA | sklearn |
+|-----------------------|---------|-----|--------|-----------------|-----------|---------|
+| MultiTaskLasso        | 100,000 | 200 | 0.15 s | --              | --        | 0.80 s  |
+| MultiTaskLasso        | 100,000 | 500 | 0.53 s | --              | --        | 1.9 s   |
+| MultiTaskLassoCV      | 1,800   | 200 | 1.5 s  | 0.33 s          | 0.08 s    | 0.51 s  |
+| MultiTaskLassoCV      | 100,000 | 200 | 1.6 s  | 0.62 s          | 1.7 s     | 76 s    |
+| MultiTaskElasticNetCV | 1,800   | 200 | 1.6 s  | 0.51 s          | 0.10 s    | 1.0 s   |
+| MultiTaskElasticNetCV | 100,000 | 200 | 1.6 s  | 0.68 s          | 1.9 s     | 164 s   |
+
+(`MultiTaskElasticNet` single fits time the same as `MultiTaskLasso`'s.
+The CUDA column was timed on the busier day of the `CovLassoCV` table
+above, when the sequential CPU rows read 1.5 / 2.0 / 1.6 / 2.1 s and
+sklearn's 0.56 / 114 / 0.99 / 241 s.)  At n=1,800 our sequential CV
+is slower than sklearn's: the default tol=1e-8 runs more sweeps than
+sklearn's 1e-4 dual-gap stop, and at that size there are no rows to
+save -- and the GPU grid takes the whole thing in a tenth of a second.
+At n=100,000 the Gram route is 48x and 100x ahead, essentially
+unchanged from n=1,800 in absolute time; the CUDA column there is
+the Gram products, the same ones the CPU path forms.  Coefficients agree with scikit-learn's to 1e-15 at tight
+tolerance, and the CV pair picks the same alpha with coefficients to
+1e-13.
 
 ## Which method
 
@@ -188,9 +245,14 @@ groups it reduces exactly to the lasso -- verified against sklearn to
 
 `bootstrap` draws pairs-bootstrap resamples and refits them all at
 one lambda.  Each resample's Gram matrix is `X' diag(m) X` for its
-multiplicity counts `m` -- one BLAS product -- and because the
-problems are independent, the descents run as one batch: on the GPU,
-one thread per resample, each reading its own Gram matrix.
+multiplicity counts `m`, written as `Xs' Xs` with `Xs = diag(sqrt m) X`
+so that numpy hands it to the symmetric rank-k product (`dsyrk`,
+half the flops of `X' (diag(m) X)`) -- one BLAS call -- and because
+the problems are independent, the descents run as one batch: on the
+GPU, one block of threads per resample, each reading its own Gram
+matrix.  The products are the cost: at n=5000, p=1000 a hundred
+resamples took 18.1 s as general products and 11.2 s as `dsyrk`,
+descents included.
 
 ```python
 betas = model.bootstrap(lam, n_boot=500, seed=0)   # (500, p)

{scm2cpp_lasso-0.5.3.1 → scm2cpp_lasso-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scm2cpp-lasso"
-version = "0.5.3.1"
+version = "0.7.0"
 description = "Lasso by covariance-update coordinate descent, translated from Scheme to C++ by scm2cpp; GPU batch solver included"
 readme = "README.md"
 requires-python = ">=3.8"