ipss 1.1.12__tar.gz → 1.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: ipss
3
- Version: 1.1.12
3
+ Version: 1.2.0
4
4
  Summary: Python implementation of integrated path stability selection (IPSS)
5
5
  Author-email: Omar Melikechi <omar.melikechi@gmail.com>
6
6
  License-Expression: MIT
@@ -23,16 +23,14 @@ Dynamic: license-file
23
23
 
24
24
  > Fast, flexible feature selection with false discovery control
25
25
 
26
- ## Associated papers
26
+ ## Papers
27
27
 
28
28
  - **Integrated path stability selection**
29
29
  [*Journal of the American Statistical Association*](https://doi.org/10.1080/01621459.2025.2525589) • [arXiv](https://arxiv.org/abs/2403.15877)
30
30
 
31
- - **Nonparametric IPSS: Fast, flexible feature selection with false discovery control**
31
+ - **Nonparametric IPSS**
32
32
  [*Bioinformatics*](https://doi.org/10.1093/bioinformatics/btaf299) • [arXiv](https://arxiv.org/abs/2410.02208)
33
33
 
34
- > "*Integrated path stability selection*" introduces IPSS and applies it to regularized models such as lasso and adaptive lasso. "*Nonparametric IPSS: Fast, flexible feature selection with false discovery control*" extends IPSS to arbitrary feature importance scores, such as those from gradient boosting and random forests.
35
-
36
34
  ## Installation
37
35
  ```
38
36
  pip install ipss
@@ -51,6 +49,7 @@ ipss_output = ipss(X,y)
51
49
  target_fdr = 0.1
52
50
  q_values = ipss_output['q_values']
53
51
  selected_features = [idx for idx, q_value in q_values.items() if q_value <= target_fdr]
52
+
54
53
  print(f'Selected features (target FDR = {target_fdr}): {selected_features}')
55
54
  ```
56
55
  ### Outputs
@@ -59,7 +58,7 @@ print(f'Selected features (target FDR = {target_fdr}): {selected_features}')
59
58
  - `q_values`: Dictionary whose keys are feature indices and values are their q-values (dict of length `p`).
60
59
  - `runtime`: Runtime of the algorithm in seconds (float).
61
60
  - `selected_features`: Indices of features selected by IPSS; empty list if `target_fp` and `target_fdr` are not specified (list of ints).
62
- - `stability_paths`: Estimated selection probabilities at each parameter value (array of shape `(n_alphas, p)`)
61
+ - `stability_paths`: Estimated selection probabilities at each parameter (array of shape `(n_alphas, p)`).
63
62
 
64
63
  ### Selecting features
65
64
  Each feature (column of `X`) is assigned:
@@ -125,7 +124,7 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
125
124
  - Users can provide their own feature importance function (see example above).
126
125
  - `selector_args`: Arguments for the base algorithm (dict; default `None`).
127
126
  - `preselect`: Preselect/filter features prior to subsampling (bool; default `True`).
128
- - `preselector`: Method used to preselect features (default `None`, which resolves to `'dcor'` for every `selector` except `'l1'`/`'adaptive_lasso'`, which use their own linear preselection; a custom `selector` function defaults to reusing itself as the preselector). Options:
127
+ - `preselector`: Method used to preselect features (default `None`, which resolves to `'dcor'` for every `selector` except `'l1'`/`'adaptive_lasso'`, which use their own linear preselection). Options:
129
128
  - `'gb'`: Gradient boosting (XGBoost) feature importances.
130
129
  - `'rf'`: Random forest (scikit-learn) feature importances.
131
130
  - `'l1'` or `'adaptive_lasso'`: (Adaptive) lasso/logistic regression coefficients.
@@ -146,9 +145,10 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
146
145
  - `'h3'`: Cubic function, ```h3(x) = (2x - 1)**3 if x >= 0.5 else 0```.
147
146
  - `cutoff`: Maximum value of the theoretical integral bound `I(Lambda)` (positive float; default `0.05`).
148
147
  - `delta`: Defines probability measure; see `Associated papers` (float; defaults depend on `selector`).
149
- - `standardize_X`: Scale features to have mean 0, standard deviation 1 (bool; default `None`).
150
- - `center_y`: Center response to have mean 0 (bool; default `None`).
148
+ - `standardize_X`: Scale features to have mean 0, standard deviation 1 (bool; default `None`, which standardizes for `'l1'` and `'adaptive_lasso'` only).
149
+ - `center_y`: Center response to have mean 0 (bool; default `None`, which centers for `'l1'` and `'adaptive_lasso'` only). Binary responses are never centered.
151
150
  - `n_jobs`: Number of jobs to run in parallel (int; default `1`).
151
+ - `random_state`: Seed for preselection, subsampling, and the `'gb'`, `'rf'`, and `'ufi'` selectors (int or `None`; default `None`, which uses NumPy's global random state). Each selector fit gets its own seed; `selector_args['random_state']` is ignored.
152
152
 
153
153
  ### General observations/recommendations:
154
154
  - `selector = 'gb'` often best for capturing nonlinear relationships.
@@ -164,4 +164,3 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
164
164
  - Results are robust to `cutoff` provided it is between `0.025` and `0.1`.
165
165
  - Features are automatically standardized for the penalized regression methods.
166
166
  - The response is automatically centered for the regularized regression methods.
167
-
@@ -2,16 +2,14 @@
2
2
 
3
3
  > Fast, flexible feature selection with false discovery control
4
4
 
5
- ## Associated papers
5
+ ## Papers
6
6
 
7
7
  - **Integrated path stability selection**
8
8
  [*Journal of the American Statistical Association*](https://doi.org/10.1080/01621459.2025.2525589) • [arXiv](https://arxiv.org/abs/2403.15877)
9
9
 
10
- - **Nonparametric IPSS: Fast, flexible feature selection with false discovery control**
10
+ - **Nonparametric IPSS**
11
11
  [*Bioinformatics*](https://doi.org/10.1093/bioinformatics/btaf299) • [arXiv](https://arxiv.org/abs/2410.02208)
12
12
 
13
- > "*Integrated path stability selection*" introduces IPSS and applies it to regularized models such as lasso and adaptive lasso. "*Nonparametric IPSS: Fast, flexible feature selection with false discovery control*" extends IPSS to arbitrary feature importance scores, such as those from gradient boosting and random forests.
14
-
15
13
  ## Installation
16
14
  ```
17
15
  pip install ipss
@@ -30,6 +28,7 @@ ipss_output = ipss(X,y)
30
28
  target_fdr = 0.1
31
29
  q_values = ipss_output['q_values']
32
30
  selected_features = [idx for idx, q_value in q_values.items() if q_value <= target_fdr]
31
+
33
32
  print(f'Selected features (target FDR = {target_fdr}): {selected_features}')
34
33
  ```
35
34
  ### Outputs
@@ -38,7 +37,7 @@ print(f'Selected features (target FDR = {target_fdr}): {selected_features}')
38
37
  - `q_values`: Dictionary whose keys are feature indices and values are their q-values (dict of length `p`).
39
38
  - `runtime`: Runtime of the algorithm in seconds (float).
40
39
  - `selected_features`: Indices of features selected by IPSS; empty list if `target_fp` and `target_fdr` are not specified (list of ints).
41
- - `stability_paths`: Estimated selection probabilities at each parameter value (array of shape `(n_alphas, p)`)
40
+ - `stability_paths`: Estimated selection probabilities at each parameter (array of shape `(n_alphas, p)`).
42
41
 
43
42
  ### Selecting features
44
43
  Each feature (column of `X`) is assigned:
@@ -104,7 +103,7 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
104
103
  - Users can provide their own feature importance function (see example above).
105
104
  - `selector_args`: Arguments for the base algorithm (dict; default `None`).
106
105
  - `preselect`: Preselect/filter features prior to subsampling (bool; default `True`).
107
- - `preselector`: Method used to preselect features (default `None`, which resolves to `'dcor'` for every `selector` except `'l1'`/`'adaptive_lasso'`, which use their own linear preselection; a custom `selector` function defaults to reusing itself as the preselector). Options:
106
+ - `preselector`: Method used to preselect features (default `None`, which resolves to `'dcor'` for every `selector` except `'l1'`/`'adaptive_lasso'`, which use their own linear preselection). Options:
108
107
  - `'gb'`: Gradient boosting (XGBoost) feature importances.
109
108
  - `'rf'`: Random forest (scikit-learn) feature importances.
110
109
  - `'l1'` or `'adaptive_lasso'`: (Adaptive) lasso/logistic regression coefficients.
@@ -125,9 +124,10 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
125
124
  - `'h3'`: Cubic function, ```h3(x) = (2x - 1)**3 if x >= 0.5 else 0```.
126
125
  - `cutoff`: Maximum value of the theoretical integral bound `I(Lambda)` (positive float; default `0.05`).
127
126
  - `delta`: Defines probability measure; see `Associated papers` (float; defaults depend on `selector`).
128
- - `standardize_X`: Scale features to have mean 0, standard deviation 1 (bool; default `None`).
129
- - `center_y`: Center response to have mean 0 (bool; default `None`).
127
+ - `standardize_X`: Scale features to have mean 0, standard deviation 1 (bool; default `None`, which standardizes for `'l1'` and `'adaptive_lasso'` only).
128
+ - `center_y`: Center response to have mean 0 (bool; default `None`, which centers for `'l1'` and `'adaptive_lasso'` only). Binary responses are never centered.
130
129
  - `n_jobs`: Number of jobs to run in parallel (int; default `1`).
130
+ - `random_state`: Seed for preselection, subsampling, and the `'gb'`, `'rf'`, and `'ufi'` selectors (int or `None`; default `None`, which uses NumPy's global random state). Each selector fit gets its own seed; `selector_args['random_state']` is ignored.
131
131
 
132
132
  ### General observations/recommendations:
133
133
  - `selector = 'gb'` often best for capturing nonlinear relationships.
@@ -143,4 +143,3 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
143
143
  - Results are robust to `cutoff` provided it is between `0.025` and `0.1`.
144
144
  - Features are automatically standardized for the penalized regression methods.
145
145
  - The response is automatically centered for the regularized regression methods.
146
-
@@ -5,6 +5,7 @@ Update (version 1.1.7): SCAD and MCP temporarily disabled due to skglm dependenc
5
5
  """
6
6
 
7
7
  from .linear import (
8
+ adaptive_weights,
8
9
  fit_l1_regressor,
9
10
  fit_l1_classifier,
10
11
  fit_adaptive_lasso_classifier,
@@ -28,6 +29,7 @@ from .ufi import (
28
29
 
29
30
  __all__ = [
30
31
  # linear
32
+ "adaptive_weights",
31
33
  "fit_l1_regressor",
32
34
  "fit_l1_classifier",
33
35
  "fit_adaptive_lasso_classifier",
@@ -5,6 +5,7 @@ import xgboost as xgb
5
5
 
6
6
  # gradient boosting classifier
7
7
  def fit_gb_classifier(X, y, **kwargs):
8
+ kwargs.setdefault('random_state', np.random.randint(np.iinfo(np.int32).max))
8
9
  model = xgb.XGBClassifier(**kwargs)
9
10
  model.fit(X,y)
10
11
  feature_importances = model.feature_importances_
@@ -12,7 +13,8 @@ def fit_gb_classifier(X, y, **kwargs):
12
13
 
13
14
  # gradient boosting regressor
14
15
  def fit_gb_regressor(X, y, **kwargs):
16
+ kwargs.setdefault('random_state', np.random.randint(np.iinfo(np.int32).max))
15
17
  model = xgb.XGBRegressor(**kwargs)
16
18
  model.fit(X,y)
17
19
  feature_importances = model.feature_importances_
18
- return feature_importances
20
+ return feature_importances
@@ -10,23 +10,29 @@ import numpy as np
10
10
  # from skglm.penalties import SCAD
11
11
  from sklearn.linear_model import Lasso, lasso_path, LogisticRegression, Ridge, RidgeClassifier
12
12
 
13
+ # adaptive lasso weights from an initial l2-penalized fit (logistic regression if binary, ridge otherwise)
14
+ def adaptive_weights(X, y, binary_response, epsilon=1e-6):
15
+ with warnings.catch_warnings():
16
+ warnings.simplefilter('ignore')
17
+ if binary_response:
18
+ init_model = LogisticRegression(penalty='l2', solver='liblinear', max_iter=1000)
19
+ else:
20
+ init_model = Ridge()
21
+ init_model.fit(X, y)
22
+ init_coef = init_model.coef_.flatten()
23
+ return 1 / (np.abs(init_coef) + epsilon)
24
+
13
25
  # adaptive lasso classifier
14
26
  def fit_adaptive_lasso_classifier(X, y, alphas, epsilon=1e-6):
15
27
  n_alphas = len(alphas)
16
28
  n_features = X.shape[1]
17
29
  coefficients = np.zeros((n_alphas, n_features))
18
30
 
19
- # Initial logistic regression with L2 penalty
20
- with warnings.catch_warnings():
21
- warnings.simplefilter('ignore')
22
- init_model = LogisticRegression(penalty='l2', solver='liblinear', max_iter=1000)
23
- init_model.fit(X, y)
24
- init_coef = init_model.coef_.flatten()
25
-
26
- weights = 1 / (np.abs(init_coef) + epsilon)
31
+ # compute adaptive weights and reweight features
32
+ weights = adaptive_weights(X, y, binary_response=True, epsilon=epsilon)
33
+ X_weighted = X / weights[np.newaxis, :]
27
34
 
28
35
  for i, alpha in enumerate(alphas):
29
- X_weighted = X / weights[np.newaxis, :]
30
36
  model = LogisticRegression(penalty='l1', C=1/alpha, solver='liblinear', max_iter=1000)
31
37
  with warnings.catch_warnings():
32
38
  warnings.simplefilter('ignore')
@@ -41,19 +47,12 @@ def fit_adaptive_lasso_regressor(X, y, alphas, epsilon=1e-6):
41
47
  n_features = X.shape[1]
42
48
  coefficients = np.zeros((n_alphas, n_features))
43
49
 
44
- # get initial coefficients from ridge regression
45
- with warnings.catch_warnings():
46
- warnings.simplefilter('ignore')
47
- init_ridge = Ridge()
48
- init_ridge.fit(X,y)
49
- init_coef = init_ridge.coef_
50
-
51
- # compute adaptive weights
52
- weights = 1 / (np.abs(init_coef) + epsilon)
50
+ # compute adaptive weights and reweight features
51
+ weights = adaptive_weights(X, y, binary_response=False, epsilon=epsilon)
52
+ X_weighted = X / weights[np.newaxis, :]
53
53
 
54
54
  # solve weighted lasso for each alpha
55
55
  for i, alpha in enumerate(alphas):
56
- X_weighted = X / weights[np.newaxis, :]
57
56
  lasso = Lasso(alpha=alpha)
58
57
  with warnings.catch_warnings():
59
58
  warnings.simplefilter('ignore')
@@ -4,6 +4,7 @@ import numpy as np
4
4
  from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
5
5
  from sklearn.inspection import permutation_importance
6
6
  from sklearn.model_selection import train_test_split
7
+ from sklearn.utils import check_random_state
7
8
 
8
9
  # random forest classifier
9
10
  def fit_rf_classifier(X, y, **kwargs):
@@ -13,8 +14,9 @@ def fit_rf_classifier(X, y, **kwargs):
13
14
  if importance_type == 'mdi':
14
15
  feature_importances = model.feature_importances_
15
16
  elif importance_type == 'permutation':
16
- X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)
17
- perm_importance = permutation_importance(model, X_test, y_test, n_repeats=1)
17
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=kwargs.get('random_state'))
18
+ model.fit(X_train, y_train)
19
+ perm_importance = permutation_importance(model, X_test, y_test, n_repeats=1, random_state=kwargs.get('random_state'))
18
20
  feature_importances = perm_importance.importances_mean
19
21
  else:
20
22
  raise ValueError("importance_type must be either 'mdi' or 'permutation'")
@@ -28,14 +30,16 @@ def fit_rf_regressor(X, y, **kwargs):
28
30
  if importance_type == 'mdi':
29
31
  feature_importances = model.feature_importances_
30
32
  elif importance_type == 'permutation':
31
- X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
32
- perm_importance = permutation_importance(model, X_test, y_test, n_repeats=1)
33
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=kwargs.get('random_state'))
34
+ model.fit(X_train, y_train)
35
+ perm_importance = permutation_importance(model, X_test, y_test, n_repeats=1, random_state=kwargs.get('random_state'))
33
36
  feature_importances = perm_importance.importances_mean
34
37
  elif importance_type == 'shadow':
35
38
  n, p = X.shape
36
39
  X_shadow = X.copy()
40
+ rng = check_random_state(kwargs.get('random_state'))
37
41
  for i in range(p):
38
- np.random.shuffle(X_shadow[:,i])
42
+ rng.shuffle(X_shadow[:,i])
39
43
  X_combined = np.hstack((X, X_shadow))
40
44
  model.fit(X_combined, y)
41
45
  importances_combined = model.feature_importances_
@@ -58,13 +58,12 @@ def resolve_selector(selector, binary_response):
58
58
  return selector
59
59
 
60
60
  def resolve_preselector(preselector, selector, binary_response):
61
- # default: dcor for every selector except (adaptive) lasso, which keeps its own linear
62
- # preselection; custom (callable) selectors default to reusing themselves as the preselector
61
+ # default: dcor for every selector except (adaptive) lasso, which keeps its own linear preselection
63
62
  if preselector is None:
64
63
  regularization_selectors = ('lasso', 'logistic_regression', 'adaptive_lasso_classifier', 'adaptive_lasso_regressor')
65
- if isinstance(selector, str) and selector not in regularization_selectors:
66
- return 'dcor'
67
- return selector
64
+ if selector in regularization_selectors:
65
+ return selector
66
+ return 'dcor'
68
67
  if not isinstance(preselector, str):
69
68
  return preselector
70
69
  if preselector == 'adaptive_lasso':
@@ -80,11 +79,7 @@ def resolve_preselector(preselector, selector, binary_response):
80
79
  return preselector
81
80
 
82
81
  def check_response_type(y, selector):
83
- unique_values = np.unique(y)
84
- if len(unique_values) == 1:
85
- print(f"Error: The response variable `y` has only one unique value: {unique_values[0]}.")
86
- return None, None
87
- binary_response = len(unique_values) == 2
82
+ binary_response = len(np.unique(y)) == 2
88
83
  return binary_response, resolve_selector(selector, binary_response)
89
84
 
90
85
  def compute_alphas(X, y, n_alphas, max_features, binary_response=False):
@@ -94,8 +89,9 @@ def compute_alphas(X, y, n_alphas, max_features, binary_response=False):
94
89
  scaled_residuals = y - y_mean * (1 - y_mean)
95
90
  alpha_max = 5 / np.max(np.abs(np.dot(X.T, scaled_residuals) / n))
96
91
  selector = LogisticRegression(penalty='l1', solver='liblinear', tol=1e-3, warm_start=True, class_weight='balanced')
97
- if np.isnan(alpha_max):
92
+ if not np.isfinite(alpha_max):
98
93
  alpha_max = 100
94
+ alpha_max = compute_alpha_max_logistic(selector, X, y, alpha_max)
99
95
  alpha_min = alpha_max * 1e-10
100
96
  test_alphas = np.logspace(np.log10(alpha_max/2), np.log10(alpha_min), 100)
101
97
  else:
@@ -120,6 +116,44 @@ def compute_alphas(X, y, n_alphas, max_features, binary_response=False):
120
116
  alphas = np.logspace(np.log10(alpha_max), np.log10(alpha_min), n_alphas)
121
117
  return alphas
122
118
 
119
+ # smallest alpha (to within a factor of tol) at which l1 logistic regression selects no features
120
+ def compute_alpha_max_logistic(selector, X, y, alpha_start, tol=1.1, max_steps=50):
121
+ def n_selected(alpha):
122
+ selector.set_params(C=1/alpha)
123
+ with warnings.catch_warnings():
124
+ warnings.simplefilter('ignore')
125
+ selector.fit(X, y)
126
+ return np.sum(selector.coef_ != 0)
127
+
128
+ # bracket the threshold: features are selected at lower, none at upper
129
+ if n_selected(alpha_start) > 0:
130
+ upper = alpha_start
131
+ for _ in range(max_steps):
132
+ upper *= 2
133
+ if n_selected(upper) == 0:
134
+ break
135
+ else:
136
+ return alpha_start
137
+ lower = upper / 2
138
+ else:
139
+ lower = alpha_start
140
+ for _ in range(max_steps):
141
+ lower /= 2
142
+ if n_selected(lower) > 0:
143
+ break
144
+ else:
145
+ return alpha_start
146
+ upper = lower * 2
147
+
148
+ # bisect on a log scale
149
+ while upper / lower > tol:
150
+ mid = np.sqrt(lower * upper)
151
+ if n_selected(mid) > 0:
152
+ lower = mid
153
+ else:
154
+ upper = mid
155
+ return upper
156
+
123
157
  def compute_correlation(X):
124
158
  corr_matrix = np.corrcoef(X, rowvar=False)
125
159
  abs_corr_matrix = np.abs(corr_matrix)
@@ -250,4 +284,3 @@ def selector_and_args(selector, selector_args):
250
284
  selector_function = selector
251
285
  return selector_function, selector_args
252
286
 
253
-
@@ -1,13 +1,21 @@
1
1
  # Integrated path stability selection
2
2
 
3
3
  import time
4
+ import warnings
4
5
 
5
6
  from joblib import Parallel, delayed
6
7
  import numpy as np
8
+ from sklearn.utils import check_random_state
7
9
 
10
+ from .base_selectors import (fit_gb_classifier, fit_gb_regressor, fit_rf_classifier, fit_rf_regressor,
11
+ fit_ufi_classifier, fit_ufi_regressor)
8
12
  from .helpers import return_null_result
9
13
  from .processing import postprocess_ipss, preprocess_ipss
10
14
 
15
+ # built-in selectors that take a random_state; each of their fits gets its own seed
16
+ seeded_selectors = (fit_gb_classifier, fit_gb_regressor, fit_rf_classifier, fit_rf_regressor,
17
+ fit_ufi_classifier, fit_ufi_regressor)
18
+
11
19
  #----------------------------------------------------------------
12
20
  # IPSS
13
21
  #----------------------------------------------------------------
@@ -20,7 +28,7 @@ Inputs:
20
28
 
21
29
  Optional
22
30
  ----------------
23
- selector: gradient boosting ('gb'), l1 regularization ('l1'), random forest ('rf''), or a custom selector
31
+ selector: gradient boosting ('gb'), l1 regularization ('l1'), random forest ('rf'), or a custom selector
24
32
  selector_args: arguments for selector
25
33
  target_fp: target number of false positives
26
34
  target_fdr: target false discovery rate
@@ -35,10 +43,12 @@ Inputs:
35
43
  cutoff: max value of theoretical integral bound I(Lambda)
36
44
  delta: determines probability measure mu_delta(dlambda) = z_delta^{-1}lambda^{-delta}dlambda
37
45
  subsample_size: size of the subsampled datasets
38
- standardize_X: whether to standardize features to have mean 0, standard deviation 1
39
- center_y: whether to center the response to have mean 0
46
+ standardize_X: whether to standardize features to have mean 0, standard deviation 1 (None: only for 'l1' and 'adaptive_lasso')
47
+ center_y: whether to center the response to have mean 0 (None: only for 'l1' and 'adaptive_lasso'; binary responses are never centered)
40
48
  n_jobs: number of jobs to run in parallel
41
49
  force_regression: force ipss to use regression (if False, classification is used if response is binary)
50
+ random_state: seed (int, RandomState, or None) for preselection, subsampling, and the 'gb', 'rf', and 'ufi'
51
+ selectors, each of whose fits gets its own seed. None uses numpy's global random state
42
52
 
43
53
  Outputs:
44
54
  efp_scores: efp (expected false positive) score for each feature
@@ -50,18 +60,20 @@ Outputs:
50
60
  def ipss(X, y, selector='gb', selector_args=None, preselect=True, preselector=None, preselector_args=None,
51
61
  B=None, n_alphas=None, ipss_function=None, cutoff=0.05, delta=None, subsample_size=None,
52
62
  target_fp=None, target_fdr=None, standardize_X=None, center_y=None, n_jobs=1,
53
- force_regression=False, _return_details=False):
63
+ force_regression=False, random_state=None, _return_details=False):
54
64
 
55
65
  # start timer
56
66
  start = time.time()
57
67
 
68
+ rng = check_random_state(random_state)
69
+
58
70
  # prepare ipss arguments and data prior to estimating selection probabilities
59
71
  prep = preprocess_ipss(X, y, selector, selector_args, preselect, preselector, preselector_args,
60
- B, n_alphas, ipss_function, delta, standardize_X, center_y, force_regression)
72
+ B, n_alphas, ipss_function, delta, standardize_X, center_y, force_regression, rng)
61
73
 
62
74
  if prep is None:
63
75
  runtime = time.time() - start
64
- return return_null_result(X.shape[1], runtime)
76
+ return return_null_result(np.shape(X)[1], runtime)
65
77
 
66
78
  X, y = prep['X'], prep['y']
67
79
  selector_function, selector_args = prep['selector_function'], prep['selector_args']
@@ -69,15 +81,22 @@ def ipss(X, y, selector='gb', selector_args=None, preselect=True, preselector=No
69
81
  p_full, preselect_indices = prep['p_full'], prep['preselect_indices']
70
82
  ipss_function = prep['ipss_function']
71
83
 
84
+ if selector_function in seeded_selectors and 'random_state' in selector_args:
85
+ warnings.warn("selector_args['random_state'] is ignored; use random_state instead.", UserWarning)
86
+ selector_args = {key: value for key, value in selector_args.items() if key != 'random_state'}
87
+
72
88
  n, p = X.shape
73
89
 
74
90
  # subsample size
75
91
  if subsample_size is None or subsample_size > n // 2:
76
92
  subsample_size = n // 2
77
93
 
94
+ # one seed per subsampling step
95
+ seeds = rng.randint(np.iinfo(np.int32).max, size=B)
96
+
78
97
  # estimate selection probabilities
79
98
  results = np.array(Parallel(n_jobs=n_jobs)(delayed(selection)(X, y, alphas, subsample_size,
80
- selector_function, **selector_args) for _ in range(B)))
99
+ selector_function, seeds[b], **selector_args) for b in range(B)))
81
100
 
82
101
  # process the estimated selection probabilities and compute final results (efp scores, q-values, etc.)
83
102
  ipss_output = postprocess_ipss(results, alphas, n_alphas, p, p_full, preselect_indices, B, cutoff, delta,
@@ -89,25 +108,24 @@ def ipss(X, y, selector='gb', selector_args=None, preselect=True, preselector=No
89
108
 
90
109
  return ipss_output
91
110
 
111
+
92
112
  # subsampler for estimating selection probabilities
93
- def selection(X, y, alphas, subsample_size, selector_function, **kwargs):
113
+ def selection(X, y, alphas, subsample_size, selector_function, seed=None, **kwargs):
94
114
  n, p = X.shape
95
- indices = np.arange(n)
96
- np.random.shuffle(indices)
115
+ rng = check_random_state(seed)
116
+ indices = rng.permutation(n)
97
117
 
98
118
  # take two disjoint subsamples
99
- idx1 = indices[:subsample_size]
100
- idx2 = indices[subsample_size:2*subsample_size]
101
-
102
- if alphas is None:
103
- indicators = np.empty((2, p))
104
- indicators[0, :] = np.array(selector_function(X[idx1, :], y[idx1], **kwargs))
105
- indicators[1, :] = np.array(selector_function(X[idx2, :], y[idx2], **kwargs))
106
-
107
- else:
108
- indicators = np.empty((len(alphas), 2, p))
109
- indicators[:, 0, :] = np.array(selector_function(X[idx1, :], y[idx1], alphas, **kwargs))
110
- indicators[:, 1, :] = np.array(selector_function(X[idx2, :], y[idx2], alphas, **kwargs))
119
+ halves = [indices[:subsample_size], indices[subsample_size:2*subsample_size]]
120
+ alpha_args = () if alphas is None else (alphas,)
121
+
122
+ # built-in tree selectors get a new seed for each fit unless one is given
123
+ seed_fits = selector_function in seeded_selectors and 'random_state' not in kwargs
124
+
125
+ indicators = np.empty((2, p)) if alphas is None else np.empty((len(alphas), 2, p))
126
+ for i, idx in enumerate(halves):
127
+ fit_args = {**kwargs, 'random_state': rng.randint(np.iinfo(np.int32).max)} if seed_fits else kwargs
128
+ indicators[..., i, :] = selector_function(X[idx, :], y[idx], *alpha_args, **fit_args)
111
129
 
112
130
  return indicators
113
131
 
@@ -5,6 +5,7 @@ import warnings
5
5
  import numpy as np
6
6
  from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
7
7
  from sklearn.linear_model import Lasso, LogisticRegression
8
+ from sklearn.utils import check_random_state
8
9
  import xgboost as xgb
9
10
 
10
11
  # squared distance correlation between response and each feature
@@ -27,8 +28,9 @@ def compute_dcor_scores(X, y):
27
28
  scores[j] = dcov_sqr / np.sqrt(denom) if denom > 0 else 0.0
28
29
  return scores
29
30
 
30
- def preselection(X, y, preselector, preselector_args=None):
31
+ def preselection(X, y, preselector, preselector_args=None, random_state=None):
31
32
  n, p = X.shape
33
+ rng = check_random_state(random_state)
32
34
 
33
35
  if preselector_args is None:
34
36
  preselector_args = {}
@@ -56,14 +58,14 @@ def preselection(X, y, preselector, preselector_args=None):
56
58
  preselector_args_local.setdefault('C', 1 / alpha)
57
59
  model = LogisticRegression(**preselector_args_local)
58
60
  else:
59
- preselector_args_local.setdefault('alpha', alpha)
61
+ preselector_args_local.setdefault('alpha', alpha * np.std(y))
60
62
  model = Lasso(**preselector_args_local)
61
63
 
62
64
  feature_importances = np.zeros(p)
63
65
  with warnings.catch_warnings():
64
66
  warnings.simplefilter('ignore')
65
67
  for _ in range(n_runs):
66
- indices = np.random.choice(n, size=n, replace=True)
68
+ indices = rng.choice(n, size=n, replace=True)
67
69
  X_sampled, y_sampled = X[indices], y[indices]
68
70
  model.fit(X_sampled, y_sampled)
69
71
  feature_importances += np.abs(model.coef_).ravel()
@@ -77,7 +79,7 @@ def preselection(X, y, preselector, preselector_args=None):
77
79
  model = model_class(**preselector_args_local)
78
80
  feature_importances = np.zeros(p)
79
81
  for _ in range(n_runs):
80
- model.set_params(random_state=np.random.randint(1e5))
82
+ model.set_params(random_state=rng.randint(1e5))
81
83
  model.fit(X,y)
82
84
  feature_importances += model.feature_importances_
83
85
  preselect_indices = np.argsort(feature_importances)[::-1][:n_keep]
@@ -91,7 +93,7 @@ def preselection(X, y, preselector, preselector_args=None):
91
93
  model = model_class(**preselector_args_local)
92
94
  feature_importances = np.zeros(p)
93
95
  for i in range(n_runs):
94
- model.set_params(random_state=np.random.randint(1e5))
96
+ model.set_params(random_state=rng.randint(1e5))
95
97
  model.fit(X, y)
96
98
  feature_importances += model.feature_importances_
97
99
  nonzero_mask = feature_importances > 0
@@ -104,7 +106,7 @@ def preselection(X, y, preselector, preselector_args=None):
104
106
  else:
105
107
  n_extra = n_keep - len(nonzero_indices)
106
108
  zero_indices = np.where(~nonzero_mask)[0]
107
- extra_indices = np.random.choice(zero_indices, size=n_extra, replace=False)
109
+ extra_indices = rng.choice(zero_indices, size=n_extra, replace=False)
108
110
  preselect_indices = np.concatenate([nonzero_indices, extra_indices])
109
111
 
110
112
  elif preselector == 'dcor':
@@ -3,15 +3,23 @@
3
3
  import warnings
4
4
 
5
5
  import numpy as np
6
+ from scipy.sparse import issparse
6
7
  from sklearn.preprocessing import StandardScaler
7
8
 
9
+ from .base_selectors import adaptive_weights
8
10
  from .helpers import (check_response_type, compute_alphas, compute_delta, compute_qvalues, integrate,
9
11
  resolve_preselector, resolve_selector, score_based_selection, selector_and_args)
10
12
  from .preselection import preselection
11
13
 
12
14
  # prepare ipss arguments and data
13
15
  def preprocess_ipss(X, y, selector, selector_args, preselect, preselector, preselector_args,
14
- B, n_alphas, ipss_function, delta, standardize_X, center_y, force_regression=False):
16
+ B, n_alphas, ipss_function, delta, standardize_X, center_y, force_regression=False, random_state=None):
17
+
18
+ # cast to float64 (float32 alpha grids underflow/overflow in integrate)
19
+ X = X.astype(np.float64) if issparse(X) else np.asarray(X, dtype=np.float64)
20
+ y = np.asarray(y)
21
+ if y.dtype.kind in 'biuf':
22
+ y = y.astype(np.float64)
15
23
 
16
24
  # specify whether base estimator is a regularization or variable importance method
17
25
  selector_type = 'importance'
@@ -21,15 +29,6 @@ def preprocess_ipss(X, y, selector, selector_args, preselect, preselector, prese
21
29
  # empty set for selector args if none specified
22
30
  selector_args = selector_args or {}
23
31
 
24
- # if no preselector is specified and the selector is a custom callable, the preselector
25
- # defaults to reusing that callable, so it shares selector_args unless overridden
26
- if preselector is None and not isinstance(selector, str):
27
- if preselector_args is None:
28
- preselector_args = {}
29
- for key in selector_args:
30
- if key not in preselector_args:
31
- preselector_args[key] = selector_args[key]
32
-
33
32
  # number of subsamples
34
33
  B = B if B is not None else 100 if selector == 'gb' else 50
35
34
 
@@ -38,12 +37,18 @@ def preprocess_ipss(X, y, selector, selector_args, preselect, preselector, prese
38
37
  y = y.ravel()
39
38
 
40
39
  # check response type
40
+ if np.unique(y).size == 1:
41
+ raise ValueError(f'The response y has only one unique value: {y[0]}.')
41
42
  if force_regression:
42
43
  binary_response = False
43
44
  selector = resolve_selector(selector, binary_response)
44
45
  else:
45
46
  binary_response, selector = check_response_type(y, selector)
46
47
 
48
+ # code binary responses as 0/1 (required by XGBoost)
49
+ if binary_response:
50
+ y = (y == np.unique(y)[1]).astype(np.float64)
51
+
47
52
  # ipss function
48
53
  if ipss_function is None:
49
54
  ipss_function = 'h2' if selector in ['lasso', 'logistic_regression'] else 'h3'
@@ -52,19 +57,21 @@ def preprocess_ipss(X, y, selector, selector_args, preselect, preselector, prese
52
57
  if delta is None:
53
58
  delta = compute_delta(X, selector)
54
59
 
55
- # standardize and center data if using l1 selectors
56
- if selector_type == 'regularization':
57
- if standardize_X is None:
58
- X = StandardScaler().fit_transform(X)
59
- if center_y is None:
60
- if not binary_response:
61
- y -= np.mean(y)
60
+ # standardize and center data (by default only for l1 selectors); never center a binary response
61
+ if standardize_X is None:
62
+ standardize_X = selector_type == 'regularization'
63
+ if center_y is None:
64
+ center_y = selector_type == 'regularization'
65
+ if standardize_X:
66
+ X = StandardScaler().fit_transform(X)
67
+ if center_y and not binary_response:
68
+ y = y - np.mean(y)
62
69
 
63
70
  # preselect features to reduce dimension
64
71
  p_full = X.shape[1]
65
72
  if preselect:
66
73
  preselector = resolve_preselector(preselector, selector, binary_response)
67
- X, preselect_indices = preselection(X, y, preselector, preselector_args)
74
+ X, preselect_indices = preselection(X, y, preselector, preselector_args, random_state)
68
75
  if preselect_indices.size == 0:
69
76
  warnings.warn('Preselection step removed all features. Returning null result.', UserWarning)
70
77
  return None
@@ -80,7 +87,14 @@ def preprocess_ipss(X, y, selector, selector_args, preselect, preselector, prese
80
87
  # alphas
81
88
  if n_alphas is None:
82
89
  n_alphas = 25 if selector_type == 'regularization' else 100
83
- alphas = compute_alphas(X, y, n_alphas, max_features, binary_response) if selector_type == 'regularization' else None
90
+ alphas = None
91
+ if selector_type == 'regularization':
92
+ # adaptive lasso fits a lasso on X / weights, so calibrate the grid on that reweighted design
93
+ X_alphas = X
94
+ if selector in ['adaptive_lasso_classifier', 'adaptive_lasso_regressor']:
95
+ epsilon = selector_args.get('epsilon', 1e-6)
96
+ X_alphas = X / adaptive_weights(X, y, binary_response, epsilon)
97
+ alphas = compute_alphas(X_alphas, y, n_alphas, max_features, binary_response)
84
98
 
85
99
  # selector function and args
86
100
  selector_function, selector_args = selector_and_args(selector, selector_args)
@@ -216,4 +230,3 @@ def ipss_scores(stability_paths, B, alphas, average_selected, ipss_function, del
216
230
 
217
231
  return scores, integral, alphas, stop_index
218
232
 
219
-
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: ipss
3
- Version: 1.1.12
3
+ Version: 1.2.0
4
4
  Summary: Python implementation of integrated path stability selection (IPSS)
5
5
  Author-email: Omar Melikechi <omar.melikechi@gmail.com>
6
6
  License-Expression: MIT
@@ -23,16 +23,14 @@ Dynamic: license-file
23
23
 
24
24
  > Fast, flexible feature selection with false discovery control
25
25
 
26
- ## Associated papers
26
+ ## Papers
27
27
 
28
28
  - **Integrated path stability selection**
29
29
  [*Journal of the American Statistical Association*](https://doi.org/10.1080/01621459.2025.2525589) • [arXiv](https://arxiv.org/abs/2403.15877)
30
30
 
31
- - **Nonparametric IPSS: Fast, flexible feature selection with false discovery control**
31
+ - **Nonparametric IPSS**
32
32
  [*Bioinformatics*](https://doi.org/10.1093/bioinformatics/btaf299) • [arXiv](https://arxiv.org/abs/2410.02208)
33
33
 
34
- > "*Integrated path stability selection*" introduces IPSS and applies it to regularized models such as lasso and adaptive lasso. "*Nonparametric IPSS: Fast, flexible feature selection with false discovery control*" extends IPSS to arbitrary feature importance scores, such as those from gradient boosting and random forests.
35
-
36
34
  ## Installation
37
35
  ```
38
36
  pip install ipss
@@ -51,6 +49,7 @@ ipss_output = ipss(X,y)
51
49
  target_fdr = 0.1
52
50
  q_values = ipss_output['q_values']
53
51
  selected_features = [idx for idx, q_value in q_values.items() if q_value <= target_fdr]
52
+
54
53
  print(f'Selected features (target FDR = {target_fdr}): {selected_features}')
55
54
  ```
56
55
  ### Outputs
@@ -59,7 +58,7 @@ print(f'Selected features (target FDR = {target_fdr}): {selected_features}')
59
58
  - `q_values`: Dictionary whose keys are feature indices and values are their q-values (dict of length `p`).
60
59
  - `runtime`: Runtime of the algorithm in seconds (float).
61
60
  - `selected_features`: Indices of features selected by IPSS; empty list if `target_fp` and `target_fdr` are not specified (list of ints).
62
- - `stability_paths`: Estimated selection probabilities at each parameter value (array of shape `(n_alphas, p)`)
61
+ - `stability_paths`: Estimated selection probabilities at each parameter (array of shape `(n_alphas, p)`).
63
62
 
64
63
  ### Selecting features
65
64
  Each feature (column of `X`) is assigned:
@@ -125,7 +124,7 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
125
124
  - Users can provide their own feature importance function (see example above).
126
125
  - `selector_args`: Arguments for the base algorithm (dict; default `None`).
127
126
  - `preselect`: Preselect/filter features prior to subsampling (bool; default `True`).
128
- - `preselector`: Method used to preselect features (default `None`, which resolves to `'dcor'` for every `selector` except `'l1'`/`'adaptive_lasso'`, which use their own linear preselection; a custom `selector` function defaults to reusing itself as the preselector). Options:
127
+ - `preselector`: Method used to preselect features (default `None`, which resolves to `'dcor'` for every `selector` except `'l1'`/`'adaptive_lasso'`, which use their own linear preselection). Options:
129
128
  - `'gb'`: Gradient boosting (XGBoost) feature importances.
130
129
  - `'rf'`: Random forest (scikit-learn) feature importances.
131
130
  - `'l1'` or `'adaptive_lasso'`: (Adaptive) lasso/logistic regression coefficients.
@@ -146,9 +145,10 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
146
145
  - `'h3'`: Cubic function, ```h3(x) = (2x - 1)**3 if x >= 0.5 else 0```.
147
146
  - `cutoff`: Maximum value of the theoretical integral bound `I(Lambda)` (positive float; default `0.05`).
148
147
  - `delta`: Defines probability measure; see `Associated papers` (float; defaults depend on `selector`).
149
- - `standardize_X`: Scale features to have mean 0, standard deviation 1 (bool; default `None`).
150
- - `center_y`: Center response to have mean 0 (bool; default `None`).
148
+ - `standardize_X`: Scale features to have mean 0, standard deviation 1 (bool; default `None`, which standardizes for `'l1'` and `'adaptive_lasso'` only).
149
+ - `center_y`: Center response to have mean 0 (bool; default `None`, which centers for `'l1'` and `'adaptive_lasso'` only). Binary responses are never centered.
151
150
  - `n_jobs`: Number of jobs to run in parallel (int; default `1`).
151
+ - `random_state`: Seed for preselection, subsampling, and the `'gb'`, `'rf'`, and `'ufi'` selectors (int or `None`; default `None`, which uses NumPy's global random state). Each selector fit gets its own seed; `selector_args['random_state']` is ignored.
152
152
 
153
153
  ### General observations/recommendations:
154
154
  - `selector = 'gb'` often best for capturing nonlinear relationships.
@@ -164,4 +164,3 @@ The [examples](https://github.com/omelikechi/ipss/tree/main/examples) folder inc
164
164
  - Results are robust to `cutoff` provided it is between `0.025` and `0.1`.
165
165
  - Features are automatically standardized for the penalized regression methods.
166
166
  - The response is automatically centered for the regularized regression methods.
167
-
@@ -1,5 +1,4 @@
1
1
  LICENSE
2
- MANIFEST.in
3
2
  README.md
4
3
  pyproject.toml
5
4
  ipss/__init__.py
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "ipss"
7
- version = "1.1.12"
7
+ version = "1.2.0"
8
8
  description = "Python implementation of integrated path stability selection (IPSS)"
9
9
  readme = "README.md"
10
10
  license = "MIT"
ipss-1.1.12/MANIFEST.in DELETED
@@ -1 +0,0 @@
1
-
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes