linearboost 0.2.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {linearboost-0.2.0 → linearboost-0.2.1}/.github/workflows/ci.yml +1 -1
- {linearboost-0.2.0 → linearboost-0.2.1}/PKG-INFO +15 -162
- {linearboost-0.2.0 → linearboost-0.2.1}/README.md +11 -159
- linearboost-0.2.1/notebooks/optuna_sefrboost_demo.ipynb +186 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/pyproject.toml +3 -2
- {linearboost-0.2.0 → linearboost-0.2.1}/requirements.txt +2 -1
- {linearboost-0.2.0 → linearboost-0.2.1}/src/linearboost/__init__.py +1 -1
- {linearboost-0.2.0 → linearboost-0.2.1}/src/linearboost/linear_boost.py +7 -4
- {linearboost-0.2.0 → linearboost-0.2.1}/src/linearboost/sefr_boost.py +10 -6
- {linearboost-0.2.0 → linearboost-0.2.1}/uv.lock +56 -20
- linearboost-0.2.0/notebooks/demo_sefrboost_usage.ipynb +0 -208
- {linearboost-0.2.0 → linearboost-0.2.1}/.github/social-preview.png +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/.gitignore +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/.pre-commit-config.yaml +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/LICENSE +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/SECURITY.md +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/notebooks/demo_linearboost_usage.ipynb +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/src/linearboost/_utils.py +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/src/linearboost/licenses/BSD-3-Clause.txt +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/src/linearboost/sefr.py +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/tests/__init__.py +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/tests/_utils.py +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/tests/test_linearboost.py +0 -0
- {linearboost-0.2.0 → linearboost-0.2.1}/tests/test_sefr.py +0 -0
|
@@ -1,7 +1,7 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
2
|
Name: linearboost
|
|
3
|
-
Version: 0.2.
|
|
4
|
-
Summary: LinearBoost
|
|
3
|
+
Version: 0.2.1
|
|
4
|
+
Summary: LinearBoost: a boosting classifier built on the linear classifier SEFR.
|
|
5
5
|
Project-URL: Homepage, https://github.com/LinearBoost/linearboost-classifier
|
|
6
6
|
Project-URL: Documentation, https://linearboost.readthedocs.io
|
|
7
7
|
Project-URL: Repository, https://github.com/LinearBoost/linearboost-classifier
|
|
@@ -20,20 +20,23 @@ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
|
20
20
|
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
21
21
|
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
22
22
|
Requires-Python: <3.14,>=3.10
|
|
23
|
+
Requires-Dist: packaging>=20.0
|
|
23
24
|
Requires-Dist: scikit-learn>=1.2.2
|
|
24
25
|
Requires-Dist: typing-extensions>=4.1.0; python_full_version < '3.11'
|
|
25
26
|
Description-Content-Type: text/markdown
|
|
26
27
|
|
|
27
|
-
# LinearBoost
|
|
28
|
+
# LinearBoost and SEFRBoost Classifiers
|
|
28
29
|
|
|
29
30
|

|
|
30
31
|
[](https://pypi.org/project/linearboost/)
|
|
31
32
|

|
|
32
33
|
[](https://pepy.tech/projects/linearboost)
|
|
33
34
|
|
|
34
|
-
**Current release: v0.2.0.** This version adds **SEFRBoost**—gradient boosting with linear SEFR splits at tree nodes—via `SEFRBoostClassifier` and `SEFRBoostRegressor` (`from linearboost import …` or `linearboost.sefr_boost`). **LinearBoost security issues have been updated** in this release; upgrade from earlier versions to stay patched.
|
|
35
|
+
**Current release: v0.2.0.** This version adds **SEFRBoost**—gradient boosting with linear SEFR splits at tree nodes—via `SEFRBoostClassifier` and `SEFRBoostRegressor` (`from linearboost import …` or `linearboost.sefr_boost`). It is pronounced /sɛfr-buːst/. **LinearBoost security issues have been updated** in this release; upgrade from earlier versions to stay patched.
|
|
35
36
|
|
|
36
|
-
## 🧪 Quickstart
|
|
37
|
+
## 🧪 Quickstart demos
|
|
38
|
+
|
|
39
|
+
### LinearBoost
|
|
37
40
|
|
|
38
41
|
Want to see how LinearBoost works in practice?
|
|
39
42
|
|
|
@@ -44,19 +47,11 @@ This Jupyter notebook shows how to:
|
|
|
44
47
|
- Train `LinearBoostClassifier`
|
|
45
48
|
- Evaluate using F1 score and cross-validation
|
|
46
49
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
In benchmarks across seven well-known datasets, LinearBoost:
|
|
50
|
-
|
|
51
|
-
- Outperformed XGBoost on all seven datasets
|
|
52
|
-
- Surpassed LightGBM on five datasets
|
|
53
|
-
- Achieved up to **98% faster runtime** compared to both algorithms
|
|
50
|
+
### SEFRBoost
|
|
54
51
|
|
|
55
|
-
|
|
52
|
+
- **[`optuna_sefrboost_demo.ipynb`](notebooks/optuna_sefrboost_demo.ipynb)** — Hyperparameter search for **`SEFRBoostClassifier`** with **[Optuna](https://optuna.org/)** on sklearn’s Breast Cancer Wisconsin data: default baseline, then **5-fold stratified CV** optimizing F1 (install `optuna` in addition to `linearboost` and `scikit-learn`).
|
|
56
53
|
|
|
57
|
-
|
|
58
|
-
- Exceptional Speed: Blazing fast training and inference times
|
|
59
|
-
- Resource Efficient: Low memory usage, ideal for large datasets
|
|
54
|
+
LinearBoost is a boosting classifier built on the linear classifier SEFR: it trains SEFR classifiers in sequence with AdaBoost-style sample reweighting (SAMME or SAMME.R), optionally on kernel features. Training and prediction are fast, and memory use is low.
|
|
60
55
|
|
|
61
56
|
---
|
|
62
57
|
|
|
@@ -71,6 +66,8 @@ from linearboost import SEFRBoostClassifier, SEFRBoostRegressor
|
|
|
71
66
|
# or: from linearboost.sefr_boost import SEFRBoostClassifier, SEFRBoostRegressor
|
|
72
67
|
```
|
|
73
68
|
|
|
69
|
+
Hands-on examples: [`notebooks/optuna_sefrboost_demo.ipynb`](notebooks/optuna_sefrboost_demo.ipynb) (Optuna tuning).
|
|
70
|
+
|
|
74
71
|
### Security updates
|
|
75
72
|
|
|
76
73
|
LinearBoost-related security issues are addressed in v0.2.0. Upgrade from older releases.
|
|
@@ -252,7 +249,7 @@ The documentation is available at https://linearboost.readthedocs.io/.
|
|
|
252
249
|
|
|
253
250
|
## Recommended Parameters for LinearBoost
|
|
254
251
|
|
|
255
|
-
|
|
252
|
+
Suggested starting values for the main parameters:
|
|
256
253
|
|
|
257
254
|
- **`n_estimators`**:
|
|
258
255
|
A range of 10 to 200 is suggested, with higher values potentially improving performance at the cost of longer training times. When using `early_stopping=True`, you can set a higher value (e.g., 500) and let training stop automatically.
|
|
@@ -305,150 +302,6 @@ The following parameters yielded optimal results during testing. All results are
|
|
|
305
302
|
|
|
306
303
|
These parameters should serve as a solid starting point for most datasets. For fine-tuning, consider using hyperparameter optimization tools like [Optuna](https://optuna.org/).
|
|
307
304
|
|
|
308
|
-
## Results
|
|
309
|
-
|
|
310
|
-
All of the results are reported based on 10-fold Cross-Validation. The weighted F1 score is reported, i.e. f1_score(y_valid, y_pred, average = 'weighted').
|
|
311
|
-
|
|
312
|
-
## Performance Comparison: F1 Scores Across Datasets
|
|
313
|
-
|
|
314
|
-
The following table presents the F1 scores of LinearBoost in comparison with XGBoost, CatBoost, and LightGBM across seven standard benchmark datasets. **Each result is obtained by running Optuna with 200 trials** to find the best hyperparameters for each algorithm and dataset, ensuring a fair and robust comparison.
|
|
315
|
-
|
|
316
|
-
| Dataset | XGBoost | CatBoost | LightGBM | LinearBoost |
|
|
317
|
-
|--------------------------------------|----------|----------|----------|-------------|
|
|
318
|
-
| Breast Cancer Wisconsin (Diagnostic) | 0.9767 | 0.9859 | 0.9771 | 0.9822 |
|
|
319
|
-
| Heart Disease | 0.8502 | 0.8529 | 0.8467 | 0.8507 |
|
|
320
|
-
| Pima Indians Diabetes Database | 0.7719 | 0.7776 | 0.7816 | 0.7753 |
|
|
321
|
-
| Banknote Authentication | 0.9985 | 1.0000 | 0.9993 | 1.0000 |
|
|
322
|
-
| Haberman's Survival | 0.7193 | 0.7427 | 0.7257 | 0.7485 |
|
|
323
|
-
| Loan Status Prediction | 0.8281 | 0.8495 | 0.8277 | 0.8387 |
|
|
324
|
-
| PCMAC | 0.9310 | 0.9351 | 0.9361 | 0.9331 |
|
|
325
|
-
|
|
326
|
-
### Experiment Details
|
|
327
|
-
- **Hyperparameter Optimization**:
|
|
328
|
-
- Each algorithm was tuned using **Optuna**, a powerful hyperparameter optimization framework.
|
|
329
|
-
- **200 trials** were conducted for each algorithm-dataset pair to identify the optimal hyperparameters.
|
|
330
|
-
- **Consistency**: This rigorous approach ensures fair comparison by evaluating each algorithm under its best-performing configuration.
|
|
331
|
-
|
|
332
|
-
### Key Highlights
|
|
333
|
-
- **LinearBoost** achieves competitive or superior F1 scores compared to the state-of-the-art algorithms.
|
|
334
|
-
- **Haberman's Survival**: LinearBoost achieves the highest F1 score (**0.7485**), outperforming all other algorithms.
|
|
335
|
-
- **Banknote Authentication**: LinearBoost matches the perfect F1 score of **1** achieved by CatBoost.
|
|
336
|
-
- LinearBoost demonstrates consistent performance across diverse datasets, making it a robust and efficient choice for classification tasks.
|
|
337
|
-
|
|
338
|
-
## Runtime Comparison: Time to Reach Best F1 Score
|
|
339
|
-
|
|
340
|
-
The following table shows the runtime (in seconds) required by LinearBoost, XGBoost, CatBoost, and LightGBM to achieve their best F1 scores. **Each result is obtained by running Optuna with 200 trials** to optimize the hyperparameters for each algorithm and dataset.
|
|
341
|
-
|
|
342
|
-
| Dataset | XGBoost | CatBoost | LightGBM | LinearBoost |
|
|
343
|
-
|--------------------------------------|----------|----------|----------|-------------|
|
|
344
|
-
| Breast Cancer Wisconsin (Diagnostic) | 3.22 | 9.68 | 4.52 | 0.30 |
|
|
345
|
-
| Heart Disease | 1.13 | 0.60 | 0.51 | 0.49 |
|
|
346
|
-
| Pima Indians Diabetes Database | 6.86 | 3.50 | 2.52 | 0.16 |
|
|
347
|
-
| Banknote Authentication | 0.46 | 4.26 | 5.54 | 0.33 |
|
|
348
|
-
| Haberman's Survival | 4.41 | 8.28 | 5.72 | 0.11 |
|
|
349
|
-
| Loan Status Prediction | 0.83 | 97.89 | 28.41 | 0.44 |
|
|
350
|
-
| PCMAC | 150.33 | 83.52 | 42.23 | 75.06 |
|
|
351
|
-
|
|
352
|
-
### Experiment Details
|
|
353
|
-
- **Hyperparameter Optimization**:
|
|
354
|
-
- Each algorithm was tuned using **Optuna** with **200 trials** per algorithm-dataset pair.
|
|
355
|
-
- The runtime includes the time to reach the best F1 score using the optimized hyperparameters.
|
|
356
|
-
- **Fair Comparison**: All algorithms were evaluated under their best configurations to ensure consistency.
|
|
357
|
-
|
|
358
|
-
### Key Highlights
|
|
359
|
-
- **LinearBoost** demonstrates exceptional runtime efficiency while achieving competitive F1 scores:
|
|
360
|
-
- **Breast Cancer Wisconsin (Diagnostic)**: LinearBoost achieves the best F1 score in just **0.30 seconds**, compared to **3.22 seconds** for XGBoost and **9.68 seconds** for CatBoost.
|
|
361
|
-
- **Loan Status Prediction**: LinearBoost runs in **0.44 seconds**, outperforming LightGBM (**28.41 seconds**) and CatBoost (**97.89 seconds**).
|
|
362
|
-
- Across most datasets, LinearBoost reduces runtime by up to **98%** compared to XGBoost and LightGBM while maintaining competitive performance.
|
|
363
|
-
|
|
364
|
-
### Tuned Hyperparameters
|
|
365
|
-
|
|
366
|
-
#### XGBoost
|
|
367
|
-
```python
|
|
368
|
-
params = {
|
|
369
|
-
'objective': 'binary:logistic',
|
|
370
|
-
'use_label_encoder': False,
|
|
371
|
-
'n_estimators': trial.suggest_int('n_estimators', 20, 1000),
|
|
372
|
-
'max_depth': trial.suggest_int('max_depth', 1, 20),
|
|
373
|
-
'learning_rate': trial.suggest_uniform('learning_rate', 0.01, 0.7),
|
|
374
|
-
'gamma': trial.suggest_loguniform('gamma', 1e-8, 1.0),
|
|
375
|
-
'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
|
|
376
|
-
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
|
|
377
|
-
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
|
|
378
|
-
'reg_alpha': trial.suggest_loguniform('reg_alpha', 1e-8, 1.0),
|
|
379
|
-
'reg_lambda': trial.suggest_loguniform('reg_lambda', 1e-8, 1.0),
|
|
380
|
-
'enable_categorical': True,
|
|
381
|
-
'eval_metric': 'logloss'
|
|
382
|
-
}
|
|
383
|
-
```
|
|
384
|
-
|
|
385
|
-
#### CatBoost
|
|
386
|
-
```python
|
|
387
|
-
params = {
|
|
388
|
-
'iterations': trial.suggest_int('iterations', 50, 500),
|
|
389
|
-
'depth': trial.suggest_int('depth', 1, 16),
|
|
390
|
-
'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.5),
|
|
391
|
-
'l2_leaf_reg': trial.suggest_loguniform('l2_leaf_reg', 1e-8, 10.0),
|
|
392
|
-
'random_strength': trial.suggest_loguniform('random_strength', 1e-8, 10.0),
|
|
393
|
-
'bagging_temperature': trial.suggest_loguniform('bagging_temperature', 1e-1, 10.0),
|
|
394
|
-
'border_count': trial.suggest_int('border_count', 32, 255),
|
|
395
|
-
'grow_policy': trial.suggest_categorical('grow_policy', ['SymmetricTree', 'Depthwise', 'Lossguide']),
|
|
396
|
-
'min_data_in_leaf': trial.suggest_int('min_data_in_leaf', 1, 100),
|
|
397
|
-
'rsm': trial.suggest_uniform('rsm', 0.1, 1.0),
|
|
398
|
-
'loss_function': 'Logloss',
|
|
399
|
-
'eval_metric': 'F1',
|
|
400
|
-
'cat_features': categorical_cols
|
|
401
|
-
}
|
|
402
|
-
```
|
|
403
|
-
|
|
404
|
-
#### LightGBM
|
|
405
|
-
```python
|
|
406
|
-
params = {
|
|
407
|
-
'objective': 'binary',
|
|
408
|
-
'metric': 'binary_logloss',
|
|
409
|
-
'boosting_type': trial.suggest_categorical('boosting_type', ['gbdt', 'dart', 'goss']),
|
|
410
|
-
'num_leaves': trial.suggest_int('num_leaves', 2, 256),
|
|
411
|
-
'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.1),
|
|
412
|
-
'n_estimators': trial.suggest_int('n_estimators', 20, 1000),
|
|
413
|
-
'max_depth': trial.suggest_int('max_depth', 1, 20),
|
|
414
|
-
'min_child_samples': trial.suggest_int('min_child_samples', 1, 100),
|
|
415
|
-
'subsample': trial.suggest_uniform('subsample', 0.5, 1.0),
|
|
416
|
-
'colsample_bytree': trial.suggest_uniform('colsample_bytree', 0.5, 1.0),
|
|
417
|
-
'reg_alpha': trial.suggest_loguniform('reg_alpha', 1e-8, 10.0),
|
|
418
|
-
'reg_lambda': trial.suggest_loguniform('reg_lambda', 1e-8, 10.0),
|
|
419
|
-
'min_split_gain': trial.suggest_loguniform('min_split_gain', 1e-8, 1.0),
|
|
420
|
-
'cat_smooth': trial.suggest_int('cat_smooth', 1, 100),
|
|
421
|
-
'cat_l2': trial.suggest_loguniform('cat_l2', 1e-8, 10.0),
|
|
422
|
-
'verbosity': -1
|
|
423
|
-
}
|
|
424
|
-
```
|
|
425
|
-
|
|
426
|
-
#### LinearBoost
|
|
427
|
-
```python
|
|
428
|
-
params = {
|
|
429
|
-
'n_estimators': trial.suggest_int('n_estimators', 10, 500),
|
|
430
|
-
'learning_rate': trial.suggest_loguniform('learning_rate', 0.01, 1),
|
|
431
|
-
'algorithm': trial.suggest_categorical('algorithm', ['SAMME', 'SAMME.R']),
|
|
432
|
-
'scaler': trial.suggest_categorical('scaler', ['minmax', 'robust', 'quantile-uniform', 'quantile-normal']),
|
|
433
|
-
'kernel': trial.suggest_categorical('kernel', ['linear', 'rbf', 'poly']),
|
|
434
|
-
'boosting_type': trial.suggest_categorical('boosting_type', ['adaboost', 'gradient']),
|
|
435
|
-
'class_weight': trial.suggest_categorical('class_weight', [None, 'balanced']),
|
|
436
|
-
'subsample': trial.suggest_float('subsample', 0.6, 1.0),
|
|
437
|
-
'shrinkage': trial.suggest_float('shrinkage', 0.7, 1.0),
|
|
438
|
-
'early_stopping': True,
|
|
439
|
-
'n_iter_no_change': 5,
|
|
440
|
-
}
|
|
441
|
-
```
|
|
442
|
-
|
|
443
|
-
### Why LinearBoost?
|
|
444
|
-
LinearBoost's combination of **runtime efficiency** and **high accuracy** makes it a powerful choice for real-world machine learning tasks, particularly in resource-constrained or real-time applications.
|
|
445
|
-
|
|
446
|
-
### 📰 Featured in:
|
|
447
|
-
- [LightGBM Alternatives: A Comprehensive Comparison](https://nightwatcherai.com/blog/lightgbm-alternatives)
|
|
448
|
-
_by Jordan Cole, March 11, 2025_
|
|
449
|
-
*Discusses how LinearBoost outperforms traditional boosting frameworks in terms of speed while maintaining accuracy.*
|
|
450
|
-
|
|
451
|
-
|
|
452
305
|
## Future Developments
|
|
453
306
|
|
|
454
307
|
These are not yet supported in this current version, but are in the future plans:
|
|
@@ -1,13 +1,15 @@
|
|
|
1
|
-
# LinearBoost
|
|
1
|
+
# LinearBoost and SEFRBoost Classifiers
|
|
2
2
|
|
|
3
3
|

|
|
4
4
|
[](https://pypi.org/project/linearboost/)
|
|
5
5
|

|
|
6
6
|
[](https://pepy.tech/projects/linearboost)
|
|
7
7
|
|
|
8
|
-
**Current release: v0.2.0.** This version adds **SEFRBoost**—gradient boosting with linear SEFR splits at tree nodes—via `SEFRBoostClassifier` and `SEFRBoostRegressor` (`from linearboost import …` or `linearboost.sefr_boost`). **LinearBoost security issues have been updated** in this release; upgrade from earlier versions to stay patched.
|
|
8
|
+
**Current release: v0.2.0.** This version adds **SEFRBoost**—gradient boosting with linear SEFR splits at tree nodes—via `SEFRBoostClassifier` and `SEFRBoostRegressor` (`from linearboost import …` or `linearboost.sefr_boost`). It is pronounced /sɛfr-buːst/. **LinearBoost security issues have been updated** in this release; upgrade from earlier versions to stay patched.
|
|
9
9
|
|
|
10
|
-
## 🧪 Quickstart
|
|
10
|
+
## 🧪 Quickstart demos
|
|
11
|
+
|
|
12
|
+
### LinearBoost
|
|
11
13
|
|
|
12
14
|
Want to see how LinearBoost works in practice?
|
|
13
15
|
|
|
@@ -18,19 +20,11 @@ This Jupyter notebook shows how to:
|
|
|
18
20
|
- Train `LinearBoostClassifier`
|
|
19
21
|
- Evaluate using F1 score and cross-validation
|
|
20
22
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
In benchmarks across seven well-known datasets, LinearBoost:
|
|
24
|
-
|
|
25
|
-
- Outperformed XGBoost on all seven datasets
|
|
26
|
-
- Surpassed LightGBM on five datasets
|
|
27
|
-
- Achieved up to **98% faster runtime** compared to both algorithms
|
|
23
|
+
### SEFRBoost
|
|
28
24
|
|
|
29
|
-
|
|
25
|
+
- **[`optuna_sefrboost_demo.ipynb`](notebooks/optuna_sefrboost_demo.ipynb)** — Hyperparameter search for **`SEFRBoostClassifier`** with **[Optuna](https://optuna.org/)** on sklearn’s Breast Cancer Wisconsin data: default baseline, then **5-fold stratified CV** optimizing F1 (install `optuna` in addition to `linearboost` and `scikit-learn`).
|
|
30
26
|
|
|
31
|
-
|
|
32
|
-
- Exceptional Speed: Blazing fast training and inference times
|
|
33
|
-
- Resource Efficient: Low memory usage, ideal for large datasets
|
|
27
|
+
LinearBoost is a boosting classifier built on the linear classifier SEFR: it trains SEFR classifiers in sequence with AdaBoost-style sample reweighting (SAMME or SAMME.R), optionally on kernel features. Training and prediction are fast, and memory use is low.
|
|
34
28
|
|
|
35
29
|
---
|
|
36
30
|
|
|
@@ -45,6 +39,8 @@ from linearboost import SEFRBoostClassifier, SEFRBoostRegressor
|
|
|
45
39
|
# or: from linearboost.sefr_boost import SEFRBoostClassifier, SEFRBoostRegressor
|
|
46
40
|
```
|
|
47
41
|
|
|
42
|
+
Hands-on examples: [`notebooks/optuna_sefrboost_demo.ipynb`](notebooks/optuna_sefrboost_demo.ipynb) (Optuna tuning).
|
|
43
|
+
|
|
48
44
|
### Security updates
|
|
49
45
|
|
|
50
46
|
LinearBoost-related security issues are addressed in v0.2.0. Upgrade from older releases.
|
|
@@ -226,7 +222,7 @@ The documentation is available at https://linearboost.readthedocs.io/.
|
|
|
226
222
|
|
|
227
223
|
## Recommended Parameters for LinearBoost
|
|
228
224
|
|
|
229
|
-
|
|
225
|
+
Suggested starting values for the main parameters:
|
|
230
226
|
|
|
231
227
|
- **`n_estimators`**:
|
|
232
228
|
A range of 10 to 200 is suggested, with higher values potentially improving performance at the cost of longer training times. When using `early_stopping=True`, you can set a higher value (e.g., 500) and let training stop automatically.
|
|
@@ -279,150 +275,6 @@ The following parameters yielded optimal results during testing. All results are
|
|
|
279
275
|
|
|
280
276
|
These parameters should serve as a solid starting point for most datasets. For fine-tuning, consider using hyperparameter optimization tools like [Optuna](https://optuna.org/).
|
|
281
277
|
|
|
282
|
-
## Results
|
|
283
|
-
|
|
284
|
-
All of the results are reported based on 10-fold Cross-Validation. The weighted F1 score is reported, i.e. f1_score(y_valid, y_pred, average = 'weighted').
|
|
285
|
-
|
|
286
|
-
## Performance Comparison: F1 Scores Across Datasets
|
|
287
|
-
|
|
288
|
-
The following table presents the F1 scores of LinearBoost in comparison with XGBoost, CatBoost, and LightGBM across seven standard benchmark datasets. **Each result is obtained by running Optuna with 200 trials** to find the best hyperparameters for each algorithm and dataset, ensuring a fair and robust comparison.
|
|
289
|
-
|
|
290
|
-
| Dataset | XGBoost | CatBoost | LightGBM | LinearBoost |
|
|
291
|
-
|--------------------------------------|----------|----------|----------|-------------|
|
|
292
|
-
| Breast Cancer Wisconsin (Diagnostic) | 0.9767 | 0.9859 | 0.9771 | 0.9822 |
|
|
293
|
-
| Heart Disease | 0.8502 | 0.8529 | 0.8467 | 0.8507 |
|
|
294
|
-
| Pima Indians Diabetes Database | 0.7719 | 0.7776 | 0.7816 | 0.7753 |
|
|
295
|
-
| Banknote Authentication | 0.9985 | 1.0000 | 0.9993 | 1.0000 |
|
|
296
|
-
| Haberman's Survival | 0.7193 | 0.7427 | 0.7257 | 0.7485 |
|
|
297
|
-
| Loan Status Prediction | 0.8281 | 0.8495 | 0.8277 | 0.8387 |
|
|
298
|
-
| PCMAC | 0.9310 | 0.9351 | 0.9361 | 0.9331 |
|
|
299
|
-
|
|
300
|
-
### Experiment Details
|
|
301
|
-
- **Hyperparameter Optimization**:
|
|
302
|
-
- Each algorithm was tuned using **Optuna**, a powerful hyperparameter optimization framework.
|
|
303
|
-
- **200 trials** were conducted for each algorithm-dataset pair to identify the optimal hyperparameters.
|
|
304
|
-
- **Consistency**: This rigorous approach ensures fair comparison by evaluating each algorithm under its best-performing configuration.
|
|
305
|
-
|
|
306
|
-
### Key Highlights
|
|
307
|
-
- **LinearBoost** achieves competitive or superior F1 scores compared to the state-of-the-art algorithms.
|
|
308
|
-
- **Haberman's Survival**: LinearBoost achieves the highest F1 score (**0.7485**), outperforming all other algorithms.
|
|
309
|
-
- **Banknote Authentication**: LinearBoost matches the perfect F1 score of **1** achieved by CatBoost.
|
|
310
|
-
- LinearBoost demonstrates consistent performance across diverse datasets, making it a robust and efficient choice for classification tasks.
|
|
311
|
-
|
|
312
|
-
## Runtime Comparison: Time to Reach Best F1 Score
|
|
313
|
-
|
|
314
|
-
The following table shows the runtime (in seconds) required by LinearBoost, XGBoost, CatBoost, and LightGBM to achieve their best F1 scores. **Each result is obtained by running Optuna with 200 trials** to optimize the hyperparameters for each algorithm and dataset.
|
|
315
|
-
|
|
316
|
-
| Dataset | XGBoost | CatBoost | LightGBM | LinearBoost |
|
|
317
|
-
|--------------------------------------|----------|----------|----------|-------------|
|
|
318
|
-
| Breast Cancer Wisconsin (Diagnostic) | 3.22 | 9.68 | 4.52 | 0.30 |
|
|
319
|
-
| Heart Disease | 1.13 | 0.60 | 0.51 | 0.49 |
|
|
320
|
-
| Pima Indians Diabetes Database | 6.86 | 3.50 | 2.52 | 0.16 |
|
|
321
|
-
| Banknote Authentication | 0.46 | 4.26 | 5.54 | 0.33 |
|
|
322
|
-
| Haberman's Survival | 4.41 | 8.28 | 5.72 | 0.11 |
|
|
323
|
-
| Loan Status Prediction | 0.83 | 97.89 | 28.41 | 0.44 |
|
|
324
|
-
| PCMAC | 150.33 | 83.52 | 42.23 | 75.06 |
|
|
325
|
-
|
|
326
|
-
### Experiment Details
|
|
327
|
-
- **Hyperparameter Optimization**:
|
|
328
|
-
- Each algorithm was tuned using **Optuna** with **200 trials** per algorithm-dataset pair.
|
|
329
|
-
- The runtime includes the time to reach the best F1 score using the optimized hyperparameters.
|
|
330
|
-
- **Fair Comparison**: All algorithms were evaluated under their best configurations to ensure consistency.
|
|
331
|
-
|
|
332
|
-
### Key Highlights
|
|
333
|
-
- **LinearBoost** demonstrates exceptional runtime efficiency while achieving competitive F1 scores:
|
|
334
|
-
- **Breast Cancer Wisconsin (Diagnostic)**: LinearBoost achieves the best F1 score in just **0.30 seconds**, compared to **3.22 seconds** for XGBoost and **9.68 seconds** for CatBoost.
|
|
335
|
-
- **Loan Status Prediction**: LinearBoost runs in **0.44 seconds**, outperforming LightGBM (**28.41 seconds**) and CatBoost (**97.89 seconds**).
|
|
336
|
-
- Across most datasets, LinearBoost reduces runtime by up to **98%** compared to XGBoost and LightGBM while maintaining competitive performance.
|
|
337
|
-
|
|
338
|
-
### Tuned Hyperparameters
|
|
339
|
-
|
|
340
|
-
#### XGBoost
|
|
341
|
-
```python
|
|
342
|
-
params = {
|
|
343
|
-
'objective': 'binary:logistic',
|
|
344
|
-
'use_label_encoder': False,
|
|
345
|
-
'n_estimators': trial.suggest_int('n_estimators', 20, 1000),
|
|
346
|
-
'max_depth': trial.suggest_int('max_depth', 1, 20),
|
|
347
|
-
'learning_rate': trial.suggest_uniform('learning_rate', 0.01, 0.7),
|
|
348
|
-
'gamma': trial.suggest_loguniform('gamma', 1e-8, 1.0),
|
|
349
|
-
'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
|
|
350
|
-
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
|
|
351
|
-
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
|
|
352
|
-
'reg_alpha': trial.suggest_loguniform('reg_alpha', 1e-8, 1.0),
|
|
353
|
-
'reg_lambda': trial.suggest_loguniform('reg_lambda', 1e-8, 1.0),
|
|
354
|
-
'enable_categorical': True,
|
|
355
|
-
'eval_metric': 'logloss'
|
|
356
|
-
}
|
|
357
|
-
```
|
|
358
|
-
|
|
359
|
-
#### CatBoost
|
|
360
|
-
```python
|
|
361
|
-
params = {
|
|
362
|
-
'iterations': trial.suggest_int('iterations', 50, 500),
|
|
363
|
-
'depth': trial.suggest_int('depth', 1, 16),
|
|
364
|
-
'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.5),
|
|
365
|
-
'l2_leaf_reg': trial.suggest_loguniform('l2_leaf_reg', 1e-8, 10.0),
|
|
366
|
-
'random_strength': trial.suggest_loguniform('random_strength', 1e-8, 10.0),
|
|
367
|
-
'bagging_temperature': trial.suggest_loguniform('bagging_temperature', 1e-1, 10.0),
|
|
368
|
-
'border_count': trial.suggest_int('border_count', 32, 255),
|
|
369
|
-
'grow_policy': trial.suggest_categorical('grow_policy', ['SymmetricTree', 'Depthwise', 'Lossguide']),
|
|
370
|
-
'min_data_in_leaf': trial.suggest_int('min_data_in_leaf', 1, 100),
|
|
371
|
-
'rsm': trial.suggest_uniform('rsm', 0.1, 1.0),
|
|
372
|
-
'loss_function': 'Logloss',
|
|
373
|
-
'eval_metric': 'F1',
|
|
374
|
-
'cat_features': categorical_cols
|
|
375
|
-
}
|
|
376
|
-
```
|
|
377
|
-
|
|
378
|
-
#### LightGBM
|
|
379
|
-
```python
|
|
380
|
-
params = {
|
|
381
|
-
'objective': 'binary',
|
|
382
|
-
'metric': 'binary_logloss',
|
|
383
|
-
'boosting_type': trial.suggest_categorical('boosting_type', ['gbdt', 'dart', 'goss']),
|
|
384
|
-
'num_leaves': trial.suggest_int('num_leaves', 2, 256),
|
|
385
|
-
'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.1),
|
|
386
|
-
'n_estimators': trial.suggest_int('n_estimators', 20, 1000),
|
|
387
|
-
'max_depth': trial.suggest_int('max_depth', 1, 20),
|
|
388
|
-
'min_child_samples': trial.suggest_int('min_child_samples', 1, 100),
|
|
389
|
-
'subsample': trial.suggest_uniform('subsample', 0.5, 1.0),
|
|
390
|
-
'colsample_bytree': trial.suggest_uniform('colsample_bytree', 0.5, 1.0),
|
|
391
|
-
'reg_alpha': trial.suggest_loguniform('reg_alpha', 1e-8, 10.0),
|
|
392
|
-
'reg_lambda': trial.suggest_loguniform('reg_lambda', 1e-8, 10.0),
|
|
393
|
-
'min_split_gain': trial.suggest_loguniform('min_split_gain', 1e-8, 1.0),
|
|
394
|
-
'cat_smooth': trial.suggest_int('cat_smooth', 1, 100),
|
|
395
|
-
'cat_l2': trial.suggest_loguniform('cat_l2', 1e-8, 10.0),
|
|
396
|
-
'verbosity': -1
|
|
397
|
-
}
|
|
398
|
-
```
|
|
399
|
-
|
|
400
|
-
#### LinearBoost
|
|
401
|
-
```python
|
|
402
|
-
params = {
|
|
403
|
-
'n_estimators': trial.suggest_int('n_estimators', 10, 500),
|
|
404
|
-
'learning_rate': trial.suggest_loguniform('learning_rate', 0.01, 1),
|
|
405
|
-
'algorithm': trial.suggest_categorical('algorithm', ['SAMME', 'SAMME.R']),
|
|
406
|
-
'scaler': trial.suggest_categorical('scaler', ['minmax', 'robust', 'quantile-uniform', 'quantile-normal']),
|
|
407
|
-
'kernel': trial.suggest_categorical('kernel', ['linear', 'rbf', 'poly']),
|
|
408
|
-
'boosting_type': trial.suggest_categorical('boosting_type', ['adaboost', 'gradient']),
|
|
409
|
-
'class_weight': trial.suggest_categorical('class_weight', [None, 'balanced']),
|
|
410
|
-
'subsample': trial.suggest_float('subsample', 0.6, 1.0),
|
|
411
|
-
'shrinkage': trial.suggest_float('shrinkage', 0.7, 1.0),
|
|
412
|
-
'early_stopping': True,
|
|
413
|
-
'n_iter_no_change': 5,
|
|
414
|
-
}
|
|
415
|
-
```
|
|
416
|
-
|
|
417
|
-
### Why LinearBoost?
|
|
418
|
-
LinearBoost's combination of **runtime efficiency** and **high accuracy** makes it a powerful choice for real-world machine learning tasks, particularly in resource-constrained or real-time applications.
|
|
419
|
-
|
|
420
|
-
### 📰 Featured in:
|
|
421
|
-
- [LightGBM Alternatives: A Comprehensive Comparison](https://nightwatcherai.com/blog/lightgbm-alternatives)
|
|
422
|
-
_by Jordan Cole, March 11, 2025_
|
|
423
|
-
*Discusses how LinearBoost outperforms traditional boosting frameworks in terms of speed while maintaining accuracy.*
|
|
424
|
-
|
|
425
|
-
|
|
426
278
|
## Future Developments
|
|
427
279
|
|
|
428
280
|
These are not yet supported in this current version, but are in the future plans:
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
{
|
|
2
|
+
"cells": [
|
|
3
|
+
{
|
|
4
|
+
"cell_type": "markdown",
|
|
5
|
+
"id": "7fb27b941602401d91542211134fc71a",
|
|
6
|
+
"metadata": {},
|
|
7
|
+
"source": [
|
|
8
|
+
"# Optuna + `SEFRBoostClassifier`\n",
|
|
9
|
+
"\n",
|
|
10
|
+
"Tune [`SEFRBoostClassifier`](https://github.com/LinearBoost/linearboost-classifier) (gradient boosting with SEFR oblique splits) using [Optuna](https://optuna.org/) on sklearn’s **Breast Cancer Wisconsin** dataset (binary).\n",
|
|
11
|
+
"\n",
|
|
12
|
+
"**Install (if needed):** `pip install linearboost optuna scikit-learn` — or install this repo editable: `pip install -e .` from the repository root."
|
|
13
|
+
]
|
|
14
|
+
},
|
|
15
|
+
{
|
|
16
|
+
"cell_type": "code",
|
|
17
|
+
"execution_count": null,
|
|
18
|
+
"id": "acae54e37e7d407bbb7b55eff062a284",
|
|
19
|
+
"metadata": {},
|
|
20
|
+
"outputs": [],
|
|
21
|
+
"source": [
|
|
22
|
+
"import warnings\n",
|
|
23
|
+
"\n",
|
|
24
|
+
"import numpy as np\n",
|
|
25
|
+
"import optuna\n",
|
|
26
|
+
"from sklearn.datasets import load_breast_cancer\n",
|
|
27
|
+
"from sklearn.metrics import f1_score, roc_auc_score\n",
|
|
28
|
+
"from sklearn.model_selection import StratifiedKFold, train_test_split\n",
|
|
29
|
+
"from sklearn.pipeline import Pipeline\n",
|
|
30
|
+
"from sklearn.preprocessing import StandardScaler\n",
|
|
31
|
+
"\n",
|
|
32
|
+
"from linearboost import SEFRBoostClassifier\n",
|
|
33
|
+
"\n",
|
|
34
|
+
"warnings.filterwarnings(\"ignore\")\n",
|
|
35
|
+
"optuna.logging.set_verbosity(optuna.logging.WARNING)"
|
|
36
|
+
]
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"cell_type": "markdown",
|
|
40
|
+
"id": "9a63283cbaf04dbcab1f6479b197f3a8",
|
|
41
|
+
"metadata": {},
|
|
42
|
+
"source": [
|
|
43
|
+
"## 1. Load data and train / test split\n",
|
|
44
|
+
"\n",
|
|
45
|
+
"`SEFRBoostClassifier` expects **dense numeric** input; we use `StandardScaler` in a pipeline."
|
|
46
|
+
]
|
|
47
|
+
},
|
|
48
|
+
{
|
|
49
|
+
"cell_type": "code",
|
|
50
|
+
"execution_count": null,
|
|
51
|
+
"id": "8dd0d8092fe74a7c96281538738b07e2",
|
|
52
|
+
"metadata": {},
|
|
53
|
+
"outputs": [],
|
|
54
|
+
"source": [
|
|
55
|
+
"X, y = load_breast_cancer(return_X_y=True)\n",
|
|
56
|
+
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
|
57
|
+
" X, y, test_size=0.25, stratify=y, random_state=42\n",
|
|
58
|
+
")\n",
|
|
59
|
+
"print(\"Train:\", X_train.shape, \"Test:\", X_test.shape, \"Classes:\", np.unique(y))"
|
|
60
|
+
]
|
|
61
|
+
},
|
|
62
|
+
{
|
|
63
|
+
"cell_type": "markdown",
|
|
64
|
+
"id": "72eea5119410473aa328ad9291626812",
|
|
65
|
+
"metadata": {},
|
|
66
|
+
"source": [
|
|
67
|
+
"## 2. Quick baseline (default hyperparameters)\n",
|
|
68
|
+
"\n",
|
|
69
|
+
"`Pipeline(StandardScaler → SEFRBoostClassifier)`."
|
|
70
|
+
]
|
|
71
|
+
},
|
|
72
|
+
{
|
|
73
|
+
"cell_type": "code",
|
|
74
|
+
"execution_count": null,
|
|
75
|
+
"id": "8edb47106e1a46a883d545849b8ab81b",
|
|
76
|
+
"metadata": {},
|
|
77
|
+
"outputs": [],
|
|
78
|
+
"source": [
|
|
79
|
+
"baseline = Pipeline(\n",
|
|
80
|
+
" [\n",
|
|
81
|
+
" (\"scale\", StandardScaler()),\n",
|
|
82
|
+
" (\"clf\", SEFRBoostClassifier(n_estimators=50, random_state=42)),\n",
|
|
83
|
+
" ]\n",
|
|
84
|
+
")\n",
|
|
85
|
+
"baseline.fit(X_train, y_train)\n",
|
|
86
|
+
"y_pred = baseline.predict(X_test)\n",
|
|
87
|
+
"y_proba = baseline.predict_proba(X_test)[:, 1]\n",
|
|
88
|
+
"print(\"Baseline F1 (weighted):\", f1_score(y_test, y_pred, average=\"weighted\"))\n",
|
|
89
|
+
"print(\"Baseline ROC-AUC:\", roc_auc_score(y_test, y_proba))"
|
|
90
|
+
]
|
|
91
|
+
},
|
|
92
|
+
{
|
|
93
|
+
"cell_type": "markdown",
|
|
94
|
+
"id": "10185d26023b46108eb7d9f57d49d2b3",
|
|
95
|
+
"metadata": {},
|
|
96
|
+
"source": [
|
|
97
|
+
"## 3. Optuna: maximize cross-validated F1\n",
|
|
98
|
+
"\n",
|
|
99
|
+
"Objective: suggest tree size, learning rate, depth, leaf constraints, and subsample; evaluate with **5-fold stratified CV** on the training set only (fast enough for local runs)."
|
|
100
|
+
]
|
|
101
|
+
},
|
|
102
|
+
{
|
|
103
|
+
"cell_type": "code",
|
|
104
|
+
"execution_count": null,
|
|
105
|
+
"id": "8763a12b2bbd4a93a75aff182afb95dc",
|
|
106
|
+
"metadata": {},
|
|
107
|
+
"outputs": [],
|
|
108
|
+
"source": [
|
|
109
|
+
"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n",
|
|
110
|
+
"\n",
|
|
111
|
+
"\n",
|
|
112
|
+
"def objective(trial: optuna.Trial) -> float:\n",
|
|
113
|
+
" params = {\n",
|
|
114
|
+
" \"n_estimators\": trial.suggest_int(\"n_estimators\", 20, 150),\n",
|
|
115
|
+
" \"learning_rate\": trial.suggest_float(\"learning_rate\", 0.02, 0.3, log=True),\n",
|
|
116
|
+
" \"max_depth\": trial.suggest_int(\"max_depth\", 2, 6),\n",
|
|
117
|
+
" \"min_samples_leaf\": trial.suggest_int(\"min_samples_leaf\", 5, 40),\n",
|
|
118
|
+
" \"min_samples_split\": trial.suggest_int(\"min_samples_split\", 10, 80),\n",
|
|
119
|
+
" \"subsample\": trial.suggest_float(\"subsample\", 0.6, 1.0),\n",
|
|
120
|
+
" \"random_state\": 42,\n",
|
|
121
|
+
" }\n",
|
|
122
|
+
" pipe = Pipeline(\n",
|
|
123
|
+
" [\n",
|
|
124
|
+
" (\"scale\", StandardScaler()),\n",
|
|
125
|
+
" (\"clf\", SEFRBoostClassifier(**params)),\n",
|
|
126
|
+
" ]\n",
|
|
127
|
+
" )\n",
|
|
128
|
+
" scores = []\n",
|
|
129
|
+
" for train_idx, val_idx in cv.split(X_train, y_train):\n",
|
|
130
|
+
" pipe.fit(X_train[train_idx], y_train[train_idx])\n",
|
|
131
|
+
" pred = pipe.predict(X_train[val_idx])\n",
|
|
132
|
+
" scores.append(f1_score(y_train[val_idx], pred, average=\"weighted\"))\n",
|
|
133
|
+
" return float(np.mean(scores))\n",
|
|
134
|
+
"\n",
|
|
135
|
+
"\n",
|
|
136
|
+
"study = optuna.create_study(direction=\"maximize\")\n",
|
|
137
|
+
"study.optimize(objective, n_trials=30, show_progress_bar=True)\n",
|
|
138
|
+
"print(\"Best trial:\", study.best_trial.number, \"F1 (CV mean):\", study.best_value)\n",
|
|
139
|
+
"print(\"Best params:\", study.best_params)"
|
|
140
|
+
]
|
|
141
|
+
},
|
|
142
|
+
{
|
|
143
|
+
"cell_type": "markdown",
|
|
144
|
+
"id": "7623eae2785240b9bd12b16a66d81610",
|
|
145
|
+
"metadata": {},
|
|
146
|
+
"source": [
|
|
147
|
+
"## 4. Fit tuned model on full training set and evaluate on held-out test"
|
|
148
|
+
]
|
|
149
|
+
},
|
|
150
|
+
{
|
|
151
|
+
"cell_type": "code",
|
|
152
|
+
"execution_count": null,
|
|
153
|
+
"id": "7cdc8c89c7104fffa095e18ddfef8986",
|
|
154
|
+
"metadata": {},
|
|
155
|
+
"outputs": [],
|
|
156
|
+
"source": [
|
|
157
|
+
"best = study.best_params.copy()\n",
|
|
158
|
+
"best[\"random_state\"] = 42\n",
|
|
159
|
+
"tuned = Pipeline(\n",
|
|
160
|
+
" [\n",
|
|
161
|
+
" (\"scale\", StandardScaler()),\n",
|
|
162
|
+
" (\"clf\", SEFRBoostClassifier(**best)),\n",
|
|
163
|
+
" ]\n",
|
|
164
|
+
")\n",
|
|
165
|
+
"tuned.fit(X_train, y_train)\n",
|
|
166
|
+
"y_pred_t = tuned.predict(X_test)\n",
|
|
167
|
+
"y_proba_t = tuned.predict_proba(X_test)[:, 1]\n",
|
|
168
|
+
"print(\"Tuned F1 (weighted):\", f1_score(y_test, y_pred_t, average=\"weighted\"))\n",
|
|
169
|
+
"print(\"Tuned ROC-AUC:\", roc_auc_score(y_test, y_proba_t))"
|
|
170
|
+
]
|
|
171
|
+
}
|
|
172
|
+
],
|
|
173
|
+
"metadata": {
|
|
174
|
+
"kernelspec": {
|
|
175
|
+
"display_name": "Python 3",
|
|
176
|
+
"language": "python",
|
|
177
|
+
"name": "python3"
|
|
178
|
+
},
|
|
179
|
+
"language_info": {
|
|
180
|
+
"name": "python",
|
|
181
|
+
"version": "3.11.0"
|
|
182
|
+
}
|
|
183
|
+
},
|
|
184
|
+
"nbformat": 4,
|
|
185
|
+
"nbformat_minor": 5
|
|
186
|
+
}
|
|
@@ -9,7 +9,7 @@ authors = [
|
|
|
9
9
|
{ name = "Hamidreza Keshavarz", email = "hamid9@outlook.com" },
|
|
10
10
|
{ name = "Reza Rawassizadeh", email = "rezar@bu.edu"},
|
|
11
11
|
]
|
|
12
|
-
description = "LinearBoost
|
|
12
|
+
description = "LinearBoost: a boosting classifier built on the linear classifier SEFR."
|
|
13
13
|
readme = "README.md"
|
|
14
14
|
keywords = [
|
|
15
15
|
"classification", "classifier", "linear", "adaboost", "boosting", "boost"
|
|
@@ -28,6 +28,7 @@ classifiers = [
|
|
|
28
28
|
]
|
|
29
29
|
requires-python = ">=3.10,<3.14"
|
|
30
30
|
dependencies = [
|
|
31
|
+
"packaging>=20.0",
|
|
31
32
|
"scikit-learn>=1.2.2",
|
|
32
33
|
"typing-extensions>=4.1.0; python_full_version < '3.11'"
|
|
33
34
|
]
|
|
@@ -38,7 +39,7 @@ dev = [
|
|
|
38
39
|
"pre-commit>=3.5.0",
|
|
39
40
|
"pytest>=7.0.0",
|
|
40
41
|
"ruff>=0.9.2",
|
|
41
|
-
"filelock>=
|
|
42
|
+
"filelock>=4.1.0", # <-- ADD THIS LINE
|
|
42
43
|
]
|
|
43
44
|
|
|
44
45
|
[project.urls]
|
|
@@ -189,7 +189,7 @@ class _DenseAdaBoostClassifier(AdaBoostClassifier):
|
|
|
189
189
|
score : generator of ndarray of shape (n_samples, k)
|
|
190
190
|
The decision function of the input samples. The order of
|
|
191
191
|
outputs is the same of that of the :term:`classes_` attribute.
|
|
192
|
-
Binary classification is a special
|
|
192
|
+
Binary classification is a special case with ``k == 1``,
|
|
193
193
|
otherwise ``k==n_classes``. For binary classification,
|
|
194
194
|
values closer to -1 or 1 mean more like the first or second
|
|
195
195
|
class in ``classes_``, respectively.
|
|
@@ -581,6 +581,7 @@ class LinearBoostClassifier(_DenseAdaBoostClassifier):
|
|
|
581
581
|
n_estimators=n_estimators,
|
|
582
582
|
learning_rate=learning_rate,
|
|
583
583
|
)
|
|
584
|
+
# AdaBoostClassifier.__init__ resets some of these, so set them again
|
|
584
585
|
self.algorithm = algorithm
|
|
585
586
|
self.scaler = scaler
|
|
586
587
|
self.class_weight = class_weight
|
|
@@ -863,7 +864,8 @@ class LinearBoostClassifier(_DenseAdaBoostClassifier):
|
|
|
863
864
|
# Store validation data (original features for kernel computation)
|
|
864
865
|
validation_data = (X_val_transformed, y_val, sample_weight_val)
|
|
865
866
|
y = y_train
|
|
866
|
-
|
|
867
|
+
if y is None:
|
|
868
|
+
raise ValueError("Target y is None after the validation split.")
|
|
867
869
|
else:
|
|
868
870
|
# For linear or approximate kernels, split after transformation
|
|
869
871
|
n_samples = training_data.shape[0]
|
|
@@ -892,7 +894,8 @@ class LinearBoostClassifier(_DenseAdaBoostClassifier):
|
|
|
892
894
|
# Store validation data for checking
|
|
893
895
|
validation_data = (training_data_val, y_val, sample_weight_val)
|
|
894
896
|
y = y_train
|
|
895
|
-
|
|
897
|
+
if y is None:
|
|
898
|
+
raise ValueError("Target y is None after the validation split.")
|
|
896
899
|
else:
|
|
897
900
|
y_train = y
|
|
898
901
|
|
|
@@ -2014,7 +2017,7 @@ class LinearBoostClassifier(_DenseAdaBoostClassifier):
|
|
|
2014
2017
|
score : ndarray of shape of (n_samples, k)
|
|
2015
2018
|
The decision function of the input samples. The order of
|
|
2016
2019
|
outputs is the same as that of the :term:`classes_` attribute.
|
|
2017
|
-
Binary classification is a special
|
|
2020
|
+
Binary classification is a special case with ``k == 1``,
|
|
2018
2021
|
otherwise ``k==n_classes``. For binary classification,
|
|
2019
2022
|
values closer to -1 or 1 mean more like the first or second
|
|
2020
2023
|
class in ``classes_``, respectively.
|
|
@@ -73,7 +73,9 @@ def _mse_leaf_value(residuals: np.ndarray, sample_weight: np.ndarray) -> float:
|
|
|
73
73
|
return float(num / den)
|
|
74
74
|
|
|
75
75
|
|
|
76
|
-
def _sanitize_sefr_hyperplane(
|
|
76
|
+
def _sanitize_sefr_hyperplane(
|
|
77
|
+
coef: np.ndarray, intercept: float
|
|
78
|
+
) -> tuple[np.ndarray, float]:
|
|
77
79
|
"""Stabilize linear split parameters from SEFR (avoid NaN/Inf and matmul overflow)."""
|
|
78
80
|
c = np.asarray(coef, dtype=np.float64).ravel()
|
|
79
81
|
c = np.nan_to_num(c, nan=0.0, posinf=0.0, neginf=0.0)
|
|
@@ -243,7 +245,9 @@ class _SEFRTree:
|
|
|
243
245
|
check_is_fitted(self, "root_")
|
|
244
246
|
n = X.shape[0]
|
|
245
247
|
out = np.zeros(n, dtype=np.float64)
|
|
246
|
-
stack: list[tuple[_SEFRTreeNode, np.ndarray]] = [
|
|
248
|
+
stack: list[tuple[_SEFRTreeNode, np.ndarray]] = [
|
|
249
|
+
(self.root_, np.ones(n, dtype=bool))
|
|
250
|
+
]
|
|
247
251
|
while stack:
|
|
248
252
|
node, mask = stack.pop()
|
|
249
253
|
if not np.any(mask):
|
|
@@ -325,7 +329,7 @@ class SEFRBoostClassifier(ClassifierMixin, BaseEstimator):
|
|
|
325
329
|
|
|
326
330
|
_parameter_constraints: dict = {
|
|
327
331
|
"n_estimators": [Interval(Integral, 1, None, closed="left")],
|
|
328
|
-
"learning_rate": [Interval(Real, 0.0, None, closed="
|
|
332
|
+
"learning_rate": [Interval(Real, 0.0, None, closed="neither")],
|
|
329
333
|
"max_depth": [Interval(Integral, 1, None, closed="left")],
|
|
330
334
|
"min_samples_leaf": [Interval(Integral, 1, None, closed="left")],
|
|
331
335
|
"min_samples_split": [Interval(Integral, 2, None, closed="left")],
|
|
@@ -383,7 +387,7 @@ class SEFRBoostClassifier(ClassifierMixin, BaseEstimator):
|
|
|
383
387
|
X, y = validate_data(
|
|
384
388
|
self,
|
|
385
389
|
X,
|
|
386
|
-
y,
|
|
390
|
+
y=y,
|
|
387
391
|
accept_sparse=False,
|
|
388
392
|
dtype=np.float64,
|
|
389
393
|
ensure_all_finite=True,
|
|
@@ -523,7 +527,7 @@ class SEFRBoostRegressor(RegressorMixin, BaseEstimator):
|
|
|
523
527
|
|
|
524
528
|
_parameter_constraints: dict = {
|
|
525
529
|
"n_estimators": [Interval(Integral, 1, None, closed="left")],
|
|
526
|
-
"learning_rate": [Interval(Real, 0.0, None, closed="
|
|
530
|
+
"learning_rate": [Interval(Real, 0.0, None, closed="neither")],
|
|
527
531
|
"max_depth": [Interval(Integral, 1, None, closed="left")],
|
|
528
532
|
"min_samples_leaf": [Interval(Integral, 1, None, closed="left")],
|
|
529
533
|
"min_samples_split": [Interval(Integral, 2, None, closed="left")],
|
|
@@ -577,7 +581,7 @@ class SEFRBoostRegressor(RegressorMixin, BaseEstimator):
|
|
|
577
581
|
X, y = validate_data(
|
|
578
582
|
self,
|
|
579
583
|
X,
|
|
580
|
-
y,
|
|
584
|
+
y=y,
|
|
581
585
|
accept_sparse=False,
|
|
582
586
|
dtype=np.float64,
|
|
583
587
|
ensure_all_finite=True,
|
|
@@ -1,6 +1,10 @@
|
|
|
1
1
|
version = 1
|
|
2
2
|
revision = 2
|
|
3
3
|
requires-python = ">=3.10, <3.14"
|
|
4
|
+
resolution-markers = [
|
|
5
|
+
"python_full_version >= '3.11'",
|
|
6
|
+
"python_full_version < '3.11'",
|
|
7
|
+
]
|
|
4
8
|
|
|
5
9
|
[[package]]
|
|
6
10
|
name = "cfgv"
|
|
@@ -22,11 +26,11 @@ wheels = [
|
|
|
22
26
|
|
|
23
27
|
[[package]]
|
|
24
28
|
name = "distlib"
|
|
25
|
-
version = "0.
|
|
29
|
+
version = "0.4.4"
|
|
26
30
|
source = { registry = "https://pypi.org/simple" }
|
|
27
|
-
sdist = { url = "https://files.pythonhosted.org/packages/
|
|
31
|
+
sdist = { url = "https://files.pythonhosted.org/packages/68/bc/d36966de7304047a1001a7559fbf266c464bb5d8cc85464536ef8c9bcc32/distlib-0.4.4.tar.gz", hash = "sha256:ca4175b4ac7f71223cc30ed7248c45d2b38c2439d8ee6e498ab67ed63cf3508d", size = 621304, upload-time = "2026-10-16T13:56:34.203Z" }
|
|
28
32
|
wheels = [
|
|
29
|
-
{ url = "https://files.pythonhosted.org/packages/
|
|
33
|
+
{ url = "https://files.pythonhosted.org/packages/c1/2b/238f02a4927fa2e27aa995371d240a468b52e42eae44a678434aff736b38/distlib-0.4.4-py2.py3-none-any.whl", hash = "sha256:cf09d55b1e85a89576b036aeb71550bfd68320f3f21761ca329eadadad918638", size = 472099, upload-time = "2026-10-16T13:56:32.053Z" },
|
|
30
34
|
]
|
|
31
35
|
|
|
32
36
|
[[package]]
|
|
@@ -43,11 +47,11 @@ wheels = [
|
|
|
43
47
|
|
|
44
48
|
[[package]]
|
|
45
49
|
name = "filelock"
|
|
46
|
-
version = "
|
|
50
|
+
version = "4.1.0"
|
|
47
51
|
source = { registry = "https://pypi.org/simple" }
|
|
48
|
-
sdist = { url = "https://files.pythonhosted.org/packages/
|
|
52
|
+
sdist = { url = "https://files.pythonhosted.org/packages/4c/58/6fd434bec86eff7c38a3168454cb132b762b2bea9b3ac094101a2f7bc32a/filelock-4.1.0.tar.gz", hash = "sha256:ad7f724afef953e731b1cc39bcd3a09166d72ed7fcdf29e6e88b1c3235c6715d", size = 561277, upload-time = "2026-10-09T19:57:20.34Z" }
|
|
49
53
|
wheels = [
|
|
50
|
-
{ url = "https://files.pythonhosted.org/packages/
|
|
54
|
+
{ url = "https://files.pythonhosted.org/packages/ee/86/032133892a5de43b5a98200b01aadcad68cc255e274a762f08b8a76d2912/filelock-4.1.0-py3-none-any.whl", hash = "sha256:2ce9818e3e2d8f284c1a964414447ef148d42a5fd5e2a477a7118e574b293ec1", size = 133003, upload-time = "2026-10-09T19:57:18.716Z" },
|
|
51
55
|
]
|
|
52
56
|
|
|
53
57
|
[[package]]
|
|
@@ -90,6 +94,7 @@ wheels = [
|
|
|
90
94
|
name = "linearboost"
|
|
91
95
|
source = { editable = "." }
|
|
92
96
|
dependencies = [
|
|
97
|
+
{ name = "packaging" },
|
|
93
98
|
{ name = "scikit-learn" },
|
|
94
99
|
{ name = "typing-extensions", marker = "python_full_version < '3.11'" },
|
|
95
100
|
]
|
|
@@ -105,13 +110,14 @@ dev = [
|
|
|
105
110
|
|
|
106
111
|
[package.metadata]
|
|
107
112
|
requires-dist = [
|
|
113
|
+
{ name = "packaging", specifier = ">=20.0" },
|
|
108
114
|
{ name = "scikit-learn", specifier = ">=1.2.2" },
|
|
109
115
|
{ name = "typing-extensions", marker = "python_full_version < '3.11'", specifier = ">=4.1.0" },
|
|
110
116
|
]
|
|
111
117
|
|
|
112
118
|
[package.metadata.requires-dev]
|
|
113
119
|
dev = [
|
|
114
|
-
{ name = "filelock", specifier = ">=
|
|
120
|
+
{ name = "filelock", specifier = ">=4.1.0" },
|
|
115
121
|
{ name = "isort" },
|
|
116
122
|
{ name = "pre-commit", specifier = ">=3.5.0" },
|
|
117
123
|
{ name = "pytest", specifier = ">=7.0.0" },
|
|
@@ -191,20 +197,35 @@ wheels = [
|
|
|
191
197
|
|
|
192
198
|
[[package]]
|
|
193
199
|
name = "packaging"
|
|
194
|
-
version = "
|
|
200
|
+
version = "26.3"
|
|
201
|
+
source = { registry = "https://pypi.org/simple" }
|
|
202
|
+
sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" }
|
|
203
|
+
wheels = [
|
|
204
|
+
{ url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" },
|
|
205
|
+
]
|
|
206
|
+
|
|
207
|
+
[[package]]
|
|
208
|
+
name = "platformdirs"
|
|
209
|
+
version = "4.12.4"
|
|
195
210
|
source = { registry = "https://pypi.org/simple" }
|
|
196
|
-
|
|
211
|
+
resolution-markers = [
|
|
212
|
+
"python_full_version < '3.11'",
|
|
213
|
+
]
|
|
214
|
+
sdist = { url = "https://files.pythonhosted.org/packages/90/a1/d5f9002a70298c64a789779077d8dd90c10aa1f47fe40c86802df874f2a6/platformdirs-4.12.4.tar.gz", hash = "sha256:63743c02414e755de4e31b8f68125c1407495b86c5a006e203c01ff8b9924250", size = 60010, upload-time = "2026-10-07T23:30:32.426Z" }
|
|
197
215
|
wheels = [
|
|
198
|
-
{ url = "https://files.pythonhosted.org/packages/
|
|
216
|
+
{ url = "https://files.pythonhosted.org/packages/f4/ba/223e00b885e960edd5d4b4d178c88acce019bd5b83786093681b8c393492/platformdirs-4.12.4-py3-none-any.whl", hash = "sha256:78bfb9db2a8471ed7eebe3c3c932da413911042994e699b384fbb4493fa872d7", size = 32596, upload-time = "2026-10-07T23:30:30.825Z" },
|
|
199
217
|
]
|
|
200
218
|
|
|
201
219
|
[[package]]
|
|
202
220
|
name = "platformdirs"
|
|
203
|
-
version = "4.
|
|
221
|
+
version = "4.14.8"
|
|
204
222
|
source = { registry = "https://pypi.org/simple" }
|
|
205
|
-
|
|
223
|
+
resolution-markers = [
|
|
224
|
+
"python_full_version >= '3.11'",
|
|
225
|
+
]
|
|
226
|
+
sdist = { url = "https://files.pythonhosted.org/packages/71/18/0b0d99f5bf164f04a7a9534c3dbdbe4d5fd2b382667685ad40c192c4d3d1/platformdirs-4.14.8.tar.gz", hash = "sha256:96893d686e84e9e6f358a9ea41a8de544df66dbf2ba6cda250e7dc73acf85b9b", size = 87203, upload-time = "2026-11-20T08:14:52.63Z" }
|
|
206
227
|
wheels = [
|
|
207
|
-
{ url = "https://files.pythonhosted.org/packages/
|
|
228
|
+
{ url = "https://files.pythonhosted.org/packages/5b/be/373cc7341f53bfd3acba3e5db62433fd960a45fcf73ad64a5f4d27e74910/platformdirs-4.14.8-py3-none-any.whl", hash = "sha256:1e34af590de4d978631b751eb2202f2fee4497537a17c1720b12fcf57b331392", size = 94267, upload-time = "2026-11-20T08:14:51.306Z" },
|
|
208
229
|
]
|
|
209
230
|
|
|
210
231
|
[[package]]
|
|
@@ -259,6 +280,18 @@ wheels = [
|
|
|
259
280
|
{ url = "https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl", hash = "sha256:2c5efc453d45394fdd706ade797c0a81091eccd1d6e4bccfcd476e2b8e0ab5d9", size = 375249, upload-time = "2026-04-07T17:16:16.13Z" },
|
|
260
281
|
]
|
|
261
282
|
|
|
283
|
+
[[package]]
|
|
284
|
+
name = "python-discovery"
|
|
285
|
+
version = "1.6.1"
|
|
286
|
+
source = { registry = "https://pypi.org/simple" }
|
|
287
|
+
dependencies = [
|
|
288
|
+
{ name = "filelock" },
|
|
289
|
+
]
|
|
290
|
+
sdist = { url = "https://files.pythonhosted.org/packages/0c/57/250bd238b966cece44328235eb85290045d059265fdaf7527a3a958123db/python_discovery-1.6.1.tar.gz", hash = "sha256:cf87d3627dfb4412437fdd5b13eae402607722998d21567993aedbc59b23c15e", size = 84338, upload-time = "2026-09-18T01:31:53.971Z" }
|
|
291
|
+
wheels = [
|
|
292
|
+
{ url = "https://files.pythonhosted.org/packages/16/7d/e9ffbadfbf89c93848412d04594135c4ae8c1d37d9e053b9c3ed718fabc4/python_discovery-1.6.1-py3-none-any.whl", hash = "sha256:d43fcdef879fe795352bd13ccf8d185ba5a9f86f36cfcd00529f596e737442b3", size = 38664, upload-time = "2026-09-18T01:31:52.448Z" },
|
|
293
|
+
]
|
|
294
|
+
|
|
262
295
|
[[package]]
|
|
263
296
|
name = "pyyaml"
|
|
264
297
|
version = "6.0.2"
|
|
@@ -463,24 +496,27 @@ wheels = [
|
|
|
463
496
|
|
|
464
497
|
[[package]]
|
|
465
498
|
name = "typing-extensions"
|
|
466
|
-
version = "4.
|
|
499
|
+
version = "4.16.0"
|
|
467
500
|
source = { registry = "https://pypi.org/simple" }
|
|
468
|
-
sdist = { url = "https://files.pythonhosted.org/packages/
|
|
501
|
+
sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" }
|
|
469
502
|
wheels = [
|
|
470
|
-
{ url = "https://files.pythonhosted.org/packages/
|
|
503
|
+
{ url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" },
|
|
471
504
|
]
|
|
472
505
|
|
|
473
506
|
[[package]]
|
|
474
507
|
name = "virtualenv"
|
|
475
|
-
version = "
|
|
508
|
+
version = "21.14.13"
|
|
476
509
|
source = { registry = "https://pypi.org/simple" }
|
|
477
510
|
dependencies = [
|
|
478
511
|
{ name = "distlib" },
|
|
479
512
|
{ name = "filelock" },
|
|
480
|
-
{ name = "
|
|
513
|
+
{ name = "packaging" },
|
|
514
|
+
{ name = "platformdirs", version = "4.12.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
|
|
515
|
+
{ name = "platformdirs", version = "4.14.8", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
|
|
516
|
+
{ name = "python-discovery" },
|
|
481
517
|
{ name = "typing-extensions", marker = "python_full_version < '3.11'" },
|
|
482
518
|
]
|
|
483
|
-
sdist = { url = "https://files.pythonhosted.org/packages/aa/
|
|
519
|
+
sdist = { url = "https://files.pythonhosted.org/packages/70/aa/bff50daee0488e16d611258a9b6c5a1d0c9e128f8ca424e9f4e1c0444ef2/virtualenv-21.14.13.tar.gz", hash = "sha256:de5307ba4b9a0001d9c8ea02832d38f63ca432880f416c1bface61c4323fd905", size = 5570301, upload-time = "2026-10-22T20:10:16.952Z" }
|
|
484
520
|
wheels = [
|
|
485
|
-
{ url = "https://files.pythonhosted.org/packages/
|
|
521
|
+
{ url = "https://files.pythonhosted.org/packages/bb/a9/c10d8a44a29fb94a18b43fdf11f38caca53ae7c9406f11a72b1cb8f624f1/virtualenv-21.14.13-py3-none-any.whl", hash = "sha256:5fbb04138a4861580b78ba9110d529a26bc48fcc93c8ec36bc0202eeed96a788", size = 5557094, upload-time = "2026-10-22T20:10:15.096Z" },
|
|
486
522
|
]
|
|
@@ -1,208 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"cells": [
|
|
3
|
-
{
|
|
4
|
-
"cell_type": "markdown",
|
|
5
|
-
"metadata": {},
|
|
6
|
-
"source": [
|
|
7
|
-
"# SEFRBoost usage demo\n",
|
|
8
|
-
"\n",
|
|
9
|
-
"**SEFRBoost** fits shallow gradient-boosted trees whose internal splits are **linear hyperplanes trained with [SEFR](https://github.com/LinearBoost/linearboost-classifier)** on pseudo-residuals—an oblique-split boosting variant shipped as `SEFRBoostClassifier` and `SEFRBoostRegressor`.\n",
|
|
10
|
-
"\n",
|
|
11
|
-
"This notebook mirrors the workflow of [`demo_linearboost_usage.ipynb`](demo_linearboost_usage.ipynb), but uses only **NumPy**, **scikit-learn** datasets, and **linearboost** (no Optuna / UCI helpers). Scaling is recommended; here we wrap models in a `Pipeline` with `StandardScaler`.\n",
|
|
12
|
-
"\n",
|
|
13
|
-
"**Binary classification only** for `SEFRBoostClassifier`. `SEFRBoostRegressor` supports single-target regression."
|
|
14
|
-
]
|
|
15
|
-
},
|
|
16
|
-
{
|
|
17
|
-
"cell_type": "code",
|
|
18
|
-
"execution_count": null,
|
|
19
|
-
"metadata": {},
|
|
20
|
-
"outputs": [],
|
|
21
|
-
"source": [
|
|
22
|
-
"import warnings\n",
|
|
23
|
-
"\n",
|
|
24
|
-
"import numpy as np\n",
|
|
25
|
-
"\n",
|
|
26
|
-
"warnings.filterwarnings(\"ignore\", category=RuntimeWarning)\n",
|
|
27
|
-
"warnings.filterwarnings(\"ignore\", category=FutureWarning)\n",
|
|
28
|
-
"\n",
|
|
29
|
-
"from linearboost import SEFRBoostClassifier, SEFRBoostRegressor"
|
|
30
|
-
]
|
|
31
|
-
},
|
|
32
|
-
{
|
|
33
|
-
"cell_type": "markdown",
|
|
34
|
-
"metadata": {},
|
|
35
|
-
"source": [
|
|
36
|
-
"## Binary classification · Breast Cancer Wisconsin\n",
|
|
37
|
-
"\n",
|
|
38
|
-
"Load data, hold out a test set, fit a pipeline, and report F1 and a short classification report."
|
|
39
|
-
]
|
|
40
|
-
},
|
|
41
|
-
{
|
|
42
|
-
"cell_type": "code",
|
|
43
|
-
"execution_count": null,
|
|
44
|
-
"metadata": {},
|
|
45
|
-
"outputs": [],
|
|
46
|
-
"source": [
|
|
47
|
-
"from sklearn.datasets import load_breast_cancer\n",
|
|
48
|
-
"from sklearn.metrics import classification_report, f1_score\n",
|
|
49
|
-
"from sklearn.model_selection import train_test_split\n",
|
|
50
|
-
"from sklearn.pipeline import Pipeline\n",
|
|
51
|
-
"from sklearn.preprocessing import StandardScaler\n",
|
|
52
|
-
"\n",
|
|
53
|
-
"X, y = load_breast_cancer(return_X_y=True)\n",
|
|
54
|
-
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
|
55
|
-
" X,\n",
|
|
56
|
-
" y,\n",
|
|
57
|
-
" test_size=0.25,\n",
|
|
58
|
-
" random_state=0,\n",
|
|
59
|
-
" stratify=y,\n",
|
|
60
|
-
")\n",
|
|
61
|
-
"\n",
|
|
62
|
-
"clf = Pipeline(\n",
|
|
63
|
-
" steps=[\n",
|
|
64
|
-
" (\"scaler\", StandardScaler()),\n",
|
|
65
|
-
" (\n",
|
|
66
|
-
" \"model\",\n",
|
|
67
|
-
" SEFRBoostClassifier(\n",
|
|
68
|
-
" n_estimators=100,\n",
|
|
69
|
-
" learning_rate=0.1,\n",
|
|
70
|
-
" max_depth=3,\n",
|
|
71
|
-
" min_samples_leaf=10,\n",
|
|
72
|
-
" random_state=0,\n",
|
|
73
|
-
" ),\n",
|
|
74
|
-
" ),\n",
|
|
75
|
-
" ]\n",
|
|
76
|
-
")\n",
|
|
77
|
-
"\n",
|
|
78
|
-
"clf.fit(X_train, y_train)\n",
|
|
79
|
-
"y_pred = clf.predict(X_test)\n",
|
|
80
|
-
"\n",
|
|
81
|
-
"print(\"F1 (test):\", f1_score(y_test, y_pred))\n",
|
|
82
|
-
"print()\n",
|
|
83
|
-
"print(classification_report(y_test, y_pred, digits=4))"
|
|
84
|
-
]
|
|
85
|
-
},
|
|
86
|
-
{
|
|
87
|
-
"cell_type": "markdown",
|
|
88
|
-
"metadata": {},
|
|
89
|
-
"source": [
|
|
90
|
-
"## Stratified cross-validation\n",
|
|
91
|
-
"\n",
|
|
92
|
-
"Quick sanity check with a few folds (adjust `n_estimators` / `cv` if this cell is slow)."
|
|
93
|
-
]
|
|
94
|
-
},
|
|
95
|
-
{
|
|
96
|
-
"cell_type": "code",
|
|
97
|
-
"execution_count": null,
|
|
98
|
-
"metadata": {},
|
|
99
|
-
"outputs": [],
|
|
100
|
-
"source": [
|
|
101
|
-
"from sklearn.model_selection import StratifiedKFold, cross_val_score\n",
|
|
102
|
-
"\n",
|
|
103
|
-
"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)\n",
|
|
104
|
-
"pipe = Pipeline(\n",
|
|
105
|
-
" steps=[\n",
|
|
106
|
-
" (\"scaler\", StandardScaler()),\n",
|
|
107
|
-
" (\n",
|
|
108
|
-
" \"model\",\n",
|
|
109
|
-
" SEFRBoostClassifier(\n",
|
|
110
|
-
" n_estimators=80,\n",
|
|
111
|
-
" learning_rate=0.1,\n",
|
|
112
|
-
" max_depth=3,\n",
|
|
113
|
-
" random_state=0,\n",
|
|
114
|
-
" ),\n",
|
|
115
|
-
" ),\n",
|
|
116
|
-
" ]\n",
|
|
117
|
-
")\n",
|
|
118
|
-
"\n",
|
|
119
|
-
"scores = cross_val_score(pipe, X, y, cv=cv, scoring=\"f1\")\n",
|
|
120
|
-
"print(\"F1 per fold:\", scores)\n",
|
|
121
|
-
"print(\"Mean F1:\", float(np.mean(scores)), \"±\", float(np.std(scores)))"
|
|
122
|
-
]
|
|
123
|
-
},
|
|
124
|
-
{
|
|
125
|
-
"cell_type": "markdown",
|
|
126
|
-
"metadata": {},
|
|
127
|
-
"source": [
|
|
128
|
-
"## Regression · Diabetes\n",
|
|
129
|
-
"\n",
|
|
130
|
-
"`SEFRBoostRegressor` uses squared-error boosting with the same oblique SEFR splits at internal nodes."
|
|
131
|
-
]
|
|
132
|
-
},
|
|
133
|
-
{
|
|
134
|
-
"cell_type": "code",
|
|
135
|
-
"execution_count": null,
|
|
136
|
-
"metadata": {},
|
|
137
|
-
"outputs": [],
|
|
138
|
-
"source": [
|
|
139
|
-
"from sklearn.datasets import load_diabetes\n",
|
|
140
|
-
"from sklearn.metrics import mean_squared_error, r2_score\n",
|
|
141
|
-
"\n",
|
|
142
|
-
"Xr, yr = load_diabetes(return_X_y=True)\n",
|
|
143
|
-
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
|
144
|
-
" Xr,\n",
|
|
145
|
-
" yr,\n",
|
|
146
|
-
" test_size=0.25,\n",
|
|
147
|
-
" random_state=0,\n",
|
|
148
|
-
")\n",
|
|
149
|
-
"\n",
|
|
150
|
-
"reg = Pipeline(\n",
|
|
151
|
-
" steps=[\n",
|
|
152
|
-
" (\"scaler\", StandardScaler()),\n",
|
|
153
|
-
" (\n",
|
|
154
|
-
" \"model\",\n",
|
|
155
|
-
" SEFRBoostRegressor(\n",
|
|
156
|
-
" n_estimators=100,\n",
|
|
157
|
-
" learning_rate=0.1,\n",
|
|
158
|
-
" max_depth=3,\n",
|
|
159
|
-
" min_samples_leaf=10,\n",
|
|
160
|
-
" random_state=0,\n",
|
|
161
|
-
" ),\n",
|
|
162
|
-
" ),\n",
|
|
163
|
-
" ]\n",
|
|
164
|
-
")\n",
|
|
165
|
-
"\n",
|
|
166
|
-
"reg.fit(X_train, y_train)\n",
|
|
167
|
-
"y_pred = reg.predict(X_test)\n",
|
|
168
|
-
"\n",
|
|
169
|
-
"print(\"R² (test):\", r2_score(y_test, y_pred))\n",
|
|
170
|
-
"print(\"RMSE (test):\", float(np.sqrt(mean_squared_error(y_test, y_pred))))"
|
|
171
|
-
]
|
|
172
|
-
},
|
|
173
|
-
{
|
|
174
|
-
"cell_type": "markdown",
|
|
175
|
-
"metadata": {},
|
|
176
|
-
"source": [
|
|
177
|
-
"## Optional knobs\n",
|
|
178
|
-
"\n",
|
|
179
|
-
"- **`class_weight`** / **`scale_pos_weight`** (classifier): same spirit as scikit-learn / XGBoost-style weighting—see the docstrings in `linearboost/sefr_boost.py`.\n",
|
|
180
|
-
"- **`subsample`**: stochastic boosting (fraction of rows per tree).\n",
|
|
181
|
-
"- **`max_depth`**, **`min_samples_leaf`**: control tree capacity and stability on small folds.\n",
|
|
182
|
-
"\n",
|
|
183
|
-
"For AdaBoost-style boosting with kernels on the base learner, see **`LinearBoostClassifier`** in [`demo_linearboost_usage.ipynb`](demo_linearboost_usage.ipynb)."
|
|
184
|
-
]
|
|
185
|
-
}
|
|
186
|
-
],
|
|
187
|
-
"metadata": {
|
|
188
|
-
"kernelspec": {
|
|
189
|
-
"display_name": "Python 3",
|
|
190
|
-
"language": "python",
|
|
191
|
-
"name": "python3"
|
|
192
|
-
},
|
|
193
|
-
"language_info": {
|
|
194
|
-
"codemirror_mode": {
|
|
195
|
-
"name": "ipython",
|
|
196
|
-
"version": 3
|
|
197
|
-
},
|
|
198
|
-
"file_extension": ".py",
|
|
199
|
-
"mimetype": "text/x-python",
|
|
200
|
-
"name": "python",
|
|
201
|
-
"nbconvert_exporter": "python",
|
|
202
|
-
"pygments_lexer": "ipython3",
|
|
203
|
-
"version": "3.12.0"
|
|
204
|
-
}
|
|
205
|
-
},
|
|
206
|
-
"nbformat": 4,
|
|
207
|
-
"nbformat_minor": 2
|
|
208
|
-
}
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|