smallgbm 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- smallgbm-1.0.0/LICENSE +21 -0
- smallgbm-1.0.0/MANIFEST.in +2 -0
- smallgbm-1.0.0/PKG-INFO +147 -0
- smallgbm-1.0.0/README.md +119 -0
- smallgbm-1.0.0/setup.cfg +4 -0
- smallgbm-1.0.0/setup.py +28 -0
- smallgbm-1.0.0/smallgbm/__init__.py +2 -0
- smallgbm-1.0.0/smallgbm/smallgbm.py +170 -0
- smallgbm-1.0.0/smallgbm/tree.py +116 -0
- smallgbm-1.0.0/smallgbm.egg-info/PKG-INFO +147 -0
- smallgbm-1.0.0/smallgbm.egg-info/SOURCES.txt +12 -0
- smallgbm-1.0.0/smallgbm.egg-info/dependency_links.txt +1 -0
- smallgbm-1.0.0/smallgbm.egg-info/requires.txt +2 -0
- smallgbm-1.0.0/smallgbm.egg-info/top_level.txt +1 -0
smallgbm-1.0.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Emelyanov Ilya
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
smallgbm-1.0.0/PKG-INFO
ADDED
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: smallgbm
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Gradient boosting optimized for small datasets (n < 1000)
|
|
5
|
+
Home-page: https://github.com/nsdmlk/SmallGBM
|
|
6
|
+
Author: Emelyanov Ilya
|
|
7
|
+
Author-email: Nsdmlk@yandex.ru
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Requires-Python: >=3.8
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
License-File: LICENSE
|
|
16
|
+
Requires-Dist: numpy>=1.20.0
|
|
17
|
+
Requires-Dist: scikit-learn>=1.0.0
|
|
18
|
+
Dynamic: author
|
|
19
|
+
Dynamic: author-email
|
|
20
|
+
Dynamic: classifier
|
|
21
|
+
Dynamic: description
|
|
22
|
+
Dynamic: description-content-type
|
|
23
|
+
Dynamic: home-page
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
Dynamic: requires-dist
|
|
26
|
+
Dynamic: requires-python
|
|
27
|
+
Dynamic: summary
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
# SmallGBM
|
|
31
|
+
|
|
32
|
+
<p align="center">
|
|
33
|
+
<b>Gradient boosting that actually works on small data.</b>
|
|
34
|
+
</p>
|
|
35
|
+
|
|
36
|
+
<p align="center">
|
|
37
|
+
<img src="https://img.shields.io/badge/version-1.0.0-blue" alt="version">
|
|
38
|
+
<img src="https://img.shields.io/badge/python-3.8+-green" alt="python">
|
|
39
|
+
<img src="https://img.shields.io/badge/license-MIT-brightgreen" alt="license">
|
|
40
|
+
<img src="https://img.shields.io/badge/pip%20install-smallgbm-orange" alt="pip">
|
|
41
|
+
</p>
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Why SmallGBM?
|
|
46
|
+
|
|
47
|
+
XGBoost and LightGBM are built for scale. They shine on thousands of rows. But when you only have **50, 100, or 500 samples**, their default hyperparameters fail — overfitting, instability, unpredictable results.
|
|
48
|
+
|
|
49
|
+
**SmallGBM** is designed from the ground up for datasets with fewer than 1000 samples.
|
|
50
|
+
|
|
51
|
+
| Feature | SmallGBM | XGBoost | LightGBM |
|
|
52
|
+
| ---------------------------- | -------- | ------- | -------- |
|
|
53
|
+
| Bayesian leaf weights | ✅ | ❌ | ❌ |
|
|
54
|
+
| Adaptive regularization | ✅ | ❌ | ❌ |
|
|
55
|
+
| No bootstrap (uses all data) | ✅ | ❌ | ❌ |
|
|
56
|
+
| Stable under label noise | ✅ | ❌ | ❌ |
|
|
57
|
+
| scikit-learn compatible | ✅ | ✅ | ✅ |
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Noise Stability
|
|
62
|
+
|
|
63
|
+
SmallGBM degrades gracefully when labels are noisy — unlike XGBoost and LightGBM which drop sharply.
|
|
64
|
+
|
|
65
|
+
<p align="center">
|
|
66
|
+
<img src="docs/noise_comparison.png" width="600" alt="Noise stability comparison">
|
|
67
|
+
</p>
|
|
68
|
+
|
|
69
|
+
*At 20% label noise, SmallGBM is the best performer. Bayesian regularization keeps it stable when others collapse.*
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Learning Curve
|
|
74
|
+
|
|
75
|
+
Clear, predictable improvement as data grows. Reliable performance starts at **n ≈ 40**.
|
|
76
|
+
|
|
77
|
+
<p align="center">
|
|
78
|
+
<img src="docs/learning_curve.png" width="600" alt="Learning curve">
|
|
79
|
+
</p>
|
|
80
|
+
|
|
81
|
+
*No sudden jumps, no catastrophic failures. A safe choice when data is limited.*
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Installation
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
pip install smallgbm
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
## Quickstart
|
|
92
|
+
|
|
93
|
+
```python
|
|
94
|
+
from smallgbm import SmallGBMClassifier
|
|
95
|
+
|
|
96
|
+
model = SmallGBMClassifier()
|
|
97
|
+
model.fit(X_train, y_train)
|
|
98
|
+
proba = model.predict_proba(X_test)
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## API
|
|
102
|
+
|
|
103
|
+
### SmallGBMClassifier
|
|
104
|
+
|
|
105
|
+
| Parameter | Default | Description |
|
|
106
|
+
| -------------------- | ------- | ------------------------- |
|
|
107
|
+
| `n_estimators` | 50 | Number of boosting rounds |
|
|
108
|
+
| `max_depth` | 3 | Maximum tree depth |
|
|
109
|
+
| `min_samples_leaf` | 3 | Minimum samples per leaf |
|
|
110
|
+
| `learning_rate` | 0.1 | Shrinkage factor |
|
|
111
|
+
| `sigma_prior` | 0.5 | Bayesian prior strength |
|
|
112
|
+
|
|
113
|
+
### SmallGBMRegressor
|
|
114
|
+
|
|
115
|
+
Same parameters, for regression tasks.
|
|
116
|
+
|
|
117
|
+
```python
|
|
118
|
+
from smallgbm import SmallGBMRegressor
|
|
119
|
+
|
|
120
|
+
model = SmallGBMRegressor()
|
|
121
|
+
model.fit(X_train, y_train)
|
|
122
|
+
preds = model.predict(X_test)
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
## Research
|
|
128
|
+
|
|
129
|
+
Full characterisation notebook with 5 experiments: `benchmark_final.ipynb`
|
|
130
|
+
|
|
131
|
+
- Noise stability analysis
|
|
132
|
+
- Prior sensitivity
|
|
133
|
+
- Sample size curve
|
|
134
|
+
- Regression performance
|
|
135
|
+
- Class imbalance tolerance
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## License
|
|
140
|
+
|
|
141
|
+
MIT © [Emelyanov Ilya](https://github.com/nsdmlk) 2026
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
<p align="center">
|
|
146
|
+
<sub>Built with ❤️ for the small data community</sub>
|
|
147
|
+
</p>
|
smallgbm-1.0.0/README.md
ADDED
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
|
|
2
|
+
# SmallGBM
|
|
3
|
+
|
|
4
|
+
<p align="center">
|
|
5
|
+
<b>Gradient boosting that actually works on small data.</b>
|
|
6
|
+
</p>
|
|
7
|
+
|
|
8
|
+
<p align="center">
|
|
9
|
+
<img src="https://img.shields.io/badge/version-1.0.0-blue" alt="version">
|
|
10
|
+
<img src="https://img.shields.io/badge/python-3.8+-green" alt="python">
|
|
11
|
+
<img src="https://img.shields.io/badge/license-MIT-brightgreen" alt="license">
|
|
12
|
+
<img src="https://img.shields.io/badge/pip%20install-smallgbm-orange" alt="pip">
|
|
13
|
+
</p>
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## Why SmallGBM?
|
|
18
|
+
|
|
19
|
+
XGBoost and LightGBM are built for scale. They shine on thousands of rows. But when you only have **50, 100, or 500 samples**, their default hyperparameters fail — overfitting, instability, unpredictable results.
|
|
20
|
+
|
|
21
|
+
**SmallGBM** is designed from the ground up for datasets with fewer than 1000 samples.
|
|
22
|
+
|
|
23
|
+
| Feature | SmallGBM | XGBoost | LightGBM |
|
|
24
|
+
| ---------------------------- | -------- | ------- | -------- |
|
|
25
|
+
| Bayesian leaf weights | ✅ | ❌ | ❌ |
|
|
26
|
+
| Adaptive regularization | ✅ | ❌ | ❌ |
|
|
27
|
+
| No bootstrap (uses all data) | ✅ | ❌ | ❌ |
|
|
28
|
+
| Stable under label noise | ✅ | ❌ | ❌ |
|
|
29
|
+
| scikit-learn compatible | ✅ | ✅ | ✅ |
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Noise Stability
|
|
34
|
+
|
|
35
|
+
SmallGBM degrades gracefully when labels are noisy — unlike XGBoost and LightGBM which drop sharply.
|
|
36
|
+
|
|
37
|
+
<p align="center">
|
|
38
|
+
<img src="docs/noise_comparison.png" width="600" alt="Noise stability comparison">
|
|
39
|
+
</p>
|
|
40
|
+
|
|
41
|
+
*At 20% label noise, SmallGBM is the best performer. Bayesian regularization keeps it stable when others collapse.*
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Learning Curve
|
|
46
|
+
|
|
47
|
+
Clear, predictable improvement as data grows. Reliable performance starts at **n ≈ 40**.
|
|
48
|
+
|
|
49
|
+
<p align="center">
|
|
50
|
+
<img src="docs/learning_curve.png" width="600" alt="Learning curve">
|
|
51
|
+
</p>
|
|
52
|
+
|
|
53
|
+
*No sudden jumps, no catastrophic failures. A safe choice when data is limited.*
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
## Installation
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
pip install smallgbm
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## Quickstart
|
|
64
|
+
|
|
65
|
+
```python
|
|
66
|
+
from smallgbm import SmallGBMClassifier
|
|
67
|
+
|
|
68
|
+
model = SmallGBMClassifier()
|
|
69
|
+
model.fit(X_train, y_train)
|
|
70
|
+
proba = model.predict_proba(X_test)
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
## API
|
|
74
|
+
|
|
75
|
+
### SmallGBMClassifier
|
|
76
|
+
|
|
77
|
+
| Parameter | Default | Description |
|
|
78
|
+
| -------------------- | ------- | ------------------------- |
|
|
79
|
+
| `n_estimators` | 50 | Number of boosting rounds |
|
|
80
|
+
| `max_depth` | 3 | Maximum tree depth |
|
|
81
|
+
| `min_samples_leaf` | 3 | Minimum samples per leaf |
|
|
82
|
+
| `learning_rate` | 0.1 | Shrinkage factor |
|
|
83
|
+
| `sigma_prior` | 0.5 | Bayesian prior strength |
|
|
84
|
+
|
|
85
|
+
### SmallGBMRegressor
|
|
86
|
+
|
|
87
|
+
Same parameters, for regression tasks.
|
|
88
|
+
|
|
89
|
+
```python
|
|
90
|
+
from smallgbm import SmallGBMRegressor
|
|
91
|
+
|
|
92
|
+
model = SmallGBMRegressor()
|
|
93
|
+
model.fit(X_train, y_train)
|
|
94
|
+
preds = model.predict(X_test)
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## Research
|
|
100
|
+
|
|
101
|
+
Full characterisation notebook with 5 experiments: `benchmark_final.ipynb`
|
|
102
|
+
|
|
103
|
+
- Noise stability analysis
|
|
104
|
+
- Prior sensitivity
|
|
105
|
+
- Sample size curve
|
|
106
|
+
- Regression performance
|
|
107
|
+
- Class imbalance tolerance
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## License
|
|
112
|
+
|
|
113
|
+
MIT © [Emelyanov Ilya](https://github.com/nsdmlk) 2026
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
<p align="center">
|
|
118
|
+
<sub>Built with ❤️ for the small data community</sub>
|
|
119
|
+
</p>
|
smallgbm-1.0.0/setup.cfg
ADDED
smallgbm-1.0.0/setup.py
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
from setuptools import setup, find_packages
|
|
2
|
+
|
|
3
|
+
with open("README.md", "r", encoding="utf-8") as fh:
|
|
4
|
+
long_description = fh.read()
|
|
5
|
+
|
|
6
|
+
setup(
|
|
7
|
+
name="smallgbm",
|
|
8
|
+
version="1.0.0",
|
|
9
|
+
author="Emelyanov Ilya",
|
|
10
|
+
author_email="Nsdmlk@yandex.ru",
|
|
11
|
+
description="Gradient boosting optimized for small datasets (n < 1000)",
|
|
12
|
+
long_description=long_description,
|
|
13
|
+
long_description_content_type="text/markdown",
|
|
14
|
+
url="https://github.com/nsdmlk/SmallGBM",
|
|
15
|
+
packages=find_packages(),
|
|
16
|
+
classifiers=[
|
|
17
|
+
"Programming Language :: Python :: 3",
|
|
18
|
+
"License :: OSI Approved :: MIT License",
|
|
19
|
+
"Operating System :: OS Independent",
|
|
20
|
+
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
|
21
|
+
"Intended Audience :: Science/Research",
|
|
22
|
+
],
|
|
23
|
+
python_requires=">=3.8",
|
|
24
|
+
install_requires=[
|
|
25
|
+
"numpy>=1.20.0",
|
|
26
|
+
"scikit-learn>=1.0.0",
|
|
27
|
+
],
|
|
28
|
+
)
|
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
import numpy as np
|
|
2
|
+
from sklearn.base import BaseEstimator, ClassifierMixin, RegressorMixin
|
|
3
|
+
from .tree import BayesianDecisionTree
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
class SmallGBMClassifier(BaseEstimator, ClassifierMixin):
|
|
7
|
+
def __init__(self, n_estimators=50, max_depth=3, min_samples_leaf=3,
|
|
8
|
+
learning_rate=0.1, sigma_prior=0.5, adaptive_prior=False,
|
|
9
|
+
dynamic_depth=False, weighted_residuals=False, soft_bootstrap=False):
|
|
10
|
+
self.n_estimators = n_estimators
|
|
11
|
+
self.max_depth = max_depth
|
|
12
|
+
self.min_samples_leaf = min_samples_leaf
|
|
13
|
+
self.learning_rate = learning_rate
|
|
14
|
+
self.sigma_prior = sigma_prior
|
|
15
|
+
self.adaptive_prior = adaptive_prior
|
|
16
|
+
self.dynamic_depth = dynamic_depth
|
|
17
|
+
self.weighted_residuals = weighted_residuals
|
|
18
|
+
self.soft_bootstrap = soft_bootstrap
|
|
19
|
+
|
|
20
|
+
def _log_odds(self, y):
|
|
21
|
+
pos = np.sum(y == 1)
|
|
22
|
+
neg = np.sum(y == 0)
|
|
23
|
+
return np.log((pos + 1e-10) / (neg + 1e-10))
|
|
24
|
+
|
|
25
|
+
def _sigmoid(self, x):
|
|
26
|
+
return 1 / (1 + np.exp(-x))
|
|
27
|
+
|
|
28
|
+
def fit(self, X, y):
|
|
29
|
+
X = np.array(X)
|
|
30
|
+
y = np.array(y)
|
|
31
|
+
n_samples = len(y)
|
|
32
|
+
|
|
33
|
+
init = self._log_odds(y)
|
|
34
|
+
self.base_score_ = init
|
|
35
|
+
self._trees = []
|
|
36
|
+
|
|
37
|
+
current_pred = np.full(y.shape, init)
|
|
38
|
+
|
|
39
|
+
for i in range(self.n_estimators):
|
|
40
|
+
proba = self._sigmoid(current_pred)
|
|
41
|
+
residuals = y - proba
|
|
42
|
+
|
|
43
|
+
# 2. Weighted residuals: weight = proba * (1 - proba)
|
|
44
|
+
if self.weighted_residuals:
|
|
45
|
+
confidence = proba * (1 - proba)
|
|
46
|
+
confidence = np.clip(confidence, 1e-10, None)
|
|
47
|
+
sample_weights = confidence / confidence.sum() * n_samples
|
|
48
|
+
else:
|
|
49
|
+
sample_weights = np.ones(n_samples)
|
|
50
|
+
|
|
51
|
+
# 4. Soft bootstrap: weighted sampling without replacement bias
|
|
52
|
+
if self.soft_bootstrap:
|
|
53
|
+
# Use all data but weight the residuals by sample_weights
|
|
54
|
+
weighted_residuals = residuals * sample_weights
|
|
55
|
+
else:
|
|
56
|
+
weighted_residuals = residuals
|
|
57
|
+
|
|
58
|
+
# 3. Dynamic depth: deeper early, shallower later
|
|
59
|
+
if self.dynamic_depth:
|
|
60
|
+
progress = i / self.n_estimators
|
|
61
|
+
current_max_depth = max(1, int(self.max_depth * (1 - progress * 0.5)))
|
|
62
|
+
else:
|
|
63
|
+
current_max_depth = self.max_depth
|
|
64
|
+
|
|
65
|
+
# 1. Adaptive sigma_prior: 1 / sqrt(n) if not specified
|
|
66
|
+
if self.adaptive_prior and self.sigma_prior is None:
|
|
67
|
+
sigma_prior = 1.0 / np.sqrt(n_samples)
|
|
68
|
+
elif self.sigma_prior is not None:
|
|
69
|
+
sigma_prior = self.sigma_prior
|
|
70
|
+
else:
|
|
71
|
+
sigma_prior = 1.0
|
|
72
|
+
|
|
73
|
+
tree = BayesianDecisionTree(
|
|
74
|
+
max_depth=current_max_depth,
|
|
75
|
+
min_samples_leaf=self.min_samples_leaf,
|
|
76
|
+
sigma_prior=sigma_prior,
|
|
77
|
+
n_splits=20
|
|
78
|
+
)
|
|
79
|
+
tree.fit(X, weighted_residuals)
|
|
80
|
+
|
|
81
|
+
update = tree.predict(X)
|
|
82
|
+
current_pred += self.learning_rate * update
|
|
83
|
+
self._trees.append(tree)
|
|
84
|
+
|
|
85
|
+
return self
|
|
86
|
+
|
|
87
|
+
def predict_proba(self, X):
|
|
88
|
+
X = np.array(X)
|
|
89
|
+
current_pred = np.full(X.shape[0], self.base_score_)
|
|
90
|
+
for tree in self._trees:
|
|
91
|
+
current_pred += self.learning_rate * tree.predict(X)
|
|
92
|
+
proba_pos = self._sigmoid(current_pred)
|
|
93
|
+
return np.column_stack([1 - proba_pos, proba_pos])
|
|
94
|
+
|
|
95
|
+
def predict(self, X):
|
|
96
|
+
proba = self.predict_proba(X)[:, 1]
|
|
97
|
+
return (proba > 0.5).astype(int)
|
|
98
|
+
|
|
99
|
+
class SmallGBMRegressor(BaseEstimator, RegressorMixin):
|
|
100
|
+
def __init__(self, n_estimators=50, max_depth=3, min_samples_leaf=3,
|
|
101
|
+
learning_rate=0.1, sigma_prior=0.5, adaptive_prior=False,
|
|
102
|
+
dynamic_depth=False, weighted_residuals=False, soft_bootstrap=False):
|
|
103
|
+
self.n_estimators = n_estimators
|
|
104
|
+
self.max_depth = max_depth
|
|
105
|
+
self.min_samples_leaf = min_samples_leaf
|
|
106
|
+
self.learning_rate = learning_rate
|
|
107
|
+
self.sigma_prior = sigma_prior
|
|
108
|
+
self.adaptive_prior = adaptive_prior
|
|
109
|
+
self.dynamic_depth = dynamic_depth
|
|
110
|
+
self.weighted_residuals = weighted_residuals
|
|
111
|
+
self.soft_bootstrap = soft_bootstrap
|
|
112
|
+
|
|
113
|
+
def fit(self, X, y):
|
|
114
|
+
X = np.array(X)
|
|
115
|
+
y = np.array(y).astype(float)
|
|
116
|
+
n_samples = len(y)
|
|
117
|
+
|
|
118
|
+
# Initial prediction: mean of y
|
|
119
|
+
init = np.mean(y)
|
|
120
|
+
self.base_score_ = init
|
|
121
|
+
self._trees = []
|
|
122
|
+
|
|
123
|
+
current_pred = np.full(y.shape, init)
|
|
124
|
+
|
|
125
|
+
for i in range(self.n_estimators):
|
|
126
|
+
residuals = y - current_pred # MSE residuals
|
|
127
|
+
|
|
128
|
+
if self.weighted_residuals:
|
|
129
|
+
confidence = np.abs(residuals)
|
|
130
|
+
confidence = np.clip(confidence, 1e-10, None)
|
|
131
|
+
sample_weights = confidence / confidence.sum() * n_samples
|
|
132
|
+
else:
|
|
133
|
+
sample_weights = np.ones(n_samples)
|
|
134
|
+
|
|
135
|
+
if self.soft_bootstrap:
|
|
136
|
+
weighted_residuals = residuals * sample_weights
|
|
137
|
+
else:
|
|
138
|
+
weighted_residuals = residuals
|
|
139
|
+
|
|
140
|
+
if self.dynamic_depth:
|
|
141
|
+
progress = i / self.n_estimators
|
|
142
|
+
current_max_depth = max(1, int(self.max_depth * (1 - progress * 0.5)))
|
|
143
|
+
else:
|
|
144
|
+
current_max_depth = self.max_depth
|
|
145
|
+
|
|
146
|
+
if self.adaptive_prior and self.sigma_prior is None:
|
|
147
|
+
sigma_prior = 1.0 / np.sqrt(n_samples)
|
|
148
|
+
elif self.sigma_prior is not None:
|
|
149
|
+
sigma_prior = self.sigma_prior
|
|
150
|
+
else:
|
|
151
|
+
sigma_prior = 1.0
|
|
152
|
+
|
|
153
|
+
tree = BayesianDecisionTree(
|
|
154
|
+
max_depth=current_max_depth,
|
|
155
|
+
min_samples_leaf=self.min_samples_leaf,
|
|
156
|
+
sigma_prior=sigma_prior)
|
|
157
|
+
tree.fit(X, weighted_residuals)
|
|
158
|
+
|
|
159
|
+
update = tree.predict(X)
|
|
160
|
+
current_pred += self.learning_rate * update
|
|
161
|
+
self._trees.append(tree)
|
|
162
|
+
|
|
163
|
+
return self
|
|
164
|
+
|
|
165
|
+
def predict(self, X):
|
|
166
|
+
X = np.array(X)
|
|
167
|
+
current_pred = np.full(X.shape[0], self.base_score_)
|
|
168
|
+
for tree in self._trees:
|
|
169
|
+
current_pred += self.learning_rate * tree.predict(X)
|
|
170
|
+
return current_pred
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
import numpy as np
|
|
2
|
+
|
|
3
|
+
class BayesianDecisionTree:
|
|
4
|
+
"""Optimized decision tree with Bayesian leaf weight estimation."""
|
|
5
|
+
|
|
6
|
+
def __init__(self, max_depth=2, min_samples_leaf=5, sigma_prior=1.0, n_splits=10):
|
|
7
|
+
self.max_depth = max_depth
|
|
8
|
+
self.min_samples_leaf = min_samples_leaf
|
|
9
|
+
self.sigma_prior = sigma_prior
|
|
10
|
+
self.n_splits = n_splits
|
|
11
|
+
self.tree_ = None
|
|
12
|
+
|
|
13
|
+
def _variance(self, y):
|
|
14
|
+
if len(y) <= 1:
|
|
15
|
+
return 0.0
|
|
16
|
+
return np.var(y)
|
|
17
|
+
|
|
18
|
+
def _bayesian_weight(self, residuals):
|
|
19
|
+
n = len(residuals)
|
|
20
|
+
sum_r = np.sum(residuals)
|
|
21
|
+
sigma_noise = self._variance(residuals)
|
|
22
|
+
if sigma_noise == 0:
|
|
23
|
+
return sum_r / n if n > 0 else 0.0
|
|
24
|
+
shrinkage = sigma_noise / (self.sigma_prior ** 2)
|
|
25
|
+
return sum_r / (n + shrinkage)
|
|
26
|
+
|
|
27
|
+
def _best_split(self, X, residuals):
|
|
28
|
+
best_gain = -np.inf
|
|
29
|
+
best_feature = None
|
|
30
|
+
best_threshold = None
|
|
31
|
+
|
|
32
|
+
n_features = X.shape[1]
|
|
33
|
+
n = len(residuals)
|
|
34
|
+
|
|
35
|
+
for feature in range(n_features):
|
|
36
|
+
values = X[:, feature]
|
|
37
|
+
sort_idx = np.argsort(values)
|
|
38
|
+
sorted_values = values[sort_idx]
|
|
39
|
+
sorted_residuals = residuals[sort_idx]
|
|
40
|
+
|
|
41
|
+
# Prefix sums for O(1) variance computation
|
|
42
|
+
cumsum = np.cumsum(sorted_residuals)
|
|
43
|
+
cumsum2 = np.cumsum(sorted_residuals ** 2)
|
|
44
|
+
total_sum = cumsum[-1]
|
|
45
|
+
total_sum2 = cumsum2[-1]
|
|
46
|
+
parent_var = (total_sum2 - total_sum**2 / n) / n if n > 1 else 0
|
|
47
|
+
|
|
48
|
+
# Full search over all valid split positions
|
|
49
|
+
for pos in range(self.min_samples_leaf, n - self.min_samples_leaf + 1):
|
|
50
|
+
left_n = pos
|
|
51
|
+
right_n = n - pos
|
|
52
|
+
|
|
53
|
+
left_sum = cumsum[pos - 1]
|
|
54
|
+
left_sum2 = cumsum2[pos - 1]
|
|
55
|
+
left_var = (left_sum2 - left_sum**2 / left_n) / left_n if left_n > 1 else 0
|
|
56
|
+
|
|
57
|
+
right_sum = total_sum - left_sum
|
|
58
|
+
right_sum2 = total_sum2 - left_sum2
|
|
59
|
+
right_var = (right_sum2 - right_sum**2 / right_n) / right_n if right_n > 1 else 0
|
|
60
|
+
|
|
61
|
+
gain = parent_var - (left_n / n * left_var + right_n / n * right_var)
|
|
62
|
+
|
|
63
|
+
if gain > best_gain:
|
|
64
|
+
best_gain = gain
|
|
65
|
+
best_feature = feature
|
|
66
|
+
best_threshold = (sorted_values[pos - 1] + sorted_values[pos]) / 2
|
|
67
|
+
|
|
68
|
+
return best_feature, best_threshold, best_gain
|
|
69
|
+
|
|
70
|
+
def _build_tree(self, X, residuals, depth=0):
|
|
71
|
+
n = len(residuals)
|
|
72
|
+
|
|
73
|
+
if (depth >= self.max_depth or
|
|
74
|
+
n < self.min_samples_leaf * 2 or
|
|
75
|
+
len(np.unique(residuals)) == 1):
|
|
76
|
+
return {
|
|
77
|
+
'type': 'leaf',
|
|
78
|
+
'weight': self._bayesian_weight(residuals),
|
|
79
|
+
'n_samples': n
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
feature, threshold, gain = self._best_split(X, residuals)
|
|
83
|
+
|
|
84
|
+
if feature is None or gain <= 0:
|
|
85
|
+
return {
|
|
86
|
+
'type': 'leaf',
|
|
87
|
+
'weight': self._bayesian_weight(residuals),
|
|
88
|
+
'n_samples': n
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
left_mask = X[:, feature] <= threshold
|
|
92
|
+
right_mask = ~left_mask
|
|
93
|
+
|
|
94
|
+
return {
|
|
95
|
+
'type': 'node',
|
|
96
|
+
'feature': feature,
|
|
97
|
+
'threshold': threshold,
|
|
98
|
+
'n_samples': n,
|
|
99
|
+
'left': self._build_tree(X[left_mask], residuals[left_mask], depth + 1),
|
|
100
|
+
'right': self._build_tree(X[right_mask], residuals[right_mask], depth + 1)
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
def fit(self, X, residuals):
|
|
104
|
+
self.tree_ = self._build_tree(np.array(X), np.array(residuals))
|
|
105
|
+
return self
|
|
106
|
+
|
|
107
|
+
def _predict_one(self, x, node):
|
|
108
|
+
if node['type'] == 'leaf':
|
|
109
|
+
return node['weight']
|
|
110
|
+
if x[node['feature']] <= node['threshold']:
|
|
111
|
+
return self._predict_one(x, node['left'])
|
|
112
|
+
else:
|
|
113
|
+
return self._predict_one(x, node['right'])
|
|
114
|
+
|
|
115
|
+
def predict(self, X):
|
|
116
|
+
return np.array([self._predict_one(x, self.tree_) for x in np.array(X)])
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: smallgbm
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Gradient boosting optimized for small datasets (n < 1000)
|
|
5
|
+
Home-page: https://github.com/nsdmlk/SmallGBM
|
|
6
|
+
Author: Emelyanov Ilya
|
|
7
|
+
Author-email: Nsdmlk@yandex.ru
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Requires-Python: >=3.8
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
License-File: LICENSE
|
|
16
|
+
Requires-Dist: numpy>=1.20.0
|
|
17
|
+
Requires-Dist: scikit-learn>=1.0.0
|
|
18
|
+
Dynamic: author
|
|
19
|
+
Dynamic: author-email
|
|
20
|
+
Dynamic: classifier
|
|
21
|
+
Dynamic: description
|
|
22
|
+
Dynamic: description-content-type
|
|
23
|
+
Dynamic: home-page
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
Dynamic: requires-dist
|
|
26
|
+
Dynamic: requires-python
|
|
27
|
+
Dynamic: summary
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
# SmallGBM
|
|
31
|
+
|
|
32
|
+
<p align="center">
|
|
33
|
+
<b>Gradient boosting that actually works on small data.</b>
|
|
34
|
+
</p>
|
|
35
|
+
|
|
36
|
+
<p align="center">
|
|
37
|
+
<img src="https://img.shields.io/badge/version-1.0.0-blue" alt="version">
|
|
38
|
+
<img src="https://img.shields.io/badge/python-3.8+-green" alt="python">
|
|
39
|
+
<img src="https://img.shields.io/badge/license-MIT-brightgreen" alt="license">
|
|
40
|
+
<img src="https://img.shields.io/badge/pip%20install-smallgbm-orange" alt="pip">
|
|
41
|
+
</p>
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Why SmallGBM?
|
|
46
|
+
|
|
47
|
+
XGBoost and LightGBM are built for scale. They shine on thousands of rows. But when you only have **50, 100, or 500 samples**, their default hyperparameters fail — overfitting, instability, unpredictable results.
|
|
48
|
+
|
|
49
|
+
**SmallGBM** is designed from the ground up for datasets with fewer than 1000 samples.
|
|
50
|
+
|
|
51
|
+
| Feature | SmallGBM | XGBoost | LightGBM |
|
|
52
|
+
| ---------------------------- | -------- | ------- | -------- |
|
|
53
|
+
| Bayesian leaf weights | ✅ | ❌ | ❌ |
|
|
54
|
+
| Adaptive regularization | ✅ | ❌ | ❌ |
|
|
55
|
+
| No bootstrap (uses all data) | ✅ | ❌ | ❌ |
|
|
56
|
+
| Stable under label noise | ✅ | ❌ | ❌ |
|
|
57
|
+
| scikit-learn compatible | ✅ | ✅ | ✅ |
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Noise Stability
|
|
62
|
+
|
|
63
|
+
SmallGBM degrades gracefully when labels are noisy — unlike XGBoost and LightGBM which drop sharply.
|
|
64
|
+
|
|
65
|
+
<p align="center">
|
|
66
|
+
<img src="docs/noise_comparison.png" width="600" alt="Noise stability comparison">
|
|
67
|
+
</p>
|
|
68
|
+
|
|
69
|
+
*At 20% label noise, SmallGBM is the best performer. Bayesian regularization keeps it stable when others collapse.*
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Learning Curve
|
|
74
|
+
|
|
75
|
+
Clear, predictable improvement as data grows. Reliable performance starts at **n ≈ 40**.
|
|
76
|
+
|
|
77
|
+
<p align="center">
|
|
78
|
+
<img src="docs/learning_curve.png" width="600" alt="Learning curve">
|
|
79
|
+
</p>
|
|
80
|
+
|
|
81
|
+
*No sudden jumps, no catastrophic failures. A safe choice when data is limited.*
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Installation
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
pip install smallgbm
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
## Quickstart
|
|
92
|
+
|
|
93
|
+
```python
|
|
94
|
+
from smallgbm import SmallGBMClassifier
|
|
95
|
+
|
|
96
|
+
model = SmallGBMClassifier()
|
|
97
|
+
model.fit(X_train, y_train)
|
|
98
|
+
proba = model.predict_proba(X_test)
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## API
|
|
102
|
+
|
|
103
|
+
### SmallGBMClassifier
|
|
104
|
+
|
|
105
|
+
| Parameter | Default | Description |
|
|
106
|
+
| -------------------- | ------- | ------------------------- |
|
|
107
|
+
| `n_estimators` | 50 | Number of boosting rounds |
|
|
108
|
+
| `max_depth` | 3 | Maximum tree depth |
|
|
109
|
+
| `min_samples_leaf` | 3 | Minimum samples per leaf |
|
|
110
|
+
| `learning_rate` | 0.1 | Shrinkage factor |
|
|
111
|
+
| `sigma_prior` | 0.5 | Bayesian prior strength |
|
|
112
|
+
|
|
113
|
+
### SmallGBMRegressor
|
|
114
|
+
|
|
115
|
+
Same parameters, for regression tasks.
|
|
116
|
+
|
|
117
|
+
```python
|
|
118
|
+
from smallgbm import SmallGBMRegressor
|
|
119
|
+
|
|
120
|
+
model = SmallGBMRegressor()
|
|
121
|
+
model.fit(X_train, y_train)
|
|
122
|
+
preds = model.predict(X_test)
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
## Research
|
|
128
|
+
|
|
129
|
+
Full characterisation notebook with 5 experiments: `benchmark_final.ipynb`
|
|
130
|
+
|
|
131
|
+
- Noise stability analysis
|
|
132
|
+
- Prior sensitivity
|
|
133
|
+
- Sample size curve
|
|
134
|
+
- Regression performance
|
|
135
|
+
- Class imbalance tolerance
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## License
|
|
140
|
+
|
|
141
|
+
MIT © [Emelyanov Ilya](https://github.com/nsdmlk) 2026
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
<p align="center">
|
|
146
|
+
<sub>Built with ❤️ for the small data community</sub>
|
|
147
|
+
</p>
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
MANIFEST.in
|
|
3
|
+
README.md
|
|
4
|
+
setup.py
|
|
5
|
+
smallgbm/__init__.py
|
|
6
|
+
smallgbm/smallgbm.py
|
|
7
|
+
smallgbm/tree.py
|
|
8
|
+
smallgbm.egg-info/PKG-INFO
|
|
9
|
+
smallgbm.egg-info/SOURCES.txt
|
|
10
|
+
smallgbm.egg-info/dependency_links.txt
|
|
11
|
+
smallgbm.egg-info/requires.txt
|
|
12
|
+
smallgbm.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
smallgbm
|