ogboost 0.5.5__tar.gz

ogboost-0.5.5/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 asmahani
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ogboost-0.5.5/PKG-INFO

@@ -0,0 +1,180 @@
+Metadata-Version: 2.2
+Name: ogboost
+Version: 0.5.5
+Summary: Ordinal Gradient Boosting
+Author-email: "Alireza S. Mahani, Mansour T.A. Sharabiani" <alireza.s.mahani@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2024 asmahani
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Keywords: ordinal regression,gradient boosting,machine learning,scikit-learn
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: scikit-learn
+Requires-Dist: scipy
+Requires-Dist: pydantic
+Requires-Dist: pytest
+Requires-Dist: matplotlib
+
+# Ordinal Gradient Boosting (`OGBoost`)
+
+## Overview
+
+`OGBoost` is a scikit-learn-compatible, Python package for gradient boosting tailored to ordinal regression problems. It does so by alternating between:
+1. Fitting a Machine Learning (ML) regression model - such as a decision tree - to predict a latent score that specifies the mean of a probability density function (PDF), and 
+1. Fitting a set of thresholds that generate discrete outcomes from the PDF.
+
+In other words, `OGBoost` implements coordinate-descent optimization that combines functional gradient descent - for updating the regression function - with ordinary gradient descent - for updating the threshold vector.
+
+The main class of the package, `GradientBoostingOrdinal`, is designed to have the same look and feel as `scikit-learn`'s `GradientBoostingClassifier`. It includes many of the same features such as custom link functions, sample weighting, early stopping using a validation set, and staged predictions.
+
+There are, however, important differences as well.
+
+## Unique Features of `OGBoost`
+
+### Latent-Score Prediction
+
+The `decision_function` method of the `GradientBoostingOrdinal` behaves differently from `scikit-learn`'s classifiers. Assuming the target variable has `K` distinct classes, a nominal classifier's decision function would return `K` values for each sample. On the other hand, `decision_function` in `ogboost` would return the latent score for each sample, which is a single value. This latent score can be considered a high-resolution alternative to class labels, and thus may have superior ranking performance.
+
+### Early Stopping using Cross-Validation (CV)
+
+In addition to using a single validation set for early stopping, similar to `GradientBoostingClassifier`, `ogboost` early stopping using CV, which means error/performance over the entire data is used for calculating out-of-sample performance. This can improve the robustness of the early-stopping strategy, especially for small and/or imbalanced datasets.
+
+### Heterogeneous Ensemble
+
+While most gradient-boosting software packages exclusively use decision trees with a predetermined set of hyperparameters as the base learner in all boosting iterations, `ogboost` offers significantly more flexibility.
+
+1. Users can pass in a `base_learner` parameter to the class initializer to override the default choice of a `DecisionTreeRegressor`. This can be any regression algorithm such as a feed-forward neural network (`MLPRegressor`), or a K-nearest-neighbor regressor (`KNeighborsRegressor`), etc.
+1. Rather than a single base learner, users can specify a list of base learners, which will be drawn from in that order in each boosting iteration. This amounts to creating a *heterogeneous* ensemble as opposed to a *homogeneous* ensemble.
+
+## Installation
+```bash
+pip install ogboost
+```
+
+## Quick Start
+### Load the Wine Quality Dataset
+The package includes a utility to load the wine quality dataset (red and white) from the UCI repository. Note that `load_wine_quality` shifts the target variable (`quality`) to start from `0`. (This is required by the `GradientBoostingOrdinal` class.)
+
+```python
+from ogboost import load_wine_quality
+red_wine, white_wine = load_wine_quality()
+X, y = red_wine.drop(columns="quality"), red_wine["quality"]
+```
+
+### Training, Prediction and Evaluation
+```python
+from ogboost import GradientBoostingOrdinal
+
+## training ##
+model = GradientBoostingOrdinal(n_estimators=100, link_function='logit', verbose=1)
+model.fit(X, y)
+
+## prediction ##
+# class labels
+predicted_labels = model.predict(X)
+# class probabilities
+predicted_probabilities = model.predict_proba(X)
+# latent score
+predicted_latent = model.decision_function(X)
+
+# evaluation
+concordance_latent = model.score(X, y) # concordance using latent scores
+concordance_label = model.score(X, y, pred_type = 'labels') # concordance using class labels
+print(f"Concordance - class labels: {concordance_label:.3f}")
+print(f"Concordance - latent scores: {concordance_latent:.3f}")
+```
+
+### Early-Stopping using Cross-Validation
+
+```python
+from sklearn.model_selection import cross_val_score
+from sklearn.model_selection import RepeatedKFold
+import time
+
+n_splits = 10
+n_repeats = 10
+kf = RepeatedKFold(n_splits=n_splits, n_repeats=n_repeats)
+
+# early-stopping using a simple holdout set
+model_earlystop_simple = GradientBoostingOrdinal(n_iter_no_change=10, validation_fraction=0.2)
+start = time.time()
+c_index_simple = cross_val_score(model_earlystop_simple, X, y, cv=kf, n_jobs=-1)
+end = time.time()
+print(f'Simple early stopping: {c_index_simple.mean():.3f} ({end - start:.1f} seconds)')
+
+# early-stopping using cross-validation
+model_earlystop_cv = GradientBoostingOrdinal(n_iter_no_change=10, cv_early_stopping_splits=5)
+start = time.time()
+c_index_cv = cross_val_score(model_earlystop_cv, X, y, cv=kf, n_jobs=-1)
+end = time.time()
+print(f'CV early stopping: {c_index_cv.mean():.3f} ({end - start:.1f} seconds)')
+
+# statistical comparison of the two methods
+from scipy.stats import ttest_rel
+t_stat, p_value = ttest_rel(c_index_simple, c_index_cv)
+print(f't-statistic: {t_stat}, p-value: {p_value}')
+```
+
+### Heterogeneous Ensemble
+
+Generating a random set of hyperparameter combinations for `DecisionTreeRegressor`:
+
+```python
+import numpy as np
+from sklearn.tree import DecisionTreeRegressor
+
+# Number of samples to generate
+n_samples = 10
+
+max_depth_choices = [3, 6, 9, None]
+max_depths = np.random.choice(max_depth_choices, size=n_samples, replace=True)
+max_leaf_nodes_choices = [10, 20, 30, None]
+max_leaf_nodes = np.random.choice(max_leaf_nodes_choices, size=n_samples, replace=True)
+
+params = list(zip(max_depths, max_leaf_nodes))
+
+# Create list of DecisionTreeRegressor models with sampled parameters
+models = [
+    DecisionTreeRegressor(
+        max_depth=max_depth,
+        max_leaf_nodes=max_leaf_nodes
+    )
+    for max_depth, max_leaf_nodes in params
+]
+```
+
+Quantifying the performance of the heterogeneous ensemble:
+```python
+learning_rate = 0.1
+
+model_heter = GradientBoostingOrdinal(
+    base_learner=models,
+    n_estimators=n_samples
+)
+cv_heter = cross_val_score(model_heter, X, y, cv=kf, n_jobs=-1)
+print(f'average cv score of heteogeneous ensemble: {np.mean(cv_heter):.3f}')
+```
+
+## License
+This package is licensed under the [MIT License](./LICENSE).

ogboost-0.5.5/README.md

@@ -0,0 +1,142 @@
+# Ordinal Gradient Boosting (`OGBoost`)
+
+## Overview
+
+`OGBoost` is a scikit-learn-compatible, Python package for gradient boosting tailored to ordinal regression problems. It does so by alternating between:
+1. Fitting a Machine Learning (ML) regression model - such as a decision tree - to predict a latent score that specifies the mean of a probability density function (PDF), and 
+1. Fitting a set of thresholds that generate discrete outcomes from the PDF.
+
+In other words, `OGBoost` implements coordinate-descent optimization that combines functional gradient descent - for updating the regression function - with ordinary gradient descent - for updating the threshold vector.
+
+The main class of the package, `GradientBoostingOrdinal`, is designed to have the same look and feel as `scikit-learn`'s `GradientBoostingClassifier`. It includes many of the same features such as custom link functions, sample weighting, early stopping using a validation set, and staged predictions.
+
+There are, however, important differences as well.
+
+## Unique Features of `OGBoost`
+
+### Latent-Score Prediction
+
+The `decision_function` method of the `GradientBoostingOrdinal` behaves differently from `scikit-learn`'s classifiers. Assuming the target variable has `K` distinct classes, a nominal classifier's decision function would return `K` values for each sample. On the other hand, `decision_function` in `ogboost` would return the latent score for each sample, which is a single value. This latent score can be considered a high-resolution alternative to class labels, and thus may have superior ranking performance.
+
+### Early Stopping using Cross-Validation (CV)
+
+In addition to using a single validation set for early stopping, similar to `GradientBoostingClassifier`, `ogboost` early stopping using CV, which means error/performance over the entire data is used for calculating out-of-sample performance. This can improve the robustness of the early-stopping strategy, especially for small and/or imbalanced datasets.
+
+### Heterogeneous Ensemble
+
+While most gradient-boosting software packages exclusively use decision trees with a predetermined set of hyperparameters as the base learner in all boosting iterations, `ogboost` offers significantly more flexibility.
+
+1. Users can pass in a `base_learner` parameter to the class initializer to override the default choice of a `DecisionTreeRegressor`. This can be any regression algorithm such as a feed-forward neural network (`MLPRegressor`), or a K-nearest-neighbor regressor (`KNeighborsRegressor`), etc.
+1. Rather than a single base learner, users can specify a list of base learners, which will be drawn from in that order in each boosting iteration. This amounts to creating a *heterogeneous* ensemble as opposed to a *homogeneous* ensemble.
+
+## Installation
+```bash
+pip install ogboost
+```
+
+## Quick Start
+### Load the Wine Quality Dataset
+The package includes a utility to load the wine quality dataset (red and white) from the UCI repository. Note that `load_wine_quality` shifts the target variable (`quality`) to start from `0`. (This is required by the `GradientBoostingOrdinal` class.)
+
+```python
+from ogboost import load_wine_quality
+red_wine, white_wine = load_wine_quality()
+X, y = red_wine.drop(columns="quality"), red_wine["quality"]
+```
+
+### Training, Prediction and Evaluation
+```python
+from ogboost import GradientBoostingOrdinal
+
+## training ##
+model = GradientBoostingOrdinal(n_estimators=100, link_function='logit', verbose=1)
+model.fit(X, y)
+
+## prediction ##
+# class labels
+predicted_labels = model.predict(X)
+# class probabilities
+predicted_probabilities = model.predict_proba(X)
+# latent score
+predicted_latent = model.decision_function(X)
+
+# evaluation
+concordance_latent = model.score(X, y) # concordance using latent scores
+concordance_label = model.score(X, y, pred_type = 'labels') # concordance using class labels
+print(f"Concordance - class labels: {concordance_label:.3f}")
+print(f"Concordance - latent scores: {concordance_latent:.3f}")
+```
+
+### Early-Stopping using Cross-Validation
+
+```python
+from sklearn.model_selection import cross_val_score
+from sklearn.model_selection import RepeatedKFold
+import time
+
+n_splits = 10
+n_repeats = 10
+kf = RepeatedKFold(n_splits=n_splits, n_repeats=n_repeats)
+
+# early-stopping using a simple holdout set
+model_earlystop_simple = GradientBoostingOrdinal(n_iter_no_change=10, validation_fraction=0.2)
+start = time.time()
+c_index_simple = cross_val_score(model_earlystop_simple, X, y, cv=kf, n_jobs=-1)
+end = time.time()
+print(f'Simple early stopping: {c_index_simple.mean():.3f} ({end - start:.1f} seconds)')
+
+# early-stopping using cross-validation
+model_earlystop_cv = GradientBoostingOrdinal(n_iter_no_change=10, cv_early_stopping_splits=5)
+start = time.time()
+c_index_cv = cross_val_score(model_earlystop_cv, X, y, cv=kf, n_jobs=-1)
+end = time.time()
+print(f'CV early stopping: {c_index_cv.mean():.3f} ({end - start:.1f} seconds)')
+
+# statistical comparison of the two methods
+from scipy.stats import ttest_rel
+t_stat, p_value = ttest_rel(c_index_simple, c_index_cv)
+print(f't-statistic: {t_stat}, p-value: {p_value}')
+```
+
+### Heterogeneous Ensemble
+
+Generating a random set of hyperparameter combinations for `DecisionTreeRegressor`:
+
+```python
+import numpy as np
+from sklearn.tree import DecisionTreeRegressor
+
+# Number of samples to generate
+n_samples = 10
+
+max_depth_choices = [3, 6, 9, None]
+max_depths = np.random.choice(max_depth_choices, size=n_samples, replace=True)
+max_leaf_nodes_choices = [10, 20, 30, None]
+max_leaf_nodes = np.random.choice(max_leaf_nodes_choices, size=n_samples, replace=True)
+
+params = list(zip(max_depths, max_leaf_nodes))
+
+# Create list of DecisionTreeRegressor models with sampled parameters
+models = [
+    DecisionTreeRegressor(
+        max_depth=max_depth,
+        max_leaf_nodes=max_leaf_nodes
+    )
+    for max_depth, max_leaf_nodes in params
+]
+```
+
+Quantifying the performance of the heterogeneous ensemble:
+```python
+learning_rate = 0.1
+
+model_heter = GradientBoostingOrdinal(
+    base_learner=models,
+    n_estimators=n_samples
+)
+cv_heter = cross_val_score(model_heter, X, y, cv=kf, n_jobs=-1)
+print(f'average cv score of heteogeneous ensemble: {np.mean(cv_heter):.3f}')
+```
+
+## License
+This package is licensed under the [MIT License](./LICENSE).

ogboost-0.5.5/ogboost/__init__.py

@@ -0,0 +1,8 @@
+# ogboost/__init__.py
+
+# Re-export key components from submodules
+from .data import load_wine_quality
+from .main import GradientBoostingOrdinal, concordance_index, LinkFunctions
+
+# Define the public API
+__all__ = ["load_wine_quality", "GradientBoostingOrdinal", "concordance_index", "LinkFunctions"]

ogboost-0.5.5/ogboost/data.py

@@ -0,0 +1,103 @@
+"""
+Module for loading the Wine Quality Dataset.
+
+This module provides functionality to download and load the Wine Quality Dataset from the 
+UCI Machine Learning Repository. The dataset includes information about physicochemical 
+tests (e.g., pH, alcohol content) and quality scores for red and white wines.
+
+The module contains the following functionality:
+- Automatically downloads the datasets if not found locally.
+- Loads the red and white wine datasets into pandas DataFrames.
+- Rescales the 'quality' column to start from 0 and ensures it is of integer type.
+
+Functions
+---------
+load_wine_quality():
+    Loads the wine quality dataset. Automatically downloads the datasets 
+    if they are not found locally and returns them as pandas DataFrames.
+
+_download_wine_quality_datasets(destination: str):
+    Downloads the wine quality datasets (red and white) from the UCI repository 
+    to the specified destination folder.
+
+Examples
+--------
+>>> from mymodule import load_wine_quality
+>>> red_wine, white_wine = load_wine_quality()
+>>> print(red_wine.head())
+>>> print(white_wine.head())
+"""
+import os
+import urllib.request
+import pandas as pd
+
+def load_wine_quality():
+    """
+    Loads the wine quality dataset from the UCI repository. 
+    If the datasets are not already downloaded,
+    they will be downloaded automatically.
+
+    Returns
+    -------
+    tuple of pandas.DataFrame
+        A tuple containing two DataFrames:
+        - The first for red wine data.
+        - The second for white wine data.
+    """
+    # Define the local paths for the datasets
+    dataset_folder = os.path.join(os.path.dirname(__file__), "data")
+    red_wine_path = os.path.join(dataset_folder, "winequality-red.csv")
+    white_wine_path = os.path.join(dataset_folder, "winequality-white.csv")
+
+    # Check if the datasets exist locally; if not, download them
+    if not os.path.exists(red_wine_path) or not os.path.exists(white_wine_path):
+        print("Datasets not found locally. Downloading from UCI repository...")
+        os.makedirs(dataset_folder, exist_ok=True)
+        _download_wine_quality_datasets(dataset_folder)
+
+    # Load the datasets into pandas DataFrames
+    red_wine_df = pd.read_csv(red_wine_path, sep=";")
+    white_wine_df = pd.read_csv(white_wine_path, sep=";")
+
+    # rescale quality to start from 0
+    red_wine_df["quality"] -= red_wine_df["quality"].min()
+    white_wine_df["quality"] -= white_wine_df["quality"].min()
+
+    # ensure data type for response is integer
+    red_wine_df["quality"] = red_wine_df["quality"].astype(int)
+    white_wine_df["quality"] = white_wine_df["quality"].astype(int)
+
+    return red_wine_df, white_wine_df
+
+
+def _download_wine_quality_datasets(destination: str):
+    """
+    Downloads the wine quality datasets from the UCI repository.
+
+    Parameters
+    ----------
+    destination : str
+        Directory where the datasets will be saved.
+    """
+    # UCI repository URLs
+    red_wine_url = (
+        "https://archive.ics.uci.edu/ml/machine-learning-databases/"
+        "wine-quality/winequality-red.csv"
+    )
+    white_wine_url = (
+        "https://archive.ics.uci.edu/ml/machine-learning-databases/"
+        "wine-quality/winequality-white.csv"
+    )
+
+    # Define file paths
+    red_wine_path = os.path.join(destination, "winequality-red.csv")
+    white_wine_path = os.path.join(destination, "winequality-white.csv")
+
+    # Download the datasets
+    print(f"Downloading red wine dataset to {red_wine_path}...")
+    urllib.request.urlretrieve(red_wine_url, red_wine_path)
+    print("Red wine dataset downloaded.")
+
+    print(f"Downloading white wine dataset to {white_wine_path}...")
+    urllib.request.urlretrieve(white_wine_url, white_wine_path)
+    print("White wine dataset downloaded.")