optuml 0.1.0__tar.gz

optuml-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 filips
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

optuml-0.1.0/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.1
+Name: optuml
+Version: 0.1.0
+Summary: Hyperparameter optimization for multiple machine learning algorithms using Optuna, with Scikit-learn API
+Home-page: https://github.com/filipsPL/optuml
+Author: Filip S.
+Author-email: filip.ursynow@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: optuna
+Requires-Dist: scikit-learn
+Requires-Dist: catboost
+Requires-Dist: xgboost
+Requires-Dist: numpy
+
+# OptuML: Hyperparameter Optimization for Multiple Machine Learning Algorithms using Optuna
+
+```
+  ⣰⡁ ⡀⣀ ⢀⡀ ⣀⣀    ⡎⢱ ⣀⡀ ⣰⡀ ⡀⢀ ⡷⢾ ⡇    ⠄ ⣀⣀  ⣀⡀ ⢀⡀ ⡀⣀ ⣰⡀   ⡎⢱ ⣀⡀ ⣰⡀ ⠄ ⣀⣀  ⠄ ⣀⣀ ⢀⡀ ⡀⣀
+  ⢸  ⠏  ⠣⠜ ⠇⠇⠇   ⠣⠜ ⡧⠜ ⠘⠤ ⠣⠼ ⠇⠸ ⠧⠤   ⠇ ⠇⠇⠇ ⡧⠜ ⠣⠜ ⠏  ⠘⠤   ⠣⠜ ⡧⠜ ⠘⠤ ⠇ ⠇⠇⠇ ⠇ ⠴⠥ ⠣⠭ ⠏ 
+```
+
+`OptuML` is a Python module that provides hyperparameter optimization for several machine learning algorithms using the [Optuna](https://optuna.org/) framework. The module supports a variety of algorithms and allows easy hyperparameter tuning through a scikit-learn-like API.
+
+[![Python tests](https://github.com/filipsPL/optuml/actions/workflows/python-package.yml/badge.svg)](https://github.com/filipsPL/optuml/actions/workflows/python-package.yml)
+
+- [OptuML: Hyperparameter Optimization for Multiple Machine Learning Algorithms using Optuna](#optuml-hyperparameter-optimization-for-multiple-machine-learning-algorithms-using-optuna)
+  - [Quick start](#quick-start)
+  - [Features](#features)
+  - [Installation](#installation)
+    - [Prerequisites](#prerequisites)
+    - [Installation](#installation-1)
+  - [Usage](#usage)
+    - [Basic Example](#basic-example)
+    - [Available Algorithms](#available-algorithms)
+    - [Example of Optimizing Different Algorithms](#example-of-optimizing-different-algorithms)
+    - [Custom Scoring and Direction](#custom-scoring-and-direction)
+    - [Controlling Verbosity](#controlling-verbosity)
+  - [API Reference](#api-reference)
+    - [`Optimizer`](#optimizer)
+      - [Parameters:](#parameters)
+      - [Methods:](#methods)
+  - [Contributing](#contributing)
+  - [Running Tests](#running-tests)
+  - [Contact](#contact)
+
+
+## Quick start
+
+```python
+from OptuML import Optimizer
+
+# Instantiate the optimizer for SVM
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict on the test set
+y_pred = optimizer.predict(X_test)
+```
+
+
+## Features
+
+- **Multiple Algorithms**: Supports hyperparameter optimization for the following algorithms:
+  - Support Vector Machine (SVM)
+  - k-Nearest Neighbors (kNN)
+  - Random Forest
+  - CatBoost
+  - XGBoost
+  - Logistic Regression
+  - Decision Tree
+- **Optuna Framework**: Leverages Optuna for powerful hyperparameter search.
+- **Maximize or Minimize**: Allows setting the optimization direction (`maximize` or `minimize`).
+- **Scikit-learn API**: Provides a consistent interface for `fit()`, `predict()`, `predict_proba()`, and `score()` methods.
+- **Control Output**: Optionally run Optuna with verbose logging (`verbose=True`) or in silent mode (`verbose=False`).
+- **Cross-validation**: Easily integrate cross-validation with custom scoring metrics (e.g., accuracy, ROC AUC).
+
+## Installation
+
+### Prerequisites
+
+Ensure that the following Python packages are installed:
+
+- `optuna`
+- `scikit-learn`
+- `catboost`
+- `xgboost`
+- `numpy`
+
+You can install the required packages via `pip`:
+
+```bash
+pip install optuna scikit-learn catboost xgboost numpy
+```
+
+### Installation
+
+Just fetch the `optuml.py` [file from the repo](optuml.py) and put it in the directory with your script.
+
+
+## Usage
+
+### Basic Example
+
+Here’s how you can use the `Optimizer` class to optimize hyperparameters for different machine learning algorithms using the **Iris** dataset:
+
+```python
+from sklearn.datasets import load_iris
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+from OptuML import Optimizer
+
+# Load the Iris dataset
+X, y = load_iris(return_X_y=True)
+
+# Convert to a binary classification problem
+X = X[y != 2]
+y = y[y != 2]
+
+# Split the dataset into training and test sets
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
+
+# Instantiate the optimizer for SVM
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True, random_state=42)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict on the test set
+y_pred = optimizer.predict(X_test)
+
+# Calculate the accuracy
+accuracy = accuracy_score(y_test, y_pred)
+print(f"Accuracy: {accuracy}")
+
+# Print the best hyperparameters found during optimization
+print(f"Best Hyperparameters: {optimizer.best_params_}")
+```
+
+### Available Algorithms
+
+The `Optimizer` class supports the following algorithms. You can specify the `algorithm` parameter to choose which one to use:
+
+- `"SVM"` (Support Vector Machine)
+- `"kNN"` (k-Nearest Neighbors)
+- `"RandomForest"` (Random Forest)
+- `"CatBoost"` (CatBoost Classifier)
+- `"XGBoost"` (XGBoost Classifier)
+- `"LogisticRegression"` (Logistic Regression)
+- `"DecisionTree"` (Decision Tree Classifier)
+
+### Example of Optimizing Different Algorithms
+
+You can iterate over different algorithms and optimize hyperparameters for each one:
+
+```python
+algorithms = ['SVM', 'kNN', 'RandomForest', 'CatBoost', 'XGBoost', 'LogisticRegression', 'DecisionTree']
+
+for algorithm in algorithms:
+    print(f"Optimizing {algorithm}")
+    optimizer = Optimizer(algorithm=algorithm, n_trials=50, cv=3, scoring="accuracy", verbose=False, random_state=42)
+    optimizer.fit(X_train, y_train)
+    
+    # Predict and calculate accuracy
+    y_pred = optimizer.predict(X_test)
+    accuracy = accuracy_score(y_test, y_pred)
+    print(f"Accuracy for {algorithm}: {accuracy}")
+    print(f"Best Hyperparameters for {algorithm}: {optimizer.best_params_}")
+    print("="*60)
+```
+
+### Custom Scoring and Direction
+
+You can also optimize for different scoring metrics like ROC AUC or use the `direction` parameter to **minimize** or **maximize** the objective:
+
+```python
+# Instantiate the optimizer for RandomForest with ROC AUC optimization
+optimizer = Optimizer(algorithm="RandomForest", n_trials=50, cv=3, scoring="roc_auc", direction="maximize", verbose=True, random_state=42)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict probabilities and calculate ROC AUC
+y_proba = optimizer.predict_proba(X_test)
+roc_auc = roc_auc_score(y_test, y_proba[:, 1])
+print(f"ROC AUC: {roc_auc}")
+
+# Print the best hyperparameters
+print(f"Best Hyperparameters: {optimizer.best_params_}")
+```
+
+### Controlling Verbosity
+
+You can control the verbosity of Optuna's output by using the `verbose` parameter:
+
+- Set `verbose=True` to enable detailed Optuna logging.
+- Set `verbose=False` to suppress logging and run the optimizer silently.
+
+```python
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True, random_state=42)
+```
+
+## API Reference
+
+### `Optimizer`
+
+#### Parameters:
+
+- **`algorithm`** (`str`): The machine learning algorithm to optimize. Options are `'SVM'`, `'kNN'`, `'RandomForest'`, `'CatBoost'`, `'XGBoost'`, `'LogisticRegression'`, `'DecisionTree'`.
+- **`direction`** (`str`, default `"maximize"`): Direction of optimization. Can be `"maximize"` or `"minimize"`.
+- **`verbose`** (`bool`, default `False`): If `True`, Optuna logging will be enabled. If `False`, the optimizer will run silently.
+- **`n_trials`** (`int`, default `100`): Number of optimization trials to run.
+- **`timeout`** (`float`, optional): Maximum time (in seconds) for the optimization process.
+- **`cv`** (`int`, default `5`): Number of cross-validation folds.
+- **`scoring`** (`str`, default `"accuracy"`): Scoring metric to use during cross-validation.
+- **`random_state`** (`int`, optional): Seed for random number generation.
+
+#### Methods:
+
+- **`fit(X, y)`**: Fit the model using hyperparameter optimization.
+- **`predict(X)`**: Make predictions using the best model found during optimization.
+- **`predict_proba(X)`**: Predict class probabilities (if supported by the model).
+- **`score(X, y)`**: Score the model using the test data.
+- **`optimization_time()`**: Get the total time taken for optimization.
+
+## Contributing
+
+Contributions to the project are welcome! Please feel free to submit issues or pull requests on GitHub. You can also fork the repository and make your changes.
+
+## Running Tests
+
+You can run unit tests using `pytest`:
+
+```bash
+pip install pytest
+pytest
+```
+
+## Contact
+
+If you have any questions or feedback, feel free to open an issue on GitHub.

optuml-0.1.0/README.md

@@ -0,0 +1,228 @@
+# OptuML: Hyperparameter Optimization for Multiple Machine Learning Algorithms using Optuna
+
+```
+  ⣰⡁ ⡀⣀ ⢀⡀ ⣀⣀    ⡎⢱ ⣀⡀ ⣰⡀ ⡀⢀ ⡷⢾ ⡇    ⠄ ⣀⣀  ⣀⡀ ⢀⡀ ⡀⣀ ⣰⡀   ⡎⢱ ⣀⡀ ⣰⡀ ⠄ ⣀⣀  ⠄ ⣀⣀ ⢀⡀ ⡀⣀
+  ⢸  ⠏  ⠣⠜ ⠇⠇⠇   ⠣⠜ ⡧⠜ ⠘⠤ ⠣⠼ ⠇⠸ ⠧⠤   ⠇ ⠇⠇⠇ ⡧⠜ ⠣⠜ ⠏  ⠘⠤   ⠣⠜ ⡧⠜ ⠘⠤ ⠇ ⠇⠇⠇ ⠇ ⠴⠥ ⠣⠭ ⠏ 
+```
+
+`OptuML` is a Python module that provides hyperparameter optimization for several machine learning algorithms using the [Optuna](https://optuna.org/) framework. The module supports a variety of algorithms and allows easy hyperparameter tuning through a scikit-learn-like API.
+
+[![Python tests](https://github.com/filipsPL/optuml/actions/workflows/python-package.yml/badge.svg)](https://github.com/filipsPL/optuml/actions/workflows/python-package.yml)
+
+- [OptuML: Hyperparameter Optimization for Multiple Machine Learning Algorithms using Optuna](#optuml-hyperparameter-optimization-for-multiple-machine-learning-algorithms-using-optuna)
+  - [Quick start](#quick-start)
+  - [Features](#features)
+  - [Installation](#installation)
+    - [Prerequisites](#prerequisites)
+    - [Installation](#installation-1)
+  - [Usage](#usage)
+    - [Basic Example](#basic-example)
+    - [Available Algorithms](#available-algorithms)
+    - [Example of Optimizing Different Algorithms](#example-of-optimizing-different-algorithms)
+    - [Custom Scoring and Direction](#custom-scoring-and-direction)
+    - [Controlling Verbosity](#controlling-verbosity)
+  - [API Reference](#api-reference)
+    - [`Optimizer`](#optimizer)
+      - [Parameters:](#parameters)
+      - [Methods:](#methods)
+  - [Contributing](#contributing)
+  - [Running Tests](#running-tests)
+  - [Contact](#contact)
+
+
+## Quick start
+
+```python
+from OptuML import Optimizer
+
+# Instantiate the optimizer for SVM
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict on the test set
+y_pred = optimizer.predict(X_test)
+```
+
+
+## Features
+
+- **Multiple Algorithms**: Supports hyperparameter optimization for the following algorithms:
+  - Support Vector Machine (SVM)
+  - k-Nearest Neighbors (kNN)
+  - Random Forest
+  - CatBoost
+  - XGBoost
+  - Logistic Regression
+  - Decision Tree
+- **Optuna Framework**: Leverages Optuna for powerful hyperparameter search.
+- **Maximize or Minimize**: Allows setting the optimization direction (`maximize` or `minimize`).
+- **Scikit-learn API**: Provides a consistent interface for `fit()`, `predict()`, `predict_proba()`, and `score()` methods.
+- **Control Output**: Optionally run Optuna with verbose logging (`verbose=True`) or in silent mode (`verbose=False`).
+- **Cross-validation**: Easily integrate cross-validation with custom scoring metrics (e.g., accuracy, ROC AUC).
+
+## Installation
+
+### Prerequisites
+
+Ensure that the following Python packages are installed:
+
+- `optuna`
+- `scikit-learn`
+- `catboost`
+- `xgboost`
+- `numpy`
+
+You can install the required packages via `pip`:
+
+```bash
+pip install optuna scikit-learn catboost xgboost numpy
+```
+
+### Installation
+
+Just fetch the `optuml.py` [file from the repo](optuml.py) and put it in the directory with your script.
+
+
+## Usage
+
+### Basic Example
+
+Here’s how you can use the `Optimizer` class to optimize hyperparameters for different machine learning algorithms using the **Iris** dataset:
+
+```python
+from sklearn.datasets import load_iris
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+from OptuML import Optimizer
+
+# Load the Iris dataset
+X, y = load_iris(return_X_y=True)
+
+# Convert to a binary classification problem
+X = X[y != 2]
+y = y[y != 2]
+
+# Split the dataset into training and test sets
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
+
+# Instantiate the optimizer for SVM
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True, random_state=42)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict on the test set
+y_pred = optimizer.predict(X_test)
+
+# Calculate the accuracy
+accuracy = accuracy_score(y_test, y_pred)
+print(f"Accuracy: {accuracy}")
+
+# Print the best hyperparameters found during optimization
+print(f"Best Hyperparameters: {optimizer.best_params_}")
+```
+
+### Available Algorithms
+
+The `Optimizer` class supports the following algorithms. You can specify the `algorithm` parameter to choose which one to use:
+
+- `"SVM"` (Support Vector Machine)
+- `"kNN"` (k-Nearest Neighbors)
+- `"RandomForest"` (Random Forest)
+- `"CatBoost"` (CatBoost Classifier)
+- `"XGBoost"` (XGBoost Classifier)
+- `"LogisticRegression"` (Logistic Regression)
+- `"DecisionTree"` (Decision Tree Classifier)
+
+### Example of Optimizing Different Algorithms
+
+You can iterate over different algorithms and optimize hyperparameters for each one:
+
+```python
+algorithms = ['SVM', 'kNN', 'RandomForest', 'CatBoost', 'XGBoost', 'LogisticRegression', 'DecisionTree']
+
+for algorithm in algorithms:
+    print(f"Optimizing {algorithm}")
+    optimizer = Optimizer(algorithm=algorithm, n_trials=50, cv=3, scoring="accuracy", verbose=False, random_state=42)
+    optimizer.fit(X_train, y_train)
+    
+    # Predict and calculate accuracy
+    y_pred = optimizer.predict(X_test)
+    accuracy = accuracy_score(y_test, y_pred)
+    print(f"Accuracy for {algorithm}: {accuracy}")
+    print(f"Best Hyperparameters for {algorithm}: {optimizer.best_params_}")
+    print("="*60)
+```
+
+### Custom Scoring and Direction
+
+You can also optimize for different scoring metrics like ROC AUC or use the `direction` parameter to **minimize** or **maximize** the objective:
+
+```python
+# Instantiate the optimizer for RandomForest with ROC AUC optimization
+optimizer = Optimizer(algorithm="RandomForest", n_trials=50, cv=3, scoring="roc_auc", direction="maximize", verbose=True, random_state=42)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict probabilities and calculate ROC AUC
+y_proba = optimizer.predict_proba(X_test)
+roc_auc = roc_auc_score(y_test, y_proba[:, 1])
+print(f"ROC AUC: {roc_auc}")
+
+# Print the best hyperparameters
+print(f"Best Hyperparameters: {optimizer.best_params_}")
+```
+
+### Controlling Verbosity
+
+You can control the verbosity of Optuna's output by using the `verbose` parameter:
+
+- Set `verbose=True` to enable detailed Optuna logging.
+- Set `verbose=False` to suppress logging and run the optimizer silently.
+
+```python
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True, random_state=42)
+```
+
+## API Reference
+
+### `Optimizer`
+
+#### Parameters:
+
+- **`algorithm`** (`str`): The machine learning algorithm to optimize. Options are `'SVM'`, `'kNN'`, `'RandomForest'`, `'CatBoost'`, `'XGBoost'`, `'LogisticRegression'`, `'DecisionTree'`.
+- **`direction`** (`str`, default `"maximize"`): Direction of optimization. Can be `"maximize"` or `"minimize"`.
+- **`verbose`** (`bool`, default `False`): If `True`, Optuna logging will be enabled. If `False`, the optimizer will run silently.
+- **`n_trials`** (`int`, default `100`): Number of optimization trials to run.
+- **`timeout`** (`float`, optional): Maximum time (in seconds) for the optimization process.
+- **`cv`** (`int`, default `5`): Number of cross-validation folds.
+- **`scoring`** (`str`, default `"accuracy"`): Scoring metric to use during cross-validation.
+- **`random_state`** (`int`, optional): Seed for random number generation.
+
+#### Methods:
+
+- **`fit(X, y)`**: Fit the model using hyperparameter optimization.
+- **`predict(X)`**: Make predictions using the best model found during optimization.
+- **`predict_proba(X)`**: Predict class probabilities (if supported by the model).
+- **`score(X, y)`**: Score the model using the test data.
+- **`optimization_time()`**: Get the total time taken for optimization.
+
+## Contributing
+
+Contributions to the project are welcome! Please feel free to submit issues or pull requests on GitHub. You can also fork the repository and make your changes.
+
+## Running Tests
+
+You can run unit tests using `pytest`:
+
+```bash
+pip install pytest
+pytest
+```
+
+## Contact
+
+If you have any questions or feedback, feel free to open an issue on GitHub.

optuml-0.1.0/optimum/optuml.py

@@ -0,0 +1,152 @@
+import optuna
+from sklearn.svm import SVC
+from sklearn.neighbors import KNeighborsClassifier
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import cross_val_score
+from sklearn.base import BaseEstimator, ClassifierMixin
+from catboost import CatBoostClassifier
+from xgboost import XGBClassifier
+import time
+import warnings
+
+class Optimizer(BaseEstimator, ClassifierMixin):
+    def __init__(self, algorithm="SVM", direction="maximize", verbose=False, n_trials=100, timeout=None, cv=5, scoring="accuracy", random_state=None):
+        """
+        Initializes the optimizer with the following parameters:
+        :param algorithm: Machine learning algorithm to optimize. Options are 'SVM', 'kNN', 'RandomForest', 'CatBoost', 'XGBoost', 'LogisticRegression', 'DecisionTree'.
+        :param direction: Optimization direction, either 'maximize' (default) or 'minimize'.
+        :param verbose: If True, enables Optuna verbose logging.
+        :param n_trials: Number of trials for Optuna optimization (default 100)
+        :param timeout: Maximum time allowed for optimization (in seconds)
+        :param cv: Number of cross-validation folds (default 5)
+        :param scoring: Scoring method for cross-validation (default "accuracy")
+        :param random_state: Random state for reproducibility (default None)
+        """
+        self.algorithm = algorithm
+        self.direction = direction
+        self.verbosity = verbose
+        self.n_trials = n_trials
+        self.timeout = timeout
+        self.cv = cv
+        self.scoring = scoring
+        self.random_state = random_state
+        self.best_params_ = None
+        self.best_estimator_ = None
+        self.study_time_ = None
+
+        # Set Optuna logging verbosity
+        if verbose:
+            optuna.logging.set_verbosity(optuna.logging.INFO)
+        else:
+            optuna.logging.set_verbosity(optuna.logging.WARNING)
+
+        # Suppress XGBoost warnings about label encoder
+        warnings.filterwarnings('ignore', category=UserWarning, message="`use_label_encoder` is deprecated")
+
+    def _objective(self, trial, X, y):
+        """Objective function for Optuna optimization"""
+        if self.algorithm == "SVM":
+            C = trial.suggest_float("C", 1e-5, 1e2, log=True)
+            gamma = trial.suggest_float("gamma", 1e-5, 1e1, log=True)
+            kernel = trial.suggest_categorical("kernel", ["linear", "rbf", "poly", "sigmoid"])
+            if kernel == "poly":
+                degree = trial.suggest_int("degree", 2, 5)
+            else:
+                degree = 3
+            model = SVC(C=C, gamma=gamma, kernel=kernel, degree=degree, random_state=self.random_state, probability=True)
+
+        elif self.algorithm == "kNN":
+            n_neighbors = trial.suggest_int("n_neighbors", 1, 20)
+            weights = trial.suggest_categorical("weights", ["uniform", "distance"])
+            p = trial.suggest_int("p", 1, 2)
+            model = KNeighborsClassifier(n_neighbors=n_neighbors, weights=weights, p=p)
+
+        elif self.algorithm == "RandomForest":
+            n_estimators = trial.suggest_int("n_estimators", 10, 200)
+            max_depth = trial.suggest_int("max_depth", 2, 32, log=True)
+            min_samples_split = trial.suggest_int("min_samples_split", 2, 20)
+            min_samples_leaf = trial.suggest_int("min_samples_leaf", 1, 20)
+            model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth,
+                                           min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf,
+                                           random_state=self.random_state)
+
+        elif self.algorithm == "CatBoost":
+            learning_rate = trial.suggest_float("learning_rate", 1e-5, 1.0, log=True)
+            depth = trial.suggest_int("depth", 4, 10)
+            l2_leaf_reg = trial.suggest_float("l2_leaf_reg", 1e-5, 10.0, log=True)
+            model = CatBoostClassifier(learning_rate=learning_rate, depth=depth, l2_leaf_reg=l2_leaf_reg,
+                                       verbose=0, random_state=self.random_state)
+
+        elif self.algorithm == "XGBoost":
+            eta = trial.suggest_float("eta", 1e-5, 1.0, log=True)
+            max_depth = trial.suggest_int("max_depth", 2, 32)
+            subsample = trial.suggest_float("subsample", 0.5, 1.0)
+            colsample_bytree = trial.suggest_float("colsample_bytree", 0.5, 1.0)
+            model = XGBClassifier(eta=eta, max_depth=max_depth, subsample=subsample,
+                                  colsample_bytree=colsample_bytree, random_state=self.random_state, use_label_encoder=False)
+
+        elif self.algorithm == "LogisticRegression":
+            C = trial.suggest_float("C", 1e-5, 1e2, log=True)
+            solver = trial.suggest_categorical("solver", ["liblinear", "saga", "lbfgs"])
+            model = LogisticRegression(C=C, solver=solver, random_state=self.random_state)
+
+        elif self.algorithm == "DecisionTree":
+            max_depth = trial.suggest_int("max_depth", 2, 32)
+            min_samples_split = trial.suggest_int("min_samples_split", 2, 20)
+            min_samples_leaf = trial.suggest_int("min_samples_leaf", 1, 20)
+            model = DecisionTreeClassifier(max_depth=max_depth, min_samples_split=min_samples_split,
+                                           min_samples_leaf=min_samples_leaf, random_state=self.random_state)
+
+        # Perform cross-validation and return the mean score
+        return cross_val_score(model, X, y, cv=self.cv, scoring=self.scoring).mean()
+
+    def fit(self, X, y):
+        """Fit the chosen ML model with hyperparameter optimization"""
+        study = optuna.create_study(direction=self.direction, sampler=optuna.samplers.TPESampler(seed=self.random_state))
+
+        start_time = time.time()
+        study.optimize(lambda trial: self._objective(trial, X, y), n_trials=self.n_trials, timeout=self.timeout)
+        self.study_time_ = time.time() - start_time
+
+        self.best_params_ = study.best_params
+
+        # Choose the best estimator based on the algorithm
+        if self.algorithm == "SVM":
+            self.best_estimator_ = SVC(**self.best_params_, random_state=self.random_state, probability=True)
+        elif self.algorithm == "kNN":
+            self.best_estimator_ = KNeighborsClassifier(**self.best_params_)
+        elif self.algorithm == "RandomForest":
+            self.best_estimator_ = RandomForestClassifier(**self.best_params_, random_state=self.random_state)
+        elif self.algorithm == "CatBoost":
+            self.best_estimator_ = CatBoostClassifier(**self.best_params_, verbose=0, random_state=self.random_state)
+        elif self.algorithm == "XGBoost":
+            self.best_estimator_ = XGBClassifier(**self.best_params_, random_state=self.random_state, use_label_encoder=False)
+        elif self.algorithm == "LogisticRegression":
+            self.best_estimator_ = LogisticRegression(**self.best_params_, random_state=self.random_state)
+        elif self.algorithm == "DecisionTree":
+            self.best_estimator_ = DecisionTreeClassifier(**self.best_params_, random_state=self.random_state)
+
+        self.best_estimator_.fit(X, y)
+
+        return self
+
+    def predict(self, X):
+        """Make predictions using the best estimator"""
+        return self.best_estimator_.predict(X)
+
+    def predict_proba(self, X):
+        """Predict class probabilities using the best estimator."""
+        if hasattr(self.best_estimator_, "predict_proba"):
+            return self.best_estimator_.predict_proba(X)
+        else:
+            raise AttributeError("This model does not support probability estimates.")
+
+    def score(self, X, y):
+        """Return the score of the model on the test data based on the selected scoring method"""
+        return self.best_estimator_.score(X, y)
+
+    def optimization_time(self):
+        """Return the time taken for the optimization process"""
+        return self.study_time_

optuml-0.1.0/optuml.egg-info/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.1
+Name: optuml
+Version: 0.1.0
+Summary: Hyperparameter optimization for multiple machine learning algorithms using Optuna, with Scikit-learn API
+Home-page: https://github.com/filipsPL/optuml
+Author: Filip S.
+Author-email: filip.ursynow@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: optuna
+Requires-Dist: scikit-learn
+Requires-Dist: catboost
+Requires-Dist: xgboost
+Requires-Dist: numpy
+
+# OptuML: Hyperparameter Optimization for Multiple Machine Learning Algorithms using Optuna
+
+```
+  ⣰⡁ ⡀⣀ ⢀⡀ ⣀⣀    ⡎⢱ ⣀⡀ ⣰⡀ ⡀⢀ ⡷⢾ ⡇    ⠄ ⣀⣀  ⣀⡀ ⢀⡀ ⡀⣀ ⣰⡀   ⡎⢱ ⣀⡀ ⣰⡀ ⠄ ⣀⣀  ⠄ ⣀⣀ ⢀⡀ ⡀⣀
+  ⢸  ⠏  ⠣⠜ ⠇⠇⠇   ⠣⠜ ⡧⠜ ⠘⠤ ⠣⠼ ⠇⠸ ⠧⠤   ⠇ ⠇⠇⠇ ⡧⠜ ⠣⠜ ⠏  ⠘⠤   ⠣⠜ ⡧⠜ ⠘⠤ ⠇ ⠇⠇⠇ ⠇ ⠴⠥ ⠣⠭ ⠏ 
+```
+
+`OptuML` is a Python module that provides hyperparameter optimization for several machine learning algorithms using the [Optuna](https://optuna.org/) framework. The module supports a variety of algorithms and allows easy hyperparameter tuning through a scikit-learn-like API.
+
+[![Python tests](https://github.com/filipsPL/optuml/actions/workflows/python-package.yml/badge.svg)](https://github.com/filipsPL/optuml/actions/workflows/python-package.yml)
+
+- [OptuML: Hyperparameter Optimization for Multiple Machine Learning Algorithms using Optuna](#optuml-hyperparameter-optimization-for-multiple-machine-learning-algorithms-using-optuna)
+  - [Quick start](#quick-start)
+  - [Features](#features)
+  - [Installation](#installation)
+    - [Prerequisites](#prerequisites)
+    - [Installation](#installation-1)
+  - [Usage](#usage)
+    - [Basic Example](#basic-example)
+    - [Available Algorithms](#available-algorithms)
+    - [Example of Optimizing Different Algorithms](#example-of-optimizing-different-algorithms)
+    - [Custom Scoring and Direction](#custom-scoring-and-direction)
+    - [Controlling Verbosity](#controlling-verbosity)
+  - [API Reference](#api-reference)
+    - [`Optimizer`](#optimizer)
+      - [Parameters:](#parameters)
+      - [Methods:](#methods)
+  - [Contributing](#contributing)
+  - [Running Tests](#running-tests)
+  - [Contact](#contact)
+
+
+## Quick start
+
+```python
+from OptuML import Optimizer
+
+# Instantiate the optimizer for SVM
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict on the test set
+y_pred = optimizer.predict(X_test)
+```
+
+
+## Features
+
+- **Multiple Algorithms**: Supports hyperparameter optimization for the following algorithms:
+  - Support Vector Machine (SVM)
+  - k-Nearest Neighbors (kNN)
+  - Random Forest
+  - CatBoost
+  - XGBoost
+  - Logistic Regression
+  - Decision Tree
+- **Optuna Framework**: Leverages Optuna for powerful hyperparameter search.
+- **Maximize or Minimize**: Allows setting the optimization direction (`maximize` or `minimize`).
+- **Scikit-learn API**: Provides a consistent interface for `fit()`, `predict()`, `predict_proba()`, and `score()` methods.
+- **Control Output**: Optionally run Optuna with verbose logging (`verbose=True`) or in silent mode (`verbose=False`).
+- **Cross-validation**: Easily integrate cross-validation with custom scoring metrics (e.g., accuracy, ROC AUC).
+
+## Installation
+
+### Prerequisites
+
+Ensure that the following Python packages are installed:
+
+- `optuna`
+- `scikit-learn`
+- `catboost`
+- `xgboost`
+- `numpy`
+
+You can install the required packages via `pip`:
+
+```bash
+pip install optuna scikit-learn catboost xgboost numpy
+```
+
+### Installation
+
+Just fetch the `optuml.py` [file from the repo](optuml.py) and put it in the directory with your script.
+
+
+## Usage
+
+### Basic Example
+
+Here’s how you can use the `Optimizer` class to optimize hyperparameters for different machine learning algorithms using the **Iris** dataset:
+
+```python
+from sklearn.datasets import load_iris
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+from OptuML import Optimizer
+
+# Load the Iris dataset
+X, y = load_iris(return_X_y=True)
+
+# Convert to a binary classification problem
+X = X[y != 2]
+y = y[y != 2]
+
+# Split the dataset into training and test sets
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
+
+# Instantiate the optimizer for SVM
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True, random_state=42)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict on the test set
+y_pred = optimizer.predict(X_test)
+
+# Calculate the accuracy
+accuracy = accuracy_score(y_test, y_pred)
+print(f"Accuracy: {accuracy}")
+
+# Print the best hyperparameters found during optimization
+print(f"Best Hyperparameters: {optimizer.best_params_}")
+```
+
+### Available Algorithms
+
+The `Optimizer` class supports the following algorithms. You can specify the `algorithm` parameter to choose which one to use:
+
+- `"SVM"` (Support Vector Machine)
+- `"kNN"` (k-Nearest Neighbors)
+- `"RandomForest"` (Random Forest)
+- `"CatBoost"` (CatBoost Classifier)
+- `"XGBoost"` (XGBoost Classifier)
+- `"LogisticRegression"` (Logistic Regression)
+- `"DecisionTree"` (Decision Tree Classifier)
+
+### Example of Optimizing Different Algorithms
+
+You can iterate over different algorithms and optimize hyperparameters for each one:
+
+```python
+algorithms = ['SVM', 'kNN', 'RandomForest', 'CatBoost', 'XGBoost', 'LogisticRegression', 'DecisionTree']
+
+for algorithm in algorithms:
+    print(f"Optimizing {algorithm}")
+    optimizer = Optimizer(algorithm=algorithm, n_trials=50, cv=3, scoring="accuracy", verbose=False, random_state=42)
+    optimizer.fit(X_train, y_train)
+    
+    # Predict and calculate accuracy
+    y_pred = optimizer.predict(X_test)
+    accuracy = accuracy_score(y_test, y_pred)
+    print(f"Accuracy for {algorithm}: {accuracy}")
+    print(f"Best Hyperparameters for {algorithm}: {optimizer.best_params_}")
+    print("="*60)
+```
+
+### Custom Scoring and Direction
+
+You can also optimize for different scoring metrics like ROC AUC or use the `direction` parameter to **minimize** or **maximize** the objective:
+
+```python
+# Instantiate the optimizer for RandomForest with ROC AUC optimization
+optimizer = Optimizer(algorithm="RandomForest", n_trials=50, cv=3, scoring="roc_auc", direction="maximize", verbose=True, random_state=42)
+
+# Fit the optimizer to the training data
+optimizer.fit(X_train, y_train)
+
+# Predict probabilities and calculate ROC AUC
+y_proba = optimizer.predict_proba(X_test)
+roc_auc = roc_auc_score(y_test, y_proba[:, 1])
+print(f"ROC AUC: {roc_auc}")
+
+# Print the best hyperparameters
+print(f"Best Hyperparameters: {optimizer.best_params_}")
+```
+
+### Controlling Verbosity
+
+You can control the verbosity of Optuna's output by using the `verbose` parameter:
+
+- Set `verbose=True` to enable detailed Optuna logging.
+- Set `verbose=False` to suppress logging and run the optimizer silently.
+
+```python
+optimizer = Optimizer(algorithm="SVM", n_trials=50, cv=3, scoring="accuracy", verbose=True, random_state=42)
+```
+
+## API Reference
+
+### `Optimizer`
+
+#### Parameters:
+
+- **`algorithm`** (`str`): The machine learning algorithm to optimize. Options are `'SVM'`, `'kNN'`, `'RandomForest'`, `'CatBoost'`, `'XGBoost'`, `'LogisticRegression'`, `'DecisionTree'`.
+- **`direction`** (`str`, default `"maximize"`): Direction of optimization. Can be `"maximize"` or `"minimize"`.
+- **`verbose`** (`bool`, default `False`): If `True`, Optuna logging will be enabled. If `False`, the optimizer will run silently.
+- **`n_trials`** (`int`, default `100`): Number of optimization trials to run.
+- **`timeout`** (`float`, optional): Maximum time (in seconds) for the optimization process.
+- **`cv`** (`int`, default `5`): Number of cross-validation folds.
+- **`scoring`** (`str`, default `"accuracy"`): Scoring metric to use during cross-validation.
+- **`random_state`** (`int`, optional): Seed for random number generation.
+
+#### Methods:
+
+- **`fit(X, y)`**: Fit the model using hyperparameter optimization.
+- **`predict(X)`**: Make predictions using the best model found during optimization.
+- **`predict_proba(X)`**: Predict class probabilities (if supported by the model).
+- **`score(X, y)`**: Score the model using the test data.
+- **`optimization_time()`**: Get the total time taken for optimization.
+
+## Contributing
+
+Contributions to the project are welcome! Please feel free to submit issues or pull requests on GitHub. You can also fork the repository and make your changes.
+
+## Running Tests
+
+You can run unit tests using `pytest`:
+
+```bash
+pip install pytest
+pytest
+```
+
+## Contact
+
+If you have any questions or feedback, feel free to open an issue on GitHub.

optuml-0.1.0/optuml.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.md
+pyproject.toml
+setup.py
+optimum/__init__.py
+optimum/optuml.py
+optuml.egg-info/PKG-INFO
+optuml.egg-info/SOURCES.txt
+optuml.egg-info/dependency_links.txt
+optuml.egg-info/requires.txt
+optuml.egg-info/top_level.txt
+tests/__init__.py
+tests/test_optimizer.py

optuml-0.1.0/optuml.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

optuml-0.1.0/optuml.egg-info/requires.txt

@@ -0,0 +1,5 @@
+optuna
+scikit-learn
+catboost
+xgboost
+numpy

optuml-0.1.0/optuml.egg-info/top_level.txt

@@ -0,0 +1,2 @@
+optimum
+tests

optuml-0.1.0/pyproject.toml

@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"

optuml-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

optuml-0.1.0/setup.py

@@ -0,0 +1,29 @@
+from setuptools import setup, find_packages
+
+with open("README.md", "r", encoding="utf-8") as fh:
+    long_description = fh.read()
+
+setup(
+    name="optuml",  # The name of your package on PyPI
+    version="0.1.0",  # Start with 0.1.0 for the initial release
+    author="Filip S.",
+    author_email="filip.ursynow@gmail.com",
+    description="Hyperparameter optimization for multiple machine learning algorithms using Optuna, with Scikit-learn API",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/filipsPL/optuml",  # URL to the package's homepage (e.g., GitHub)
+    packages=find_packages(),
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+    ],
+    python_requires=">=3.6",
+    install_requires=[
+        "optuna",
+        "scikit-learn",
+        "catboost",
+        "xgboost",
+        "numpy"
+    ],
+)

optuml-0.1.0/tests/test_optimizer.py

@@ -0,0 +1,108 @@
+import pytest
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score, roc_auc_score
+from optuml import Optimizer
+
+@pytest.fixture
+def binary_classification_data():
+    """Generate a synthetic binary classification dataset."""
+    X, y = make_classification(n_samples=300, n_features=5, n_classes=2, random_state=42)
+    return train_test_split(X, y, test_size=0.3, random_state=42)
+
+@pytest.mark.parametrize("algorithm", ['SVM', 'kNN', 'RandomForest', 'CatBoost', 'XGBoost', 'LogisticRegression', 'DecisionTree'])
+def test_optimizer_accuracy(binary_classification_data, algorithm):
+    """Test accuracy optimization for each supported algorithm."""
+    X_train, X_test, y_train, y_test = binary_classification_data
+
+    # Initialize Optimizer with accuracy optimization
+    optimizer = Optimizer(algorithm=algorithm, direction="maximize", verbose=False, n_trials=10, cv=3, scoring="accuracy", random_state=42)
+    
+    # Fit the model
+    optimizer.fit(X_train, y_train)
+
+    # Predict and calculate accuracy
+    y_pred = optimizer.predict(X_test)
+    accuracy = accuracy_score(y_test, y_pred)
+
+    # Ensure that accuracy is above a reasonable threshold (e.g., 0.7 for this dataset)
+    assert accuracy > 0.7, f"Accuracy for {algorithm} is too low: {accuracy}"
+
+    # Check if best_params_ is populated
+    assert optimizer.best_params_ is not None, f"Best parameters for {algorithm} were not found"
+
+    # Check if optimization time was recorded
+    assert optimizer.optimization_time() > 0, f"Optimization time was not recorded for {algorithm}"
+
+@pytest.mark.parametrize("algorithm", ['SVM', 'RandomForest', 'CatBoost', 'XGBoost'])
+def test_optimizer_roc_auc(binary_classification_data, algorithm):
+    """Test ROC AUC optimization for selected algorithms that support predict_proba."""
+    X_train, X_test, y_train, y_test = binary_classification_data
+
+    # Initialize Optimizer with ROC AUC optimization
+    optimizer = Optimizer(algorithm=algorithm, direction="maximize", verbose=False, n_trials=10, cv=3, scoring="roc_auc", random_state=42)
+    
+    # Fit the model
+    optimizer.fit(X_train, y_train)
+
+    # Predict probabilities and calculate ROC AUC
+    y_proba = optimizer.predict_proba(X_test)
+    roc_auc = roc_auc_score(y_test, y_proba[:, 1])
+
+    # Ensure that ROC AUC is above a reasonable threshold (e.g., 0.7 for this dataset)
+    assert roc_auc > 0.7, f"ROC AUC for {algorithm} is too low: {roc_auc}"
+
+    # Check if best_params_ is populated
+    assert optimizer.best_params_ is not None, f"Best parameters for {algorithm} were not found"
+
+    # Check if optimization time was recorded
+    assert optimizer.optimization_time() > 0, f"Optimization time was not recorded for {algorithm}"
+
+def test_optimizer_direction_minimize(binary_classification_data):
+    """Test the 'minimize' direction in the optimizer for accuracy."""
+    X_train, X_test, y_train, y_test = binary_classification_data
+
+    # Initialize Optimizer to minimize accuracy (artificial test, not practical)
+    optimizer = Optimizer(algorithm="SVM", direction="minimize", verbose=False, n_trials=10, cv=3, scoring="accuracy", random_state=42)
+    
+    # Fit the model
+    optimizer.fit(X_train, y_train)
+
+    # Predict and calculate accuracy
+    y_pred = optimizer.predict(X_test)
+    accuracy = accuracy_score(y_test, y_pred)
+
+    # Check that the minimization didn't yield absurdly high accuracy (should be reasonable)
+    assert accuracy < 1.0, f"Minimization should not result in perfect accuracy: {accuracy}"
+
+# def test_optimizer_verbose_flag(binary_classification_data, capsys):
+#     """Test that verbose flag controls Optuna logging output."""
+#     X_train, X_test, y_train, y_test = binary_classification_data
+
+#     # Initialize Optimizer with verbose=True
+#     optimizer = Optimizer(algorithm="SVM", direction="maximize", verbose=True, n_trials=5, cv=3, scoring="accuracy", random_state=42)
+    
+#     # Fit the model
+#     optimizer.fit(X_train, y_train)
+
+#     # Capture the stdout output
+#     captured = capsys.readouterr()
+
+#     # Check that the output contains Optuna log info (verbose=True should produce output)
+#     assert "Trial" in captured.out, "Verbose output should include trial information but it was missing."
+
+# def test_optimizer_silent_mode(binary_classification_data, capsys):
+#     """Test that the optimizer runs silently when verbose=False."""
+#     X_train, X_test, y_train, y_test = binary_classification_data
+
+#     # Initialize Optimizer with verbose=False
+#     optimizer = Optimizer(algorithm="SVM", direction="maximize", verbose=False, n_trials=5, cv=3, scoring="accuracy", random_state=42)
+    
+#     # Fit the model
+#     optimizer.fit(X_train, y_train)
+
+#     # Capture the stdout output
+#     captured = capsys.readouterr()
+
+#     # Check that the output does not contain Optuna log info (verbose=False should produce no output)
+#     assert "Trial" not in captured.out, "Verbose output should be suppressed but it was displayed."