bayesian-gp-cvloss 0.1.1__tar.gz

bayesian_gp_cvloss-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Shifa Zhong
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

bayesian_gp_cvloss-0.1.1/PKG-INFO

@@ -0,0 +1,234 @@
+Metadata-Version: 2.4
+Name: bayesian-gp-cvloss
+Version: 0.1.1
+Summary: A Python package for Gaussian Process Regression with hyperparameter optimization using Hyperopt and cross-validation, focusing on optimizing cross-validated loss.
+Home-page: https://github.com/Shifa-Zhong/bayesian-gp-cvloss
+Author: Shifa Zhong
+Author-email: sfzhong@tongji.edu.cn
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: gpflow>=2.0.0
+Requires-Dist: hyperopt>=0.2.0
+Requires-Dist: scikit-learn>=0.23.0
+Requires-Dist: pandas>=1.0.0
+Requires-Dist: numpy>=1.18.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Bayesian GP CVLoss: Gaussian Process Regression with Cross-Validated Hyperparameter Optimization
+
+[![PyPI version](https://badge.fury.io/py/bayesian-gp-cvloss.svg)](https://badge.fury.io/py/bayesian-gp-cvloss) <!-- Placeholder for PyPI badge -->
+
+`bayesian_gp_cvloss` is a Python package designed to simplify the process of training Gaussian Process (GP) models by finding optimal hyperparameters through Bayesian optimization (using Hyperopt) with k-fold cross-validation. The key feature of this package is its direct optimization of the cross-validated Root Mean Squared Error (RMSE), aligning the hyperparameter tuning process closely with the model's predictive performance.
+
+This package is particularly useful for researchers and practitioners who want to apply GP models without manually tuning hyperparameters or relying solely on maximizing marginal likelihood, offering a more direct approach to achieving good generalization on unseen data.
+
+## Core Idea
+
+The traditional approach to training GP models often involves maximizing the log marginal likelihood of the model parameters. While effective, this doesn't always directly translate to the best predictive performance on unseen data, especially when the model assumptions are not perfectly met or when working with smaller datasets.
+
+This library implements an alternative strategy:
+
+1.  **Define a search space** for the GP kernel parameters (e.g., length scales, kernel variance) and likelihood parameters (e.g., noise variance).
+2.  Use **Bayesian optimization (Hyperopt)** to intelligently search this space.
+3.  For each set of hyperparameters evaluated by Hyperopt, perform **k-fold cross-validation** on the training data.
+4.  The **objective function** for Hyperopt is the mean RMSE across these k folds.
+5.  The set of hyperparameters yielding the **minimum average cross-validated RMSE** is selected as optimal.
+6.  A final GP model is then **refitted on the entire training dataset** using these best-found hyperparameters.
+
+This method directly targets the minimization of prediction error, which can be a more robust approach for many real-world regression tasks.
+
+## Features
+
+*   Automated hyperparameter optimization for GP models using Hyperopt.
+*   Cross-validation (k-fold) integrated into the optimization loop to find parameters that generalize well.
+*   Directly optimizes for mean cross-validated RMSE.
+*   Supports various GPflow kernels (e.g., RBF, Matern32, Matern52, RationalQuadratic by default, easily extensible).
+*   Data-dependent default hyperparameter search space generation based on the target variable's statistics.
+*   Handles mean centering of the target variable internally for potentially improved stability.
+*   Simple API: provide your preprocessed numerical `X_train` and `y_train` data.
+
+## Installation
+
+```bash
+# Coming soon to PyPI!
+# pip install bayesian-gp-cvloss
+# For now, you can install from source if you have the code:
+# pip install .
+```
+
+## Dependencies
+
+*   gpflow >= 2.0.0
+*   hyperopt >= 0.2.0
+*   scikit-learn >= 0.23.0
+*   pandas >= 1.0.0
+*   numpy >= 1.18.0
+
+Users are responsible for their own data preprocessing (e.g., encoding categorical features, feature scaling) before using this library. The optimizer expects purely numerical `X_train` and `y_train` inputs.
+
+## Quick Start
+
+```python
+import numpy as np
+import pandas as pd
+from sklearn.preprocessing import StandardScaler
+from sklearn.model_selection import train_test_split
+
+from bayesian_gp_cvloss.optimizer import GPCrossValidatedOptimizer
+
+# 0. (User Responsibility) Load and Preprocess Data
+# Example: Assume you have X (features) and y (target) as pandas DataFrames/Series
+# Ensure X is purely numerical. All encoding and scaling is up to the user.
+
+# Let's create some synthetic data for demonstration
+np.random.seed(42)
+N_train = 100
+N_features = 3
+X_synth = np.random.rand(N_train, N_features)
+# A simple function for y with some noise
+y_synth = np.sin(X_synth[:, 0] * 2 * np.pi) + X_synth[:, 1]**2 + np.random.randn(N_train) * 0.1
+
+# Convert to pandas DataFrame/Series if your data isn't already
+X_df = pd.DataFrame(X_synth, columns=[f'feature_{i}' for i in range(N_features)])
+y_series = pd.Series(y_synth, name='target')
+
+# Split data (optional, but good practice to have a final test set)
+# The optimizer does its own CV on the X_train_opt, y_train_opt
+X_train_data, X_test_data, y_train_data, y_test_data = train_test_split(
+    X_df, y_series, test_size=0.2, random_state=42
+)
+
+# Scale features (example - user should choose appropriate scaling)
+scaler = StandardScaler()
+X_train_scaled = scaler.fit_transform(X_train_data)
+X_test_scaled = scaler.transform(X_test_data)
+
+# Optimizer expects numpy arrays
+X_train_np = X_train_scaled
+y_train_np = y_train_data.values
+
+X_test_np = X_test_scaled
+y_test_np = y_test_data.values
+
+# 1. Initialize the Optimizer
+# You can specify kernels, number of folds, max_evals for Hyperopt, etc.
+optimizer = GPCrossValidatedOptimizer(
+    n_folds=5, 
+    max_evals=50, # Number of Hyperopt trials (increase for better results)
+    random_state_hyperopt=42
+)
+
+# 2. Run Optimization
+# This will find the best hyperparameters based on cross-validated RMSE
+# using X_train_np and y_train_np
+optimizer.optimize(X_train_np, y_train_np)
+
+print(f"Best hyperparameters found: {optimizer.best_params}")
+print(f"Best CV validation RMSE: {optimizer.best_cv_val_rmse_}")
+print(f"Best CV train RMSE: {optimizer.best_cv_train_rmse_}")
+
+# 3. Get the Refitted Model (Optional)
+# The optimizer automatically refits a model on the full X_train_np, y_train_np 
+# using the best hyperparameters. You can access it if needed.
+# best_gpr_model = optimizer.get_refitted_model()
+# print_summary(best_gpr_model) # Requires gpflow
+
+# 4. Make Predictions
+# The predict method uses the refitted model.
+# Input to predict should be preprocessed in the same way as X_train_np
+y_pred_test, y_pred_var_test = optimizer.predict(X_test_np)
+
+# Evaluate (example)
+from sklearn.metrics import mean_squared_error
+rmse_test = np.sqrt(mean_squared_error(y_test_np, y_pred_test))
+print(f"Test RMSE: {rmse_test}")
+
+# Plot results (example)
+import matplotlib.pyplot as plt
+plt.figure(figsize=(8, 6))
+plt.scatter(y_test_np, y_pred_test, alpha=0.7, label='Test Predictions')
+plt.plot([min(y_test_np), max(y_test_np)], [min(y_test_np), max(y_test_np)], 'r--', label='Ideal')
+plt.xlabel("True Values")
+plt.ylabel("Predicted Values")
+plt.title("GPR Predictions vs True Values on Test Set")
+plt.legend()
+plt.grid(True)
+plt.show()
+
+```
+
+## How it Works Internally
+
+1.  **`__init__(...)`**: Initializes settings like number of folds (`n_folds`), Hyperopt maximum evaluations (`max_evals`), desired kernels (`kernels_to_try`), and random states.
+2.  **`optimize(X_train, y_train)`**:
+    *   Stores `X_train` and `y_train`.
+    *   Calculates `self.y_train_mean_` for internal centering.
+    *   Calls `_get_default_data_dependent_space(y_train)` to define the hyperparameter search space for Hyperopt. This space is dynamically adjusted based on the variance and standard deviation of `y_train` to provide sensible default ranges for kernel variance and likelihood noise.
+    *   Initializes `hyperopt.Trials()`.
+    *   Runs `hyperopt.fmin()` with the `_objective` function, the defined `space`, `tpe.suggest` algorithm, and `max_evals`.
+    *   Stores the best parameters (`self.best_params`), best cross-validated validation RMSE (`self.best_cv_val_rmse_`), and corresponding training RMSE (`self.best_cv_train_rmse_`).
+    *   Calls `refit_best_model(X_train, y_train)` to train a final GPR model on the full training data using `self.best_params`.
+3.  **`_objective(params)`**:
+    *   This is the function minimized by Hyperopt.
+    *   It takes a dictionary of `params` (hyperparameters for a single trial).
+    *   Performs k-fold cross-validation:
+        *   For each fold, splits `X_train`, `y_train` into `X_train_fold`, `y_train_fold` and `X_val_fold`, `y_val_fold`.
+        *   **Important**: `y_train_fold` and `y_val_fold` are centered by subtracting the mean of the *current* `y_train_fold`.
+        *   Constructs a GPflow GPR model using the hyperparameters from `params` and the current fold's training data.
+        *   Predicts on `X_val_fold` and calculates RMSE.
+    *   Averages the RMSEs from all validation folds.
+    *   Returns a dictionary including `{'loss': avg_val_rmse, 'status': STATUS_OK, ...}`.
+4.  **`_get_default_data_dependent_space(y_train)`**: 
+    *   Defines the search space for Hyperopt for each hyperparameter:
+        *   `lengthscales`: `hp.quniform` between 1 and 100 (step 0.01) for each input dimension.
+        *   `kernel_variance`: `hp.uniform` between 0 and `y_train.var()`.
+        *   `likelihood_noise_variance`: `hp.loguniform` between `(y_train.std()/100)**2` and `(y_train.std()/2)**2` (with safety checks for small/zero std dev).
+        *   `kernel_class`: `hp.choice` among the kernels specified in `self.kernels_to_try`.
+5.  **`refit_best_model(X_data_refit, y_data_refit)`**:
+    *   Trains a new GPflow GPR model using `self.best_params` on the *entire* `X_data_refit` and `y_data_refit` (which are centered using `self.y_train_mean_`).
+    *   Stores this model as `self.final_model_`.
+6.  **`predict(X_new)`**:
+    *   Takes new, preprocessed data `X_new`.
+    *   Uses `self.final_model_` to predict mean and variance.
+    *   Adds back `self.y_train_mean_` to the predicted mean to return predictions on the original scale.
+
+## Customization
+
+*   **Kernels**: Pass a list of GPflow kernel classes to the `kernels_to_try` argument in the `GPCrossValidatedOptimizer` constructor (e.g., `[gpflow.kernels.Matern52, gpflow.kernels.RBF]`).
+*   **Hyperparameter Space**: While a data-dependent default space is provided, you can supply your own `hyperopt_space` dictionary to the `optimize` method if you need finer control or different distributions for hyperparameters.
+*   **Cross-Validation**: Change `n_folds` and `random_state_kfold`.
+*   **Hyperopt**: Adjust `max_evals` and `random_state_hyperopt`.
+
+## Contributing
+
+Contributions are welcome! If you have suggestions for improvements or find any issues, please open an issue or submit a pull request to the GitHub repository: https://github.com/Shifa-Zhong/bayesian-gp-cvloss
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Author
+
+Shifa Zhong (sfzhong@tongji.edu.cn)
+GitHub: [Shifa-Zhong](https://github.com/Shifa-Zhong) 

bayesian_gp_cvloss-0.1.1/README.md

@@ -0,0 +1,197 @@
+# Bayesian GP CVLoss: Gaussian Process Regression with Cross-Validated Hyperparameter Optimization
+
+[![PyPI version](https://badge.fury.io/py/bayesian-gp-cvloss.svg)](https://badge.fury.io/py/bayesian-gp-cvloss) <!-- Placeholder for PyPI badge -->
+
+`bayesian_gp_cvloss` is a Python package designed to simplify the process of training Gaussian Process (GP) models by finding optimal hyperparameters through Bayesian optimization (using Hyperopt) with k-fold cross-validation. The key feature of this package is its direct optimization of the cross-validated Root Mean Squared Error (RMSE), aligning the hyperparameter tuning process closely with the model's predictive performance.
+
+This package is particularly useful for researchers and practitioners who want to apply GP models without manually tuning hyperparameters or relying solely on maximizing marginal likelihood, offering a more direct approach to achieving good generalization on unseen data.
+
+## Core Idea
+
+The traditional approach to training GP models often involves maximizing the log marginal likelihood of the model parameters. While effective, this doesn't always directly translate to the best predictive performance on unseen data, especially when the model assumptions are not perfectly met or when working with smaller datasets.
+
+This library implements an alternative strategy:
+
+1.  **Define a search space** for the GP kernel parameters (e.g., length scales, kernel variance) and likelihood parameters (e.g., noise variance).
+2.  Use **Bayesian optimization (Hyperopt)** to intelligently search this space.
+3.  For each set of hyperparameters evaluated by Hyperopt, perform **k-fold cross-validation** on the training data.
+4.  The **objective function** for Hyperopt is the mean RMSE across these k folds.
+5.  The set of hyperparameters yielding the **minimum average cross-validated RMSE** is selected as optimal.
+6.  A final GP model is then **refitted on the entire training dataset** using these best-found hyperparameters.
+
+This method directly targets the minimization of prediction error, which can be a more robust approach for many real-world regression tasks.
+
+## Features
+
+*   Automated hyperparameter optimization for GP models using Hyperopt.
+*   Cross-validation (k-fold) integrated into the optimization loop to find parameters that generalize well.
+*   Directly optimizes for mean cross-validated RMSE.
+*   Supports various GPflow kernels (e.g., RBF, Matern32, Matern52, RationalQuadratic by default, easily extensible).
+*   Data-dependent default hyperparameter search space generation based on the target variable's statistics.
+*   Handles mean centering of the target variable internally for potentially improved stability.
+*   Simple API: provide your preprocessed numerical `X_train` and `y_train` data.
+
+## Installation
+
+```bash
+# Coming soon to PyPI!
+# pip install bayesian-gp-cvloss
+# For now, you can install from source if you have the code:
+# pip install .
+```
+
+## Dependencies
+
+*   gpflow >= 2.0.0
+*   hyperopt >= 0.2.0
+*   scikit-learn >= 0.23.0
+*   pandas >= 1.0.0
+*   numpy >= 1.18.0
+
+Users are responsible for their own data preprocessing (e.g., encoding categorical features, feature scaling) before using this library. The optimizer expects purely numerical `X_train` and `y_train` inputs.
+
+## Quick Start
+
+```python
+import numpy as np
+import pandas as pd
+from sklearn.preprocessing import StandardScaler
+from sklearn.model_selection import train_test_split
+
+from bayesian_gp_cvloss.optimizer import GPCrossValidatedOptimizer
+
+# 0. (User Responsibility) Load and Preprocess Data
+# Example: Assume you have X (features) and y (target) as pandas DataFrames/Series
+# Ensure X is purely numerical. All encoding and scaling is up to the user.
+
+# Let's create some synthetic data for demonstration
+np.random.seed(42)
+N_train = 100
+N_features = 3
+X_synth = np.random.rand(N_train, N_features)
+# A simple function for y with some noise
+y_synth = np.sin(X_synth[:, 0] * 2 * np.pi) + X_synth[:, 1]**2 + np.random.randn(N_train) * 0.1
+
+# Convert to pandas DataFrame/Series if your data isn't already
+X_df = pd.DataFrame(X_synth, columns=[f'feature_{i}' for i in range(N_features)])
+y_series = pd.Series(y_synth, name='target')
+
+# Split data (optional, but good practice to have a final test set)
+# The optimizer does its own CV on the X_train_opt, y_train_opt
+X_train_data, X_test_data, y_train_data, y_test_data = train_test_split(
+    X_df, y_series, test_size=0.2, random_state=42
+)
+
+# Scale features (example - user should choose appropriate scaling)
+scaler = StandardScaler()
+X_train_scaled = scaler.fit_transform(X_train_data)
+X_test_scaled = scaler.transform(X_test_data)
+
+# Optimizer expects numpy arrays
+X_train_np = X_train_scaled
+y_train_np = y_train_data.values
+
+X_test_np = X_test_scaled
+y_test_np = y_test_data.values
+
+# 1. Initialize the Optimizer
+# You can specify kernels, number of folds, max_evals for Hyperopt, etc.
+optimizer = GPCrossValidatedOptimizer(
+    n_folds=5, 
+    max_evals=50, # Number of Hyperopt trials (increase for better results)
+    random_state_hyperopt=42
+)
+
+# 2. Run Optimization
+# This will find the best hyperparameters based on cross-validated RMSE
+# using X_train_np and y_train_np
+optimizer.optimize(X_train_np, y_train_np)
+
+print(f"Best hyperparameters found: {optimizer.best_params}")
+print(f"Best CV validation RMSE: {optimizer.best_cv_val_rmse_}")
+print(f"Best CV train RMSE: {optimizer.best_cv_train_rmse_}")
+
+# 3. Get the Refitted Model (Optional)
+# The optimizer automatically refits a model on the full X_train_np, y_train_np 
+# using the best hyperparameters. You can access it if needed.
+# best_gpr_model = optimizer.get_refitted_model()
+# print_summary(best_gpr_model) # Requires gpflow
+
+# 4. Make Predictions
+# The predict method uses the refitted model.
+# Input to predict should be preprocessed in the same way as X_train_np
+y_pred_test, y_pred_var_test = optimizer.predict(X_test_np)
+
+# Evaluate (example)
+from sklearn.metrics import mean_squared_error
+rmse_test = np.sqrt(mean_squared_error(y_test_np, y_pred_test))
+print(f"Test RMSE: {rmse_test}")
+
+# Plot results (example)
+import matplotlib.pyplot as plt
+plt.figure(figsize=(8, 6))
+plt.scatter(y_test_np, y_pred_test, alpha=0.7, label='Test Predictions')
+plt.plot([min(y_test_np), max(y_test_np)], [min(y_test_np), max(y_test_np)], 'r--', label='Ideal')
+plt.xlabel("True Values")
+plt.ylabel("Predicted Values")
+plt.title("GPR Predictions vs True Values on Test Set")
+plt.legend()
+plt.grid(True)
+plt.show()
+
+```
+
+## How it Works Internally
+
+1.  **`__init__(...)`**: Initializes settings like number of folds (`n_folds`), Hyperopt maximum evaluations (`max_evals`), desired kernels (`kernels_to_try`), and random states.
+2.  **`optimize(X_train, y_train)`**:
+    *   Stores `X_train` and `y_train`.
+    *   Calculates `self.y_train_mean_` for internal centering.
+    *   Calls `_get_default_data_dependent_space(y_train)` to define the hyperparameter search space for Hyperopt. This space is dynamically adjusted based on the variance and standard deviation of `y_train` to provide sensible default ranges for kernel variance and likelihood noise.
+    *   Initializes `hyperopt.Trials()`.
+    *   Runs `hyperopt.fmin()` with the `_objective` function, the defined `space`, `tpe.suggest` algorithm, and `max_evals`.
+    *   Stores the best parameters (`self.best_params`), best cross-validated validation RMSE (`self.best_cv_val_rmse_`), and corresponding training RMSE (`self.best_cv_train_rmse_`).
+    *   Calls `refit_best_model(X_train, y_train)` to train a final GPR model on the full training data using `self.best_params`.
+3.  **`_objective(params)`**:
+    *   This is the function minimized by Hyperopt.
+    *   It takes a dictionary of `params` (hyperparameters for a single trial).
+    *   Performs k-fold cross-validation:
+        *   For each fold, splits `X_train`, `y_train` into `X_train_fold`, `y_train_fold` and `X_val_fold`, `y_val_fold`.
+        *   **Important**: `y_train_fold` and `y_val_fold` are centered by subtracting the mean of the *current* `y_train_fold`.
+        *   Constructs a GPflow GPR model using the hyperparameters from `params` and the current fold's training data.
+        *   Predicts on `X_val_fold` and calculates RMSE.
+    *   Averages the RMSEs from all validation folds.
+    *   Returns a dictionary including `{'loss': avg_val_rmse, 'status': STATUS_OK, ...}`.
+4.  **`_get_default_data_dependent_space(y_train)`**: 
+    *   Defines the search space for Hyperopt for each hyperparameter:
+        *   `lengthscales`: `hp.quniform` between 1 and 100 (step 0.01) for each input dimension.
+        *   `kernel_variance`: `hp.uniform` between 0 and `y_train.var()`.
+        *   `likelihood_noise_variance`: `hp.loguniform` between `(y_train.std()/100)**2` and `(y_train.std()/2)**2` (with safety checks for small/zero std dev).
+        *   `kernel_class`: `hp.choice` among the kernels specified in `self.kernels_to_try`.
+5.  **`refit_best_model(X_data_refit, y_data_refit)`**:
+    *   Trains a new GPflow GPR model using `self.best_params` on the *entire* `X_data_refit` and `y_data_refit` (which are centered using `self.y_train_mean_`).
+    *   Stores this model as `self.final_model_`.
+6.  **`predict(X_new)`**:
+    *   Takes new, preprocessed data `X_new`.
+    *   Uses `self.final_model_` to predict mean and variance.
+    *   Adds back `self.y_train_mean_` to the predicted mean to return predictions on the original scale.
+
+## Customization
+
+*   **Kernels**: Pass a list of GPflow kernel classes to the `kernels_to_try` argument in the `GPCrossValidatedOptimizer` constructor (e.g., `[gpflow.kernels.Matern52, gpflow.kernels.RBF]`).
+*   **Hyperparameter Space**: While a data-dependent default space is provided, you can supply your own `hyperopt_space` dictionary to the `optimize` method if you need finer control or different distributions for hyperparameters.
+*   **Cross-Validation**: Change `n_folds` and `random_state_kfold`.
+*   **Hyperopt**: Adjust `max_evals` and `random_state_hyperopt`.
+
+## Contributing
+
+Contributions are welcome! If you have suggestions for improvements or find any issues, please open an issue or submit a pull request to the GitHub repository: https://github.com/Shifa-Zhong/bayesian-gp-cvloss
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Author
+
+Shifa Zhong (sfzhong@tongji.edu.cn)
+GitHub: [Shifa-Zhong](https://github.com/Shifa-Zhong) 

bayesian_gp_cvloss-0.1.1/bayesian_gp_cvloss/__init__.py

@@ -0,0 +1,6 @@
+# This file makes Python treat the directory as a package.
+# It can also be used to expose parts of the package at the top level.
+
+from .optimizer import GPCrossValidatedOptimizer
+
+__all__ = ['GPCrossValidatedOptimizer'] 

bayesian_gp_cvloss-0.1.1/bayesian_gp_cvloss/optimizer.py

@@ -0,0 +1,287 @@
+import pandas as pd
+import numpy as np
+import gpflow
+from gpflow.utilities import set_trainable
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error
+# StandardScaler and JamesSteinEncoder are no longer direct dependencies for the class
+# but might be used by the user or the optional preprocessing utility.
+from hyperopt import fmin, tpe, hp, Trials, STATUS_OK
+import tensorflow as tf
+import logging
+
+# Configure basic logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+# Default GP Kernels from GPflow - user can specify these in the space
+DEFAULT_KERNELS = {
+    #"Matern12": gpflow.kernels.Matern12,
+    "Matern32": gpflow.kernels.Matern32,
+    "Matern52": gpflow.kernels.Matern52,
+    "RBF": gpflow.kernels.RBF,
+    "RationalQuadratic": gpflow.kernels.RationalQuadratic,
+    #"Exponential": gpflow.kernels.Exponential
+}
+
+class GPCrossValidatedOptimizer:
+    """
+    Optimizes hyperparameters for a Gaussian Process Regressor using Hyperopt
+    with k-fold cross-validation, minimizing RMSE. 
+    Assumes input data X_train and y_train are already preprocessed (numerical and scaled).
+    Generates a data-dependent default hyperparameter space if none is provided.
+    """
+    def __init__(self, X_train, y_train, 
+                 hyperopt_space=None, n_splits=5, random_state=None):
+        """
+        Args:
+            X_train (pd.DataFrame or np.ndarray): The preprocessed training feature dataset.
+            y_train (pd.Series or np.ndarray): The preprocessed training target variable.
+            hyperopt_space (dict, optional): Hyperopt search space. If None, a data-dependent default space is generated.
+            n_splits (int): Number of folds for KFold cross-validation.
+            random_state (int, optional): Random seed for KFold and Hyperopt for reproducibility.
+        """
+        if not isinstance(X_train, (pd.DataFrame, np.ndarray)):
+            raise ValueError("X_train must be a pandas DataFrame or NumPy ndarray.")
+        if not isinstance(y_train, (pd.Series, np.ndarray)):
+            raise ValueError("y_train must be a pandas Series or NumPy ndarray.")
+        
+        if isinstance(X_train, np.ndarray) and len(X_train.shape) != 2:
+             raise ValueError("X_train as NumPy array must be 2D.")
+        
+        _y_data_internal = y_train.values if isinstance(y_train, pd.Series) else np.asarray(y_train)
+        if len(_y_data_internal.shape) != 1 and not (len(_y_data_internal.shape) == 2 and _y_data_internal.shape[1] == 1):
+            raise ValueError("y_train must be 1D or 2D with one column.")
+        if len(_y_data_internal.shape) == 2:
+            _y_data_internal = _y_data_internal.flatten()
+
+        if X_train.shape[0] != _y_data_internal.shape[0]:
+            raise ValueError("X_train and y_train must have the same number of samples.")
+
+        self.X_train = X_train
+        self.y_train_1d = _y_data_internal # Store as 1D for var/std calculations and mean subtraction reference
+        self.y_train_mean_ = np.mean(self.y_train_1d) # Store the mean of the original y_train
+        self.n_splits = n_splits
+        self.random_state = random_state
+        
+        self.num_features = X_train.shape[1]
+        self.hyperopt_space = hyperopt_space if hyperopt_space is not None else self._get_default_data_dependent_space()
+        self.trials = Trials()
+        self.best_params = None
+        self.best_model_ = None
+        self._iteration_count = 0
+
+    def _get_default_data_dependent_space(self):
+        """Generates a data-dependent default hyperparameter search space."""
+        logger.info("Generating data-dependent default hyperparameter space.")
+        # Using y_train_1d which is y_train before any internal mean centering for GP models
+        space = {f'lengthscales_{i}': hp.quniform(f'lengthscales_{i}', 0.1, 100, 0.01) for i in range(self.num_features)}
+        
+        y_var = np.var(self.y_train_1d) 
+        y_std = np.std(self.y_train_1d)
+
+        kernel_var_upper = float(y_var)
+        kernel_var_lower = 0 
+        logger.info(f"Default kernel_variance range: ({kernel_var_lower:.2e}, {kernel_var_upper:.2e})")
+        space['kernel_variance'] = hp.uniform('kernel_variance', kernel_var_lower, kernel_var_upper)
+
+        if y_std > 1e-9:
+            noise_var_lower_bound = (y_std / 100.0)**2
+            noise_var_upper_bound = (y_std / 2.0)**2
+        else:
+            noise_var_lower_bound = 1e-9 
+            noise_var_upper_bound = 1e-2
+        
+        noise_var_lower_bound = max(1e-9, float(noise_var_lower_bound))
+        noise_var_upper_bound = max(noise_var_lower_bound * 1.1 + 1e-9, float(noise_var_upper_bound))
+
+        logger.info(f"Default likelihood_noise_variance loguniform range (effective): ({noise_var_lower_bound:.2e}, {noise_var_upper_bound:.2e})")
+        space['likelihood_noise_variance'] = hp.loguniform(
+            'likelihood_noise_variance', 
+            np.log(noise_var_lower_bound), 
+            np.log(noise_var_upper_bound)
+        )
+        
+        space['kernel_name'] = hp.choice('kernel_name', list(DEFAULT_KERNELS.keys()))
+        return space
+
+    def _objective(self, params):
+        self._iteration_count += 1
+        iteration_num = self._iteration_count
+
+        kf = KFold(n_splits=self.n_splits, shuffle=True, random_state=self.random_state)
+        fold_rmses = []
+        fold_train_rmses = []
+        
+        kernel_name = params['kernel_name']
+        selected_kernel_class = DEFAULT_KERNELS[kernel_name]
+        
+        kernel_hparams = {}
+        try:
+            lengthscales = np.array([params[f'lengthscales_{i}'] for i in range(self.num_features)], dtype=float)
+        except KeyError as e:
+            logger.error(f"Missing lengthscale param: {e}. Params: {params}")
+            return {'loss': np.inf, 'status': STATUS_OK, 'params': params, 'iteration': iteration_num}
+        
+        if not lengthscales.shape[0] == self.num_features:
+            logger.error(f"LS dim mismatch. Expected {self.num_features}, got {lengthscales.shape[0]}")
+            return {'loss': np.inf, 'status': STATUS_OK, 'params': params, 'iteration': iteration_num}
+
+        kernel_hparams['lengthscales'] = lengthscales
+        kernel_hparams['variance'] = float(params['kernel_variance'])
+        current_noise_variance = float(params['likelihood_noise_variance'])
+
+        X_data_for_cv = self.X_train.values if isinstance(self.X_train, pd.DataFrame) else self.X_train
+        y_data_1d_for_cv = self.y_train_1d 
+
+        for fold_idx, (train_index, val_index) in enumerate(kf.split(X_data_for_cv)):
+            X_train_fold, X_val_fold = X_data_for_cv[train_index], X_data_for_cv[val_index]
+            y_train_fold_1d, y_val_fold_1d = y_data_1d_for_cv[train_index], y_data_1d_for_cv[val_index]
+            
+            # Mean centering for this fold based on this fold's training y
+            current_fold_y_train_mean = np.mean(y_train_fold_1d)
+            y_train_fold_centered = y_train_fold_1d - current_fold_y_train_mean
+            y_val_fold_centered = y_val_fold_1d - current_fold_y_train_mean
+            
+            y_train_fold_2d = y_train_fold_centered.reshape(-1,1)
+            y_val_fold_2d = y_val_fold_centered.reshape(-1,1) # This is y_val - mean(y_train_fold)
+
+            try:
+                fold_kernel = selected_kernel_class(**kernel_hparams)
+                model = gpflow.models.GPR(data=(X_train_fold, y_train_fold_2d), 
+                                         kernel=fold_kernel, 
+                                         noise_variance=current_noise_variance)
+                
+                set_trainable(model.kernel.variance, False)
+                set_trainable(model.kernel.lengthscales, False)
+                set_trainable(model.likelihood.variance, False)
+
+                # Predictions are on the centered scale
+                y_pred_val_centered, _ = model.predict_y(X_val_fold)
+                y_pred_train_centered, _ = model.predict_y(X_train_fold)
+                
+                # RMSE is calculated on centered y_val and centered predictions
+                # OR add mean back to predictions and compare with original y_val_fold_1d
+                # The current notebook code subtracts fold_train_mean from y_val_fold_2d for training GP model, 
+                # so predictions are already centered around that. So RMSE on y_val_fold_centered is correct.
+                fold_rmse = np.sqrt(mean_squared_error(y_val_fold_centered, y_pred_val_centered.numpy()))
+                fold_train_rmse = np.sqrt(mean_squared_error(y_train_fold_centered, y_pred_train_centered.numpy()))
+                fold_rmses.append(fold_rmse)
+                fold_train_rmses.append(fold_train_rmse)
+            except Exception as e:
+                logger.warning(f"Fold {fold_idx+1} error for params {params}: {e}. High loss.")
+                fold_rmses.append(np.inf)
+                fold_train_rmses.append(np.inf)
+                break 
+        
+        avg_cv_rmse = np.mean(fold_rmses) if fold_rmses and np.all(np.isfinite(fold_rmses)) else np.inf
+        avg_train_rmse = np.mean(fold_train_rmses) if fold_train_rmses and np.all(np.isfinite(fold_train_rmses)) else np.inf
+        
+        ls_rounded = np.round(lengthscales,2)
+        logger.info(f"Iter: {iteration_num:>3} | CV RMSE: {avg_cv_rmse:<8.4f} | Train RMSE: {avg_train_rmse:<8.4f} | Kernel: {kernel_name} | Var: {kernel_hparams['variance']:.4f} | Noise: {current_noise_variance:.6f} | LS: {ls_rounded}")
+        
+        return {
+            'loss': avg_cv_rmse, 
+            'status': STATUS_OK, 
+            'params': params, 
+            'iteration': iteration_num,
+            'train_loss': avg_train_rmse
+        }
+
+    def optimize(self, max_evals=100, tpe_algo=tpe.suggest, early_stop_fn=None, rstate_seed=None):
+        self._iteration_count = 0 
+        if rstate_seed is None and self.random_state is not None:
+            rstate_seed = self.random_state
+        
+        rstate = np.random.default_rng(rstate_seed) if rstate_seed is not None else None
+
+        self.best_params_raw_ = fmin(
+            fn=self._objective, 
+            space=self.hyperopt_space, 
+            algo=tpe_algo, 
+            max_evals=max_evals, 
+            trials=self.trials,
+            early_stop_fn=early_stop_fn,
+            rstate=rstate
+        )
+        
+        logger.info(f"Optimization finished. Best raw params from fmin: {self.best_params_raw_}")
+        
+        if self.trials.best_trial and 'result' in self.trials.best_trial and self.trials.best_trial['result']['status'] == STATUS_OK:
+            self.best_params = self.trials.best_trial['result']['params']
+            logger.info(f"Best full params from trials: {self.best_params}")
+            logger.info(f"Best CV RMSE from trials: {self.trials.best_trial['result']['loss']}")
+            logger.info(f"Best CV Train RMSE from trials: {self.trials.best_trial['result']['train_loss']}")
+            self.refit_best_model()
+        else:
+            self.best_params = None
+            logger.warning("Optimization did not yield a valid best trial. Model not refitted.")
+
+        return self.best_params
+
+    def refit_best_model(self):
+        if not self.best_params:
+            logger.warning("No valid best parameters. Cannot refit model.")
+            self.best_model_ = None
+            return None
+
+        params_for_refit = self.best_params
+        kernel_name = params_for_refit['kernel_name']
+        selected_kernel_class = DEFAULT_KERNELS.get(kernel_name)
+
+        if selected_kernel_class is None:
+            logger.error(f"Kernel '{kernel_name}' not found. Cannot refit.")
+            self.best_model_ = None
+            return None
+            
+        kernel_hparams = {}
+        lengthscales = np.array([params_for_refit[f'lengthscales_{i}'] for i in range(self.num_features)], dtype=float)
+        kernel_hparams['lengthscales'] = lengthscales
+        kernel_hparams['variance'] = float(params_for_refit['kernel_variance'])
+        noise_var_refit = float(params_for_refit['likelihood_noise_variance'])
+
+        X_data_refit = self.X_train.values if isinstance(self.X_train, pd.DataFrame) else self.X_train
+        # For refitting, use the original y_train_1d and subtract its overall mean (self.y_train_mean_)
+        y_train_centered_for_refit = (self.y_train_1d - self.y_train_mean_).reshape(-1,1)
+
+        try:
+            best_kernel = selected_kernel_class(**kernel_hparams)
+            self.best_model_ = gpflow.models.GPR(
+                data=(X_data_refit, y_train_centered_for_refit),
+                kernel=best_kernel,
+                noise_variance=noise_var_refit
+            )
+            set_trainable(self.best_model_.kernel.variance, False)
+            set_trainable(self.best_model_.kernel.lengthscales, False)
+            set_trainable(self.best_model_.likelihood.variance, False)
+            logger.info(f"Successfully refitted GPR model: {params_for_refit}")
+        except Exception as e:
+            logger.error(f"Error refitting model with params {params_for_refit}: {e}")
+            self.best_model_ = None
+        return self.best_model_
+
+    def predict(self, X_new_processed):
+        if self.best_model_ is None:
+            logger.error("No best model. Run optimize() and ensure refit was successful.")
+            return None, None
+        if not isinstance(X_new_processed, (pd.DataFrame, np.ndarray)):
+            raise ValueError("X_new_processed must be pd.DataFrame or np.ndarray.")
+        
+        if X_new_processed.shape[1] != self.num_features:
+            raise ValueError(f"X_new has {X_new_processed.shape[1]} features, model expects {self.num_features}.")
+
+        X_new_values = X_new_processed.values if isinstance(X_new_processed, pd.DataFrame) else X_new_processed
+
+        try:
+            pred_mean_centered, pred_var = self.best_model_.predict_y(X_new_values)
+            # Add back the overall mean of the original y_train used for refitting
+            pred_mean_original_scale = pred_mean_centered.numpy() + self.y_train_mean_ 
+            return pred_mean_original_scale, pred_var.numpy()
+        except Exception as e:
+            logger.error(f"Error during prediction: {e}")
+            return None, None
+
+    def get_optimization_results(self):
+        return self.trials
+
+# End of GPCrossValidatedOptimizer class 

bayesian_gp_cvloss-0.1.1/bayesian_gp_cvloss.egg-info/PKG-INFO

@@ -0,0 +1,234 @@
+Metadata-Version: 2.4
+Name: bayesian-gp-cvloss
+Version: 0.1.1
+Summary: A Python package for Gaussian Process Regression with hyperparameter optimization using Hyperopt and cross-validation, focusing on optimizing cross-validated loss.
+Home-page: https://github.com/Shifa-Zhong/bayesian-gp-cvloss
+Author: Shifa Zhong
+Author-email: sfzhong@tongji.edu.cn
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: gpflow>=2.0.0
+Requires-Dist: hyperopt>=0.2.0
+Requires-Dist: scikit-learn>=0.23.0
+Requires-Dist: pandas>=1.0.0
+Requires-Dist: numpy>=1.18.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Bayesian GP CVLoss: Gaussian Process Regression with Cross-Validated Hyperparameter Optimization
+
+[![PyPI version](https://badge.fury.io/py/bayesian-gp-cvloss.svg)](https://badge.fury.io/py/bayesian-gp-cvloss) <!-- Placeholder for PyPI badge -->
+
+`bayesian_gp_cvloss` is a Python package designed to simplify the process of training Gaussian Process (GP) models by finding optimal hyperparameters through Bayesian optimization (using Hyperopt) with k-fold cross-validation. The key feature of this package is its direct optimization of the cross-validated Root Mean Squared Error (RMSE), aligning the hyperparameter tuning process closely with the model's predictive performance.
+
+This package is particularly useful for researchers and practitioners who want to apply GP models without manually tuning hyperparameters or relying solely on maximizing marginal likelihood, offering a more direct approach to achieving good generalization on unseen data.
+
+## Core Idea
+
+The traditional approach to training GP models often involves maximizing the log marginal likelihood of the model parameters. While effective, this doesn't always directly translate to the best predictive performance on unseen data, especially when the model assumptions are not perfectly met or when working with smaller datasets.
+
+This library implements an alternative strategy:
+
+1.  **Define a search space** for the GP kernel parameters (e.g., length scales, kernel variance) and likelihood parameters (e.g., noise variance).
+2.  Use **Bayesian optimization (Hyperopt)** to intelligently search this space.
+3.  For each set of hyperparameters evaluated by Hyperopt, perform **k-fold cross-validation** on the training data.
+4.  The **objective function** for Hyperopt is the mean RMSE across these k folds.
+5.  The set of hyperparameters yielding the **minimum average cross-validated RMSE** is selected as optimal.
+6.  A final GP model is then **refitted on the entire training dataset** using these best-found hyperparameters.
+
+This method directly targets the minimization of prediction error, which can be a more robust approach for many real-world regression tasks.
+
+## Features
+
+*   Automated hyperparameter optimization for GP models using Hyperopt.
+*   Cross-validation (k-fold) integrated into the optimization loop to find parameters that generalize well.
+*   Directly optimizes for mean cross-validated RMSE.
+*   Supports various GPflow kernels (e.g., RBF, Matern32, Matern52, RationalQuadratic by default, easily extensible).
+*   Data-dependent default hyperparameter search space generation based on the target variable's statistics.
+*   Handles mean centering of the target variable internally for potentially improved stability.
+*   Simple API: provide your preprocessed numerical `X_train` and `y_train` data.
+
+## Installation
+
+```bash
+# Coming soon to PyPI!
+# pip install bayesian-gp-cvloss
+# For now, you can install from source if you have the code:
+# pip install .
+```
+
+## Dependencies
+
+*   gpflow >= 2.0.0
+*   hyperopt >= 0.2.0
+*   scikit-learn >= 0.23.0
+*   pandas >= 1.0.0
+*   numpy >= 1.18.0
+
+Users are responsible for their own data preprocessing (e.g., encoding categorical features, feature scaling) before using this library. The optimizer expects purely numerical `X_train` and `y_train` inputs.
+
+## Quick Start
+
+```python
+import numpy as np
+import pandas as pd
+from sklearn.preprocessing import StandardScaler
+from sklearn.model_selection import train_test_split
+
+from bayesian_gp_cvloss.optimizer import GPCrossValidatedOptimizer
+
+# 0. (User Responsibility) Load and Preprocess Data
+# Example: Assume you have X (features) and y (target) as pandas DataFrames/Series
+# Ensure X is purely numerical. All encoding and scaling is up to the user.
+
+# Let's create some synthetic data for demonstration
+np.random.seed(42)
+N_train = 100
+N_features = 3
+X_synth = np.random.rand(N_train, N_features)
+# A simple function for y with some noise
+y_synth = np.sin(X_synth[:, 0] * 2 * np.pi) + X_synth[:, 1]**2 + np.random.randn(N_train) * 0.1
+
+# Convert to pandas DataFrame/Series if your data isn't already
+X_df = pd.DataFrame(X_synth, columns=[f'feature_{i}' for i in range(N_features)])
+y_series = pd.Series(y_synth, name='target')
+
+# Split data (optional, but good practice to have a final test set)
+# The optimizer does its own CV on the X_train_opt, y_train_opt
+X_train_data, X_test_data, y_train_data, y_test_data = train_test_split(
+    X_df, y_series, test_size=0.2, random_state=42
+)
+
+# Scale features (example - user should choose appropriate scaling)
+scaler = StandardScaler()
+X_train_scaled = scaler.fit_transform(X_train_data)
+X_test_scaled = scaler.transform(X_test_data)
+
+# Optimizer expects numpy arrays
+X_train_np = X_train_scaled
+y_train_np = y_train_data.values
+
+X_test_np = X_test_scaled
+y_test_np = y_test_data.values
+
+# 1. Initialize the Optimizer
+# You can specify kernels, number of folds, max_evals for Hyperopt, etc.
+optimizer = GPCrossValidatedOptimizer(
+    n_folds=5, 
+    max_evals=50, # Number of Hyperopt trials (increase for better results)
+    random_state_hyperopt=42
+)
+
+# 2. Run Optimization
+# This will find the best hyperparameters based on cross-validated RMSE
+# using X_train_np and y_train_np
+optimizer.optimize(X_train_np, y_train_np)
+
+print(f"Best hyperparameters found: {optimizer.best_params}")
+print(f"Best CV validation RMSE: {optimizer.best_cv_val_rmse_}")
+print(f"Best CV train RMSE: {optimizer.best_cv_train_rmse_}")
+
+# 3. Get the Refitted Model (Optional)
+# The optimizer automatically refits a model on the full X_train_np, y_train_np 
+# using the best hyperparameters. You can access it if needed.
+# best_gpr_model = optimizer.get_refitted_model()
+# print_summary(best_gpr_model) # Requires gpflow
+
+# 4. Make Predictions
+# The predict method uses the refitted model.
+# Input to predict should be preprocessed in the same way as X_train_np
+y_pred_test, y_pred_var_test = optimizer.predict(X_test_np)
+
+# Evaluate (example)
+from sklearn.metrics import mean_squared_error
+rmse_test = np.sqrt(mean_squared_error(y_test_np, y_pred_test))
+print(f"Test RMSE: {rmse_test}")
+
+# Plot results (example)
+import matplotlib.pyplot as plt
+plt.figure(figsize=(8, 6))
+plt.scatter(y_test_np, y_pred_test, alpha=0.7, label='Test Predictions')
+plt.plot([min(y_test_np), max(y_test_np)], [min(y_test_np), max(y_test_np)], 'r--', label='Ideal')
+plt.xlabel("True Values")
+plt.ylabel("Predicted Values")
+plt.title("GPR Predictions vs True Values on Test Set")
+plt.legend()
+plt.grid(True)
+plt.show()
+
+```
+
+## How it Works Internally
+
+1.  **`__init__(...)`**: Initializes settings like number of folds (`n_folds`), Hyperopt maximum evaluations (`max_evals`), desired kernels (`kernels_to_try`), and random states.
+2.  **`optimize(X_train, y_train)`**:
+    *   Stores `X_train` and `y_train`.
+    *   Calculates `self.y_train_mean_` for internal centering.
+    *   Calls `_get_default_data_dependent_space(y_train)` to define the hyperparameter search space for Hyperopt. This space is dynamically adjusted based on the variance and standard deviation of `y_train` to provide sensible default ranges for kernel variance and likelihood noise.
+    *   Initializes `hyperopt.Trials()`.
+    *   Runs `hyperopt.fmin()` with the `_objective` function, the defined `space`, `tpe.suggest` algorithm, and `max_evals`.
+    *   Stores the best parameters (`self.best_params`), best cross-validated validation RMSE (`self.best_cv_val_rmse_`), and corresponding training RMSE (`self.best_cv_train_rmse_`).
+    *   Calls `refit_best_model(X_train, y_train)` to train a final GPR model on the full training data using `self.best_params`.
+3.  **`_objective(params)`**:
+    *   This is the function minimized by Hyperopt.
+    *   It takes a dictionary of `params` (hyperparameters for a single trial).
+    *   Performs k-fold cross-validation:
+        *   For each fold, splits `X_train`, `y_train` into `X_train_fold`, `y_train_fold` and `X_val_fold`, `y_val_fold`.
+        *   **Important**: `y_train_fold` and `y_val_fold` are centered by subtracting the mean of the *current* `y_train_fold`.
+        *   Constructs a GPflow GPR model using the hyperparameters from `params` and the current fold's training data.
+        *   Predicts on `X_val_fold` and calculates RMSE.
+    *   Averages the RMSEs from all validation folds.
+    *   Returns a dictionary including `{'loss': avg_val_rmse, 'status': STATUS_OK, ...}`.
+4.  **`_get_default_data_dependent_space(y_train)`**: 
+    *   Defines the search space for Hyperopt for each hyperparameter:
+        *   `lengthscales`: `hp.quniform` between 1 and 100 (step 0.01) for each input dimension.
+        *   `kernel_variance`: `hp.uniform` between 0 and `y_train.var()`.
+        *   `likelihood_noise_variance`: `hp.loguniform` between `(y_train.std()/100)**2` and `(y_train.std()/2)**2` (with safety checks for small/zero std dev).
+        *   `kernel_class`: `hp.choice` among the kernels specified in `self.kernels_to_try`.
+5.  **`refit_best_model(X_data_refit, y_data_refit)`**:
+    *   Trains a new GPflow GPR model using `self.best_params` on the *entire* `X_data_refit` and `y_data_refit` (which are centered using `self.y_train_mean_`).
+    *   Stores this model as `self.final_model_`.
+6.  **`predict(X_new)`**:
+    *   Takes new, preprocessed data `X_new`.
+    *   Uses `self.final_model_` to predict mean and variance.
+    *   Adds back `self.y_train_mean_` to the predicted mean to return predictions on the original scale.
+
+## Customization
+
+*   **Kernels**: Pass a list of GPflow kernel classes to the `kernels_to_try` argument in the `GPCrossValidatedOptimizer` constructor (e.g., `[gpflow.kernels.Matern52, gpflow.kernels.RBF]`).
+*   **Hyperparameter Space**: While a data-dependent default space is provided, you can supply your own `hyperopt_space` dictionary to the `optimize` method if you need finer control or different distributions for hyperparameters.
+*   **Cross-Validation**: Change `n_folds` and `random_state_kfold`.
+*   **Hyperopt**: Adjust `max_evals` and `random_state_hyperopt`.
+
+## Contributing
+
+Contributions are welcome! If you have suggestions for improvements or find any issues, please open an issue or submit a pull request to the GitHub repository: https://github.com/Shifa-Zhong/bayesian-gp-cvloss
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Author
+
+Shifa Zhong (sfzhong@tongji.edu.cn)
+GitHub: [Shifa-Zhong](https://github.com/Shifa-Zhong) 

bayesian_gp_cvloss-0.1.1/bayesian_gp_cvloss.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+LICENSE
+README.md
+setup.py
+bayesian_gp_cvloss/__init__.py
+bayesian_gp_cvloss/optimizer.py
+bayesian_gp_cvloss.egg-info/PKG-INFO
+bayesian_gp_cvloss.egg-info/SOURCES.txt
+bayesian_gp_cvloss.egg-info/dependency_links.txt
+bayesian_gp_cvloss.egg-info/requires.txt
+bayesian_gp_cvloss.egg-info/top_level.txt

bayesian_gp_cvloss-0.1.1/bayesian_gp_cvloss.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

bayesian_gp_cvloss-0.1.1/bayesian_gp_cvloss.egg-info/requires.txt

@@ -0,0 +1,5 @@
+gpflow>=2.0.0
+hyperopt>=0.2.0
+scikit-learn>=0.23.0
+pandas>=1.0.0
+numpy>=1.18.0

bayesian_gp_cvloss-0.1.1/bayesian_gp_cvloss.egg-info/top_level.txt

@@ -0,0 +1 @@
+bayesian_gp_cvloss

bayesian_gp_cvloss-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

bayesian_gp_cvloss-0.1.1/setup.py

@@ -0,0 +1,34 @@
+from setuptools import setup, find_packages
+
+setup(
+    name='bayesian-gp-cvloss',
+    version='0.1.1',
+    author='Shifa Zhong',
+    author_email='sfzhong@tongji.edu.cn',
+    description='A Python package for Gaussian Process Regression with hyperparameter optimization using Hyperopt and cross-validation, focusing on optimizing cross-validated loss.',
+    long_description=open('README.md').read(),
+    long_description_content_type='text/markdown',
+    url='https://github.com/Shifa-Zhong/bayesian-gp-cvloss',
+    packages=find_packages(),
+    install_requires=[
+        'gpflow>=2.0.0',
+        'hyperopt>=0.2.0',
+        'scikit-learn>=0.23.0',
+        'pandas>=1.0.0',
+        'numpy>=1.18.0',
+    ],
+    classifiers=[
+        'Development Status :: 3 - Alpha',
+        'Intended Audience :: Science/Research',
+        'Intended Audience :: Developers',
+        'License :: OSI Approved :: MIT License',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Operating System :: OS Independent',
+        'Topic :: Scientific/Engineering :: Artificial Intelligence',
+        'Topic :: Software Development :: Libraries :: Python Modules',
+    ],
+    python_requires='>=3.8',
+)