robert 1.2.0__tar.gz → 1.2.1__tar.gz

{robert-1.2.0 → robert-1.2.1}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.1
 Name: robert
-Version: 1.2.0
+Version: 1.2.1
 Summary: Refiner and Optimizer of a Bunch of Existing Regression Tools
 Home-page: https://github.com/jvalegre/robert
-Download-URL: https://github.com/jvalegre/robert/archive/refs/tags/1.2.0.tar.gz
+Download-URL: https://github.com/jvalegre/robert/archive/refs/tags/1.2.1.tar.gz
 Author: Juan V. Alegre Requena, David Dalmau
 Author-email: jv.alegre@csic.es
 License: MIT

{robert-1.2.0 → robert-1.2.1}/README.md

@@ -7,15 +7,22 @@
 [![Codecov](https://img.shields.io/codecov/c/github/jvalegre/robert?label=Codecov&logo=codecov)](https://codecov.io/gh/jvalegre/robert)
 [![Downloads](https://img.shields.io/pepy/dt/robert?label=Downloads&logo=pypi)](https://www.pepy.tech/projects/robert)
 [![Read the Docs](https://img.shields.io/readthedocs/robert?label=Read%20the%20Docs&logo=readthedocs)](https://robert.readthedocs.io/)
-
-ROBERT is an ensemble of automated machine learning protocols that can be run sequentially 
-through a single command line or a graphical user interface. The program works for **regression and classification problems**.
-Comprehensive workflows have been designed to meet state-of-the-art standards for cheminformatics studies, including:
+[![PyPI](https://img.shields.io/pypi/v/robert)](https://pypi.org/project/robert/)
 
 ## Documentation  
 Full documentation with installation instructions, technical details and examples can be found in [Read the Docs](https://robert.readthedocs.io).  
 
-Don't miss out the latest hands-on tutorials from our [YouTube channel](https://www.youtube.com/channel/UCHRqI8N61bYxWV9BjbUI4Xw)!  
+Don't miss out the latest hands-on tutorials from our [YouTube channel](https://www.youtube.com/channel/UCHRqI8N61bYxWV9BjbUI4Xw)!
+
+## Recommended installation
+1. (Only once) Create new conda environment: `conda create -n robert python=3.10`  
+2. Activate conda environment: `conda activate robert`  
+3. Install ROBERT using pip: `pip install robert` 
+4. Install libraries for the PDF report `conda install -y -c conda-forge glib gtk3 pango mscorefonts`
+5. (Only for compatible devices) Install Intelex accelerator: `pip install scikit-learn-intelex`  
+* Inexperienced users should visit the *Users with no Python experience* section in [Read the Docs](https://robert.readthedocs.io).
+## Update the program
+1. Update to the latest version: `pip install robert --upgrade`  
 
 ## Developers and help desk  
 List of main developers and contact emails:  
@@ -32,6 +39,8 @@ J.V.A.R. - The acronym ROBERT is dedicated to **ROBERT Paton**, who was a mentor
 
 D.D.G. - The style of the ROBERT_report.pdf file was created with the help of **Oliver Lee** (2023, Zysman-Colman group at University of St Andrews).
 
+J.V.A.R. and D.D.G. - The improvements from v1.0 to v1.2 are largely the result of insightful discussions with **Matthew Sigman** and his students, **Jamie Cadge** and **Simone Gallarati** (2024, University of Utah).
+
 We really THANK all the testers for their feedback and for participating in the reproducibility tests, including:
 
 * **David Valiente** (2022-2023, Universidad Miguel Hernández)
@@ -44,11 +53,11 @@ We really THANK all the testers for their feedback and for participating in the
 
 ## How to cite ROBERT
 If you use any of the ROBERT modules, please include this citation:  
-* Dalmau, D.; Alegre Requena, J. V. ChemRxiv, 2023, DOI: 10.26434/chemrxiv-2023-k994h.  
+* Dalmau, D.; Alegre Requena, J. V. ROBERT: Bridging the Gap between Machine Learning and Chemistry. *Wiley Interdiscip. Rev. Comput. Mol. Sci.* **2024**, *accepted*. DOI: 10.1002/WCMS.1733.  
 
 If you use the AQME module, please include this citation:  
-* Alegre-Requena et al., AQME: Automated Quantum Mechanical Environments for Researchers and Educators. Wiley Interdiscip. Rev. Comput. Mol. Sci. 2023, 13, e1663.
+* Alegre-Requena et al., AQME: Automated Quantum Mechanical Environments for Researchers and Educators. *Wiley Interdiscip. Rev. Comput. Mol. Sci.* **2023**, *13*, e1663.
 
 Additionally, please include the corresponding reference for Scikit-learn and SHAP:  
-* Pedregosa et al., Scikit-learn: Machine Learning in Python, J. Mach. Learn. Res. 2011, 12, 2825-2830.  
-* Lundberg et al., From local explanations to global understanding with explainable AI for trees, Nat. Mach. Intell. 2020, 2, 56–67.  
+* Pedregosa et al., Scikit-learn: Machine Learning in Python. *J. Mach. Learn. Res.* **2011**, *12*, 2825-2830.  
+* Lundberg et al., From local explanations to global understanding with explainable AI for trees. *Nat. Mach. Intell.* **2020**, *2*, 56–67.  

{robert-1.2.0 → robert-1.2.1}/robert/__main__.py

@@ -13,7 +13,7 @@ from robert import robert
 # This allows the usage python pip-*.whl/pip install pip-*.whl
 
 if __package__ != 'robert':
-    print('ROBERT is not installed! Use: pip install robert or conda install -c conda-forge robert.')
+    print('ROBERT is not installed! Use: pip install robert or conda install -y -c conda-forge robert.')
 
 if __name__ == '__main__':
     robert.main()

{robert-1.2.0 → robert-1.2.1}/robert/aqme.py

@@ -15,6 +15,8 @@ Parameters
     csearch_keywords : str, default='--sample 50'
         Add extra keywords to the AQME-CSEARCH run (i.e. csearch_keywords='--sample 10'). CREST can be
         used instead of RDKit by adding "--program crest".
+    descp_lvl : str, default='interpret'
+        Type of descriptor to be used in the AQME-ROBERT workflow. Options are 'interpret', 'denovo' or 'full'.
 
 """
 #####################################################.
@@ -95,7 +97,7 @@ class aqme:
                     os.remove(new_sdf)
                 shutil.move(sdf_file, new_sdf)
                       
-        #Load database
+        # Load database
         csv_df = load_database(self,csv_target,job_type)
 
         # find if there is more than one SMILES column in the CSV file
@@ -137,29 +139,29 @@ class aqme:
                 if smi_suffix is not None:
                     # Change column names by adding suffix
                     try:
-                        df_temp = pd.read_csv(f'AQME-ROBERT_{aqme_indv_name}.csv', encoding='utf-8')
+                        df_temp = pd.read_csv(f'AQME-ROBERT_{self.args.descp_lvl}_{aqme_indv_name}.csv', encoding='utf-8')
                     except FileNotFoundError:
                         self.args.log.write("x  WARNING! ROBERT stopped due to a problem with the AQME job. Please, check the previous AQME warnings.")
                         sys.exit()
                     df_temp.columns = [f'{col}_{smi_suffix}' if col not in ['code_name','SMILES'] and col not in aqme_args else col for col in df_temp.columns]
-                    df_temp.to_csv(f'AQME-ROBERT_{aqme_indv_name}.csv', index=False)
+                    df_temp.to_csv(f'AQME-ROBERT_{self.args.descp_lvl}_{aqme_indv_name}.csv', index=False)
 
                     # Check if there are missing rows in the AQME-ROBERT_{aqme_indv_name}.csv
                     if len(df_temp) < len(csv_temp):
                         missing_rows = csv_temp.loc[~csv_temp['code_name'].isin(df_temp['code_name'])]
-                        missing_rows[['code_name', 'SMILES']].to_csv(f'AQME-ROBERT_{aqme_indv_name}.csv', mode='a', header=False, index=False)
+                        missing_rows[['code_name', 'SMILES']].to_csv(f'AQME-ROBERT_{self.args.descp_lvl}_{aqme_indv_name}.csv', mode='a', header=False, index=False)
 
                     # Get the order of code_name in aqme_indv_name
                     order = csv_temp['code_name'].tolist()
 
                     # Sort the rows in 'AQME-ROBERT_{aqme_indv_name}.csv' based on the order
-                    df_temp = pd.read_csv(f'AQME-ROBERT_{aqme_indv_name}.csv', encoding='utf-8')
+                    df_temp = pd.read_csv(f'AQME-ROBERT_{self.args.descp_lvl}_{aqme_indv_name}.csv', encoding='utf-8')
                     df_temp = df_temp.sort_values(by='code_name', key=lambda x: x.map({v: i for i, v in enumerate(order)}))
 
                     # Fill missing values with corresponding SMILES row
                     df_temp = df_temp.fillna(df_temp.groupby('SMILES').transform('first'))
 
-                    df_temp.to_csv(f'AQME-ROBERT_{aqme_indv_name}.csv', index=False)
+                    df_temp.to_csv(f'AQME-ROBERT_{self.args.descp_lvl}_{aqme_indv_name}.csv', index=False)
 
                 # return SDF files after csv_test
                 if aqme_test:
@@ -169,23 +171,23 @@ class aqme:
                         shutil.rmtree(path_sdf)
 
         # if AQME-ROBERT_AQME_indiv_n.csv >0 in folder:
-        if len(glob.glob('AQME-ROBERT_AQME_indiv*.csv')) > 0:
+        if len(glob.glob(f'AQME-ROBERT_{self.args.descp_lvl}_AQME_indiv*.csv')) > 0:
 
             df_concat = pd.DataFrame()
 
             # Read and concatenate CSV files 
-            for file in sorted(glob.glob('AQME-ROBERT_AQME_indiv*.csv'), key=os.path.getmtime,reverse=True):
+            for file in sorted(glob.glob(f'AQME-ROBERT_{self.args.descp_lvl}_AQME_indiv*.csv'), key=os.path.getmtime,reverse=True):
                 columns_to_drop = ['code_name', 'SMILES'] + aqme_args
                 df_temp = pd.read_csv(file, encoding='utf-8')
                 columns_to_drop = [col for col in columns_to_drop if col in df_temp.columns]
                 df_temp = df_temp.drop(columns=columns_to_drop)
                 df_concat = pd.concat([df_temp, df_concat], axis=1)
             df_concat = pd.concat([csv_df, df_concat], axis=1)
-            df_concat.to_csv(f'AQME-ROBERT_{csv_target}', index=False)
+            df_concat.to_csv(f'AQME-ROBERT_{self.args.descp_lvl}_{csv_target}', index=False)
 
 
         # if no qdesc_atom is set, only keep molecular properties and discard atomic properties
-        aqme_db = f'AQME-ROBERT_{csv_target}'
+        aqme_db = f'AQME-ROBERT_{self.args.descp_lvl}_{csv_target}'
 
         # ensure that the AQME database was successfully created
         if not os.path.exists(aqme_db):
@@ -226,7 +228,7 @@ class aqme:
         try:
             from aqme.qprep import qprep
         except ModuleNotFoundError:
-            self.args.log.write("x  AQME is not installed (required for the --aqme option)! You can install the program with 'conda install -c conda-forge aqme'")
+            self.args.log.write("x  AQME is not installed (required for the --aqme option)! The program is typically installed within 2-5 minutes (https://aqme.readthedocs.io, see the Installation section)")
             sys.exit()
 
 

{robert-1.2.0 → robert-1.2.1}/robert/argument_parser.py

@@ -30,7 +30,7 @@ var_dict = {
     "corr_filter" : True,
     "desc_thres" : 25,
     "thres_y" : 0.001,
-    "thres_x" : 0.9,
+    "thres_x" : 0.7,
     "test_set" : 0.1,
     "auto_test" : True,
     "train" : [60,70,80,90],
@@ -58,6 +58,7 @@ var_dict = {
     "pfi_show" : 10,
     "names" : '',
     "qdescp_keywords" : '',
+    "descp_lvl": "interpret",
     "csearch_keywords": '--sample 10',
     "report_modules" : ['AQME','CURATE','GENERATE','VERIFY','PREDICT'],
     "debug_report": False

{robert-1.2.0 → robert-1.2.1}/robert/curate.py

@@ -32,9 +32,9 @@ Parameters
     desc_thres : float, default=25
         Threshold for the descriptor-to-datapoints ratio to loose the correlation filter. By default,
         the correlation filter is loosen if there are 25 times more datapoints than descriptors.
-    thres_x : float, default=0.9
+    thres_x : float, default=0.7
         Thresolhold to discard descriptors based on high R**2 correlation with other descriptors (i.e. 
-        if thres_x=0.9, variables that show R**2 > 0.9 will be discarded).
+        if thres_x=0.7, variables that show R**2 > 0.7 will be discarded).
     thres_y : float, default=0.001
         Thresolhold to discard descriptors with poor correlation with the y values based on R**2 (i.e.
         if thres_y=0.001, variables that show R**2 < 0.001 will be discarded).

{robert-1.2.0 → robert-1.2.1}/robert/predict_utils.py

@@ -503,15 +503,15 @@ def print_predict(self,Xy_data,params_dict,path_n_suffix,loaded_model):
         if 'y_pred_csv_test' in Xy_data and not Xy_data['y_csv_test'].isnull().values.any() and len(Xy_data['y_csv_test']) > 0:
             print_results += f"\n      -  csv_test : R2 = {Xy_data['r2_csv_test']:.2}, MAE = {Xy_data['mae_csv_test']:.2}, RMSE = {Xy_data['rmse_csv_test']:.2}"
         # add equation for linear models
-        if params_dict['model'].upper() == 'MVL':
+        if params_dict['model'].upper() == 'MVL' or self.args.evaluate == 'True':
             desc_mvl = ast.literal_eval(params_dict['X_descriptors'])
             print_results += f"\n\n   o  Linear model equation, with coefficients obtained using standardized data (coefficient values/importances can be compared):"
             print_results += f"\n      - {params_dict['y']} = {loaded_model.intercept_:.2} "
-            for i,coeff in enumerate(loaded_model.coef_):
+            for i, coeff in enumerate(loaded_model.coef_):
                 if float(coeff) >= 0:
-                    print_results += f"+ {coeff:.2}·{desc_mvl[i]} "
+                    print_results += f"+ ({coeff:.2} * {desc_mvl[i]}) "
                 else:
-                    print_results += f"- {np.abs(coeff):.2}·{desc_mvl[i]} "
+                    print_results += f"- ({np.abs(coeff):.2} * {desc_mvl[i]}) "
 
     elif params_dict['type'].lower() == 'clas':
         print_results += f"\n      -  Train : Accuracy = {Xy_data['acc_train']:.2}, F1 score = {Xy_data['f1_train']:.2}, MCC = {Xy_data['mcc_train']:.2}"
@@ -1040,9 +1040,9 @@ def pearson_map_predict(self,Xy_data,params_dir):
         max_r = corr_dict['r'][abs_r_list.index(abs_max_r)]
         max_descp_1 = corr_dict['descp_1'][abs_r_list.index(abs_max_r)]
         max_descp_2 = corr_dict['descp_2'][abs_r_list.index(abs_max_r)]
-        if abs_max_r > 0.95:
+        if abs_max_r > 0.84:
             print_corr += f"\n      x  WARNING! High correlations observed (up to r = {round(max_r,2)} or R2 = {round(max_r*max_r,2)}, for {max_descp_1} and {max_descp_2})"
-        elif abs_max_r > 0.8:
+        elif abs_max_r > 0.71:
             print_corr += f"\n      x  WARNING! Noticeable correlations observed (up to r = {round(max_r,2)} or R2 = {round(max_r*max_r,2)}, for {max_descp_1} and {max_descp_2})"
 
     self.args.log.write(print_corr)

{robert-1.2.0 → robert-1.2.1}/robert/report.py

@@ -68,7 +68,7 @@ class report:
         try:
             from weasyprint import HTML
         except (OSError, ModuleNotFoundError):
-            print(f"\n  x The REPORT module requires weasyprint but this module is missing, the PDF with the summary of the results has not been created. Try installing ROBERT with 'conda install -c conda-forge robert'")
+            print(f"\n  x The REPORT module requires weasyprint but this module is missing, the PDF with the summary of the results has not been created. Try installing ROBERT with 'conda install -y -c conda-forge robert'")
             sys.exit()
 
         # load default and user-specified variables
@@ -676,8 +676,27 @@ class report:
         
         # starts with the icon of feature importances
         feature_dat = ''
-        feature_dat = self.module_lines('features',feature_dat) 
+        feature_dat = self.module_lines('features',feature_dat)
         
+        # Add linear model equations
+        spacing = get_spacing_col('PFI',spacing_PFI)
+        with open(f'{os.getcwd()}/PREDICT/PREDICT_data.dat', 'r') as file:
+            lines = file.readlines()
+            linear_model_eqs = [lines[i + 1].strip().lstrip('- ') for i, line in enumerate(lines) if 'o  Linear model equation' in line]
+
+        if linear_model_eqs:
+            feature_dat += "<p style='margin-top:-5px; margin-bottom:-6px'><span style='font-weight:bold;'>Linear model equation_No_PFI</span></p>"
+            if len(linear_model_eqs) == 1:
+                feature_dat += f"<p style='margin-top:10px; margin-bottom:35px'>{linear_model_eqs[0]}</p>"
+            else:
+                columns_eq = []
+                for i, eq in enumerate(linear_model_eqs):
+                    if i == 0:
+                        columns_eq.append(f"<p style='margin-top:10px; margin-bottom:35px'>{eq}</p>")
+                    else:
+                        columns_eq.append(f"<p style='margin-top:-5px; margin-bottom:-6px'><span style='font-weight:bold;'>{spacing*3}Linear model equation_PFI</span></p><p style='margin-top:10px; margin-bottom:35px'>{spacing*3}{eq}</p>")
+                feature_dat += combine_cols(columns_eq)
+
         # add corresponding images
         module_path = Path(f'{os.getcwd()}/PREDICT')
                 
@@ -855,7 +874,7 @@ class report:
             repro_dat += f"""<p style="text-align: justify; margin-top: -44px;"><br><strong>2. Install and adjust the versions of the following Python modules:</strong></p>"""
         else:
             repro_dat += f"""{first_line}<br><strong>2. Install and adjust the versions of the following Python modules:</strong></p>"""
-        repro_dat += f"""{reduced_line}{space}- Install ROBERT and its dependencies: conda install -c conda-forge robert</p>"""
+        repro_dat += f"""{reduced_line}{space}- Install ROBERT and its dependencies: conda install -y -c conda-forge robert</p>"""
         repro_dat += f"""{reduced_line}{space}- Adjust ROBERT version: pip install robert=={robert_version}</p>"""
 
         if intelex_version != 'not installed':
@@ -868,7 +887,7 @@ class report:
         if aqme_workflow:
             if not find_aqme:
                 repro_dat += f"""{reduced_line}{space}- AQME is required, but no version was found:</p>"""
-            repro_dat += f"""{reduced_line}{space}- Install AQME and its dependencies: conda install -c conda-forge aqme</p>"""
+            repro_dat += f"""{reduced_line}{space}- Install AQME and its dependencies: conda install -y -c conda-forge aqme</p>"""
             if find_aqme:
                 repro_dat += f"""{reduced_line}{space}- Adjust AQME version: pip install aqme=={aqme_version}</p>"""
 
@@ -884,9 +903,9 @@ class report:
                 find_xtb = False
             if not find_xtb:
                 repro_dat += f"""{reduced_line}{space}- xTB is required, but no version was found:</p>"""
-            repro_dat += f"""{reduced_line}{space}- Install xTB: conda install -c conda-forge xtb</p>"""
+            repro_dat += f"""{reduced_line}{space}- Install xTB: conda install -y -c conda-forge xtb</p>"""
             if find_xtb:
-                repro_dat += f"""{reduced_line}{space}- Adjust xTB version (if possible): conda install -c conda-forge xtb={xtb_version}</p>"""
+                repro_dat += f"""{reduced_line}{space}- Adjust xTB version (if possible): conda install -y -c conda-forge xtb={xtb_version}</p>"""
 
         if crest_workflow:
             try:

{robert-1.2.0 → robert-1.2.1}/robert/report_utils.py

@@ -434,6 +434,7 @@ def get_col_score(score_info,data_score,suffix,spacing,eval_only):
     column = f"""<p style="margin-top:-18px;"><span style="font-weight:bold;">{title_line}</span></p>
     {ML_line_format}Model = {data_score['ML_model']}&nbsp;&nbsp;·&nbsp;&nbsp;{partitions_ratio}</p>
     {part_line_format}Points(train+valid.):descriptors = {data_score[f'points_descp_ratio_{suffix}']}</p>
+    {part_line_format}Score = {data_score[f'robert_score_{suffix}']} / 10</p>
     <p style="margin-top: 4px;">{score_info}
     <p style="margin-bottom: 18px;"></p>
     """

{robert-1.2.0 → robert-1.2.1}/robert/robert.py

@@ -73,9 +73,9 @@ def main(exe_type='command',sys_args=None):
         if args.aqme:
             aqme(**vars(args))
             # set the path to the database created by AQME to continue in the full_workflow
-            args.csv_name = Path(os.path.dirname(args.csv_name)).joinpath(f'AQME-ROBERT_{os.path.basename(args.csv_name)}')
+            args.csv_name = Path(os.path.dirname(args.csv_name)).joinpath(f'AQME-ROBERT_{args.descp_lvl}_{os.path.basename(args.csv_name)}')
             if args.csv_test != '':
-                args.csv_test = Path(os.path.dirname(args.csv_test)).joinpath(f'AQME-ROBERT_{os.path.basename(args.csv_test)}')
+                args.csv_test = Path(os.path.dirname(args.csv_test)).joinpath(f'AQME-ROBERT_{args.descp_lvl}_{os.path.basename(args.csv_test)}')
 
         # CURATE
         if args.curate or full_workflow:

{robert-1.2.0 → robert-1.2.1}/robert/utils.py

@@ -39,14 +39,15 @@ from sklearn.ensemble import (
 from sklearn.gaussian_process import GaussianProcessRegressor, GaussianProcessClassifier
 from sklearn.neural_network import MLPRegressor, MLPClassifier
 from sklearn.linear_model import LinearRegression
+from sklearn.impute import KNNImputer
 from mapie.regression import MapieRegressor
 from mapie.conformity_scores import AbsoluteConformityScore
 import warnings # this avoids warnings from sklearn
 warnings.filterwarnings("ignore")
 
-robert_version = "1.2.0"
+robert_version = "1.2.1"
 time_run = time.strftime("%Y/%m/%d %H:%M:%S", time.localtime())
-robert_ref = "Dalmau, D.; Alegre Requena, J. V. ChemRxiv, 2023, DOI: 10.26434/chemrxiv-2023-k994h"
+robert_ref = "Dalmau, D.; Alegre Requena, J. V. WIREs Comput Mol Sci. 2024, DOI: 10.1002/WCMS.1733"
 
 
 # load paramters from yaml file
@@ -128,7 +129,7 @@ def command_line_args(exe_type,sys_args):
         "aqme",
         "report",
         "cheers",
-        "evaluate"
+        "evaluate",
     ]
     list_args = [
         "discard",
@@ -136,7 +137,7 @@ def command_line_args(exe_type,sys_args):
         "train",
         "model",
         "report_modules",
-        "seed"
+        "seed",
     ]
     int_args = [
         'pfi_epochs',
@@ -229,6 +230,7 @@ o Other common options:
 * Affecting SMILES workflows, AQME:
   --qdescp_keywords STR (default="") : extra keywords in QDESCP (i.e. "--qdescp_atoms [Ir] --alpb h2o") 
   --csearch_keywords STR (default="--sample 50") : extra keywords in CSEARCH
+  --descp_lvl (default="interpret") "interpret", "denovo" or "full" : type of descriptor calculation
 
 
 o How to cite ROBERT:
@@ -304,6 +306,10 @@ def load_variables(kwargs, robert_module):
     if not os.path.exists(f"{self.csv_name}") and os.path.exists(f'{self.csv_name}.csv'):
         self.csv_name = f'{self.csv_name}.csv'
 
+    # check if user used .csv in csv_test
+    if self.csv_test and not os.path.exists(f"{self.csv_test}") and os.path.exists(f'{self.csv_test}.csv'):
+        self.csv_test = f'{self.csv_test}.csv'
+
     if robert_module != "command":
         self.initial_dir = Path(os.getcwd())
 
@@ -789,8 +795,14 @@ def load_database(self,csv_load,module):
         txt_load += f'\nx  WARNING! The original database was not a valid CSV (i.e., formatting issues from Microsoft Excel?). A new database using commas as separators was created and used instead, and the original database was stored as {new_csv_name}. To prevent this issue from happening again, you should use commas as separators: https://support.edapp.com/change-csv-separator.\n\n'
 
     csv_df = pd.read_csv(csv_load, encoding='utf-8')
-    # Fill missing values with zeros
-    csv_df = csv_df.fillna(0)
+    # Fill missing values using KNN imputer
+    csv_df = csv_df.dropna(axis=1, thresh=int(0.7 * len(csv_df))) # Remove columns with less than 70% of the data
+    int_columns = csv_df.select_dtypes(include=['int']).columns
+    numeric_columns = csv_df.select_dtypes(include=['float']).columns
+    imputer = KNNImputer(n_neighbors=5)
+    if csv_df[numeric_columns].isna().any().any(): 
+        csv_df[numeric_columns] = pd.DataFrame(imputer.fit_transform(csv_df[numeric_columns]), columns=numeric_columns)
+    csv_df[int_columns] = csv_df[int_columns]
 
     if module.lower() not in ['verify','no_print','evaluate']:
         sanity_checks(self.args,'csv_db',module,csv_df.columns)
@@ -901,6 +913,7 @@ def load_model_reg(params):
         params = correct_hidden_layers(params)
 
         loaded_model = MLPRegressor(batch_size=params['batch_size'],
+                                solver='lbfgs',
                                 hidden_layer_sizes=params['hidden_layer_sizes'],
                                 learning_rate_init=params['learning_rate_init'],
                                 max_iter=params['max_iter'],
@@ -980,6 +993,7 @@ def load_model_clas(params):
         params = correct_hidden_layers(params)
 
         loaded_model = MLPClassifier(batch_size=params['batch_size'],
+                                solver='lbfgs',
                                 hidden_layer_sizes=params['hidden_layer_sizes'],
                                 learning_rate_init=params['learning_rate_init'],
                                 max_iter=params['max_iter'],

{robert-1.2.0 → robert-1.2.1}/robert.egg-info/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.1
 Name: robert
-Version: 1.2.0
+Version: 1.2.1
 Summary: Refiner and Optimizer of a Bunch of Existing Regression Tools
 Home-page: https://github.com/jvalegre/robert
-Download-URL: https://github.com/jvalegre/robert/archive/refs/tags/1.2.0.tar.gz
+Download-URL: https://github.com/jvalegre/robert/archive/refs/tags/1.2.1.tar.gz
 Author: Juan V. Alegre Requena, David Dalmau
 Author-email: jv.alegre@csic.es
 License: MIT

{robert-1.2.0 → robert-1.2.1}/setup.py

@@ -1,5 +1,5 @@
 from setuptools import setup, find_packages
-version="1.2.0"
+version="1.2.1"
 setup(
     name="robert",
     packages=find_packages(exclude=["tests"]),

{robert-1.2.0 → robert-1.2.1}/tests/test_1curate.py

@@ -86,7 +86,7 @@ def test_CURATE(test_job):
         # 1. Duplicate entries are removed
         assert len(db_final['Name']) == 37
         # 2. Correlated variables with X and noise (low R2 with y) are removed
-        discard_vars = ['x1','x3','ynoise']
+        discard_vars = ['x1','x3','x5', 'x6','ynoise']
         for var in discard_vars:
             assert var not in db_final.columns
         # 3. Ignored variables and y are kept
@@ -95,12 +95,12 @@ def test_CURATE(test_job):
         # 4. Discarded variables are removed
         assert 'xtest' not in db_final.columns
         # 5. The rest of the variables are kept
-        final_vars = ['x2','x5','x6','x7','x8','x9','x10','x11']
+        final_vars = ['x2','x7','x8','x9','x10','x11']
         for var in final_vars:
             assert var in db_final.columns
 
         # check that categorical variables are converted with one-hot encoding
-        categ_vars = ['Csub-H','Csub-Csub','H-O','Csub-O']
+        categ_vars = ['Csub-H','Csub-Csub','H-O']
         for var in categ_vars:
             assert var in db_final.columns
         assert 'x4' not in db_final.columns
@@ -144,11 +144,11 @@ def test_CURATE(test_job):
         ]
         subprocess.run(cmd_robert)
 
-        # check that all the missing values were filled with 0s
+        # check that all the missing values were filled
         db_final = pd.read_csv(f"{path_curate}/Robert_example_NaNs_CURATE.csv")
         # 1. Duplicate entries are removed
-        assert db_final['x3'][0] == 0
-        assert db_final['x3'][8] == 0
+        assert isinstance(db_final['x2'][4], (int, float))
+        assert isinstance(db_final['x2'][0], (int, float))
 
     elif test_job == 'corr_filter':
         cmd_robert = [

{robert-1.2.0 → robert-1.2.1}/tests/test_2generate.py

@@ -143,7 +143,7 @@ def test_GENERATE(test_job):
         assert "ROBERT v" in outlines[0]
         assert "- 37 datapoints" in outlines[9]
         if test_job != 'reduced_clas':
-            assert "- 12 accepted descriptors" in outlines[10]
+            assert "- 9 accepted descriptors" in outlines[10]
         else:
             assert "- 9 accepted descriptors" in outlines[10]
         assert "- 1 ignored descriptors" in outlines[11]
@@ -185,26 +185,26 @@ def test_GENERATE(test_job):
             if test_job in ['reduced','reduced_PFImax','reduced_rnd']:
                 if folder == 'No_PFI':
                     if test_job != 'reduced_clas':
-                        desc_list = ['x2', 'x5', 'x6', 'x7', 'x8', 'x9', 'x10', 'x11', 'Csub-Csub', 'Csub-H', 'Csub-O', 'H-O']
+                        desc_list = ['x2', 'x7', 'x8', 'x9', 'x10', 'x11', 'Csub-Csub', 'Csub-H', 'H-O']
                     else:
                         desc_list = ['x1', 'x2', 'x3', 'x5', 'x6', 'x7', 'x8', 'x9', 'x10']
                 elif folder == 'PFI':
                     if test_job == 'reduced':
-                        desc_list = ['x6', 'x7', 'x10']
+                        desc_list = ['x7', 'x9', 'x10']
                     elif test_job == 'reduced_rnd':
-                        desc_list = ['x6', 'x7', 'x9', 'x10']
+                        desc_list = ['x7', 'x9', 'x10']
                     elif test_job =='reduced_PFImax':
-                        desc_list = ['x6', 'x7']
+                        desc_list = ['x7', 'x9']
 
                 if test_job in ['reduced','reduced_rnd']:    
                     sum_split = 0
                     if db_best['Set'][0] == 'Training':
                         sum_split += 1
-                    if db_best['Set'][1] == 'Training':
+                    if db_best['Set'][1] == 'Validation':
                         sum_split += 1
                     if db_best['Set'][2] == 'Training':
                         sum_split += 1
-                    if db_best['Set'][3] == 'Validation':
+                    if db_best['Set'][3] == 'Training':
                         sum_split += 1
                     if test_job == 'reduced':
                         assert sum_split == 4

{robert-1.2.0 → robert-1.2.1}/tests/test_4predict.py

@@ -106,7 +106,7 @@ def test_PREDICT(test_job):
             if 'x  WARNING! High correlations observed (up to r = 1.0 or R2 = 1.0, for x1 and x3)' in line:
                 pearson_found = True
         else:
-            if 'x  WARNING! Noticeable correlations observed (up to r = -0.92 or R2 = 0.85, for x6 and x10)' in line:
+            if 'o  Correlations between variables are acceptable' in line:
                 pearson_found = True
         if test_job == "clas":
             if 'o  Your data seems quite uniform' in line:

{robert-1.2.0 → robert-1.2.1}/tests/test_5aqme_n_full.py

@@ -80,7 +80,7 @@ def test_AQME(test_job):
         "--epochs", "5",
         "--seed", "[0]",
         "--model", "['RF']",
-        "--train", "[60]",
+        "--train", "[70]",
         "--pfi_epochs", "1",
         "--debug_report", "True"
     ]
@@ -150,9 +150,9 @@ def test_AQME(test_job):
         assert len(glob.glob(f'{path_main}/PREDICT/*.csv')) == 4
 
     if test_job == 'aqme':
-        assert os.path.exists(f'{path_main}/AQME-ROBERT_solubility.csv')
-        db_aqme = pd.read_csv(f'{path_main}/AQME-ROBERT_solubility.csv')
-        descps = ['code_name','solub','C_FUKUI+','MolLogP']
+        assert os.path.exists(f'{path_main}/AQME-ROBERT_interpret_solubility.csv')
+        db_aqme = pd.read_csv(f'{path_main}/AQME-ROBERT_interpret_solubility.csv')
+        descps = ['code_name','solub','HOMO','Total charge']
         for descp in descps:
             assert descp in db_aqme.columns
         assert 'smiles' in db_aqme.columns
@@ -164,12 +164,12 @@ def test_AQME(test_job):
         assert "ROBERT v" in outlines[0]
         assert os.path.exists(f'{path_aqme}/CSEARCH')
         assert os.path.exists(f'{path_aqme}/QDESCP')
-        assert len(glob.glob(f'{path_aqme}/*.csv')) == 1
+        assert len(glob.glob(f'{path_aqme}/*.csv')) == 0
         assert len(glob.glob(f'{path_aqme}/*.dat')) == 3
 
     if test_job == '2smiles_columns':
-        assert os.path.exists(f'{path_main}/AQME-ROBERT_solubility_solvent.csv')
-        db_aqme = pd.read_csv(f'{path_main}/AQME-ROBERT_solubility_solvent.csv')
+        assert os.path.exists(f'{path_main}/AQME-ROBERT_interpret_solubility_solvent.csv')
+        db_aqme = pd.read_csv(f'{path_main}/AQME-ROBERT_interpret_solubility_solvent.csv')
         assert 'smiles_sub' and 'smiles_solvent' in db_aqme.columns
 
     # find important parts in ROBERT_report
@@ -185,19 +185,19 @@ def test_AQME(test_job):
     for line in outlines:
         if 'Heatmap_ML_models_No_PFI.png' in line:
             find_heatmap += 1
-        if 'VERIFY_tests_RF_60_PFI.png' in line:
+        if 'VERIFY_tests_RF_70_PFI.png' in line:
             find_verify += 1
-        if 'SHAP_RF_60_PFI.png' in line:
+        if 'SHAP_RF_70_PFI.png' in line:
             find_shap += 1
-        if 'PFI_RF_60_PFI.png' in line:
+        if 'PFI_RF_70_PFI.png' in line:
             find_pfi += 1
-        if 'Outliers_RF_60_No_PFI.png' in line:
+        if 'Outliers_RF_70_No_PFI.png' in line:
             find_outliers += 1
-        if 'Results_RF_60_No_PFI.png' in line:
+        if 'Results_RF_70_No_PFI.png' in line:
             find_results_reg += 1
-        if 'Results_RF_60_No_PFI_valid.png' in line:
+        if 'Results_RF_70_No_PFI_valid.png' in line:
             find_results_valid_clas += 1
-        if 'Results_RF_60_No_PFI_csv_test.png' in line:
+        if 'Results_RF_70_No_PFI_csv_test.png' in line:
             find_results_test_clas += 1
         if 'csv_test :' in line:
             find_test += 1
@@ -210,7 +210,7 @@ def test_AQME(test_job):
         predict_graphs,flawed_image,cv_r2_image,cv_sd_image = False,False,False,False
         y_distrib_image,pearson_pred_image = False,False
         find_severe_red,find_severe_blue = False,False
-        find_moder_pred,find_moder_y_dist,find_moder_corr = False,False,False
+        find_moder_pred,find_moder_y_dist,unclear_test = False,False,False
         find_assess_yellow,find_assess_red = False,False
 
         for i,line in enumerate(outlines):
@@ -218,31 +218,31 @@ def test_AQME(test_job):
                 robert_score.append(line.split('robert/report/score_')[1][0])
             if 'Model = RF' in line:
                 ml_model_count += 1
-            if 'Train:Validation = 59:41' in line:
+            if 'Train:Validation = 68:32' in line:
                 partition_count += 1
             if 'Points(train+valid.):descriptors = ' in line:
                 points_desc.append(line.split('Points(train+valid.):descriptors = ')[1].split('</p>')[0])
-            if 'Results_RF_60_No_PFI.png' in line:
+            if 'Results_RF_70_No_PFI.png' in line:
                 predict_graphs = True
             if 'Train : R<sup>2</sup> = ' in line:
                 metrics_count += 1
             if '1. Model vs "flawed" models' in line:
                 flawed_models.append(line)
-            if 'VERIFY/VERIFY_tests_RF_60_No_PFI.png' in line:
+            if 'VERIFY/VERIFY_tests_RF_70_No_PFI.png' in line:
                 flawed_image = True
             if '2. Predictive ability of the model' in line:
                 pred_ability.append(line)
             if '3a. CV predictions train + valid.' in line:
                 cv_r2_models.append(line)
-            if 'VERIFY/CV_train_valid_predict_RF_60_No_PFI.png' in line:
+            if 'VERIFY/CV_train_valid_predict_RF_70_No_PFI.png' in line:
                 cv_r2_image = True
             if '3b. Avg. standard deviation (SD)' in line:
                 cv_sd_models.append(line)
-            if 'PREDICT/CV_variability_RF_60_No_PFI.png' in line:
+            if 'PREDICT/CV_variability_RF_70_No_PFI.png' in line:
                 cv_sd_image = True
             if '4. Points(train+valid.):descriptors' in line:
                 points_descp.append(line)
-            if 'y_distribution_RF_60_No_PFI.png' in line:
+            if 'y_distribution_RF_70_No_PFI.png' in line:
                 y_distrib_image = True
             if 'Pearson_heatmap_No_PFI.png' in line:
                 pearson_pred_image = True
@@ -254,8 +254,8 @@ def test_AQME(test_job):
                 find_moder_pred = True
             if 'Slightly uneven y distribution (Section C)' in line and 'color: #c5c57d' in outlines[i-1]:
                 find_moder_y_dist = True
-            if 'Moderately correlated features (Section D)' in line and 'color: #c5c57d' in outlines[i-1]:
-                find_moder_corr = True
+            if 'Some tests are unclear (Section B.1)' in line and 'color: #c5c57d' in outlines[i-1]:
+                unclear_test = True
             if 'Decent model, but it has limitations' in line and 'color: #c5c57d' in outlines[i-1]:
                 find_assess_yellow = True
             if 'The model is unreliable' in line and 'color: #c56666' in outlines[i-1]:
@@ -264,23 +264,23 @@ def test_AQME(test_job):
                 break
 
         # model summary, robert score, predict graphs and model metrics
-        assert robert_score[0] == '3'
-        assert robert_score[1] == '8'
+        assert robert_score[0] == '4'
+        assert robert_score[1] == '7'
         assert ml_model_count == 2
         assert partition_count == 2
-        assert points_desc[0] == '37:12'
+        assert points_desc[0] == '37:9'
         assert points_desc[1] == '37:3'
         assert predict_graphs
         assert metrics_count == 2
         # advanced analysis, flawed models section 1
         assert '1 / 3' in flawed_models[0]
         assert 'robert/report/score_w_3_1.jpg' in flawed_models[0]
-        assert '3 / 3' in flawed_models[1]
-        assert 'robert/report/score_w_3_3.jpg' in flawed_models[1]
+        assert '2 / 3' in flawed_models[1]
+        assert 'robert/report/score_w_3_2.jpg' in flawed_models[1]
         assert flawed_image
         # advanced analysis, predictive ability section 2
-        assert '1 / 2' in pred_ability[0]
-        assert 'robert/report/score_w_2_1.jpg' in pred_ability[0]
+        assert '2 / 2' in pred_ability[0]
+        assert 'robert/report/score_w_2_2.jpg' in pred_ability[0]
         assert '2 / 2' in pred_ability[1]
         assert 'robert/report/score_w_2_2.jpg' in pred_ability[1]
         # advanced analysis, CV predictive ability section 3a
@@ -308,7 +308,7 @@ def test_AQME(test_job):
         assert find_severe_blue
         assert find_moder_pred
         assert find_moder_y_dist
-        assert find_moder_corr
+        assert unclear_test
         assert find_assess_yellow
         assert find_assess_red
 

{robert-1.2.0 → robert-1.2.1}/tests/test_6evaluate.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 
 ######################################################.
-# 	          Testing AQME with pytest   	         #
+# 	          Testing EVALUATE with pytest   	     #
 ######################################################.
 
 import os
@@ -32,7 +32,7 @@ path_tests = os.getcwd() + "/tests"
         ),  # test that if the --csv_train or --csv_valid options are empty, a prompt pops up and asks for them 
     ],
 )
-def test_eval(test_job):
+def test_EVALUATE(test_job):
 
     # reset the folders (to avoid interferences with previous failed tests)
     folders = ['CURATE','GENERATE','GENERATE_reg','GENERATE_clas','PREDICT','VERIFY','AQME','EVALUATE']