nkululeko 0.88.12__tar.gz → 0.89.1__tar.gz

{nkululeko-0.88.12 → nkululeko-0.89.1}/CHANGELOG.md

@@ -1,8 +1,17 @@
 Changelog
 =========
 
-Version 0.88.12
+Version 0.89.1
+--------------
+* print and save result of feature importance
+
+Version 0.89.0
 --------------
+* added Roc plots and classification report on Debug
+
+
+Version 0.88.12
+---------------
 * added n_jobs for sklearn processing
 * re_named num_workers n_jobs
 
@@ -475,9 +484,9 @@ Version 0.66.3
 
 Version 0.66.2
 --------------
-* enabled data-pathes with quotes 
+* enabled data-pacthes with quotes 
 * enabled missing category labels
-* used tgdm for progress display
+* used tqdm for progress display
 
 Version 0.66.1
 --------------

{nkululeko-0.88.12/nkululeko.egg-info → nkululeko-0.89.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nkululeko
-Version: 0.88.12
+Version: 0.89.1
 Summary: Machine learning audio prediction experiments based on templates
 Home-page: https://github.com/felixbur/nkululeko
 Author: Felix Burkhardt
@@ -204,7 +204,7 @@ All of them take *--config <my_config.ini>* as an argument.
   * *configurations*: which experiments to combine
   * *--method* (optional): majority_voting, mean (default), max, sum, uncertainty, uncertainty_weighted, confidence_weighted, performance_weighted  
   * *--threshold*: uncertainty threshold (1.0 means no threshold)
-  * *--weightes*: weights for performance_weighted method (could be from previous UAR, ACC)
+  * *--weights*: weights for performance_weighted method (could be from previous UAR, ACC)
   * *--outfile* (optional): name of CSV file for output (default: ensemble_result.csv)
   * *--no_labels* (optional): indicate that no ground truth is given 
 * **nkululeko.multidb**: do [multiple experiments](http://blog.syntheticspeech.de/2024/01/02/nkululeko-compare-several-databases/), comparing several databases cross and in itself
@@ -220,14 +220,11 @@ All of them take *--config <my_config.ini>* as an argument.
 * **nkululeko.predict**: [predict features](http://blog.syntheticspeech.de/2023/08/16/nkululeko-how-to-predict-labels-for-your-data-from-existing-models-and-check-them/) like SNR, MOS, arousal/valence, age/gender, with DNN models
 * **nkululeko.segment**: [segment a database](http://blog.syntheticspeech.de/2023/07/14/nkululeko-segmenting-a-database/) based on VAD (voice activity detection)
 * **nkululeko.resample**: check on all [sampling rates and change](http://blog.syntheticspeech.de/2023/08/31/how-to-fix-different-sampling-rates-in-a-dataset-with-nkululeko/) to 16kHz 
-* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line.
-  * usage: nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET]
-                    [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
- 
-
-
-
+* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line. Usage:
 
+  ```bash
+  $ python -m nkululeko.nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET] [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
+  ```
 
 There's my [blog](http://blog.syntheticspeech.de/?s=nkululeko) with tutorials:
 * [Introduction](http://blog.syntheticspeech.de/2021/08/04/machine-learning-experiment-framework/)
@@ -359,8 +356,17 @@ F. Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben and Björn Schulle
 Changelog
 =========
 
-Version 0.88.12
+Version 0.89.1
 --------------
+* print and save result of feature importance
+
+Version 0.89.0
+--------------
+* added Roc plots and classification report on Debug
+
+
+Version 0.88.12
+---------------
 * added n_jobs for sklearn processing
 * re_named num_workers n_jobs
 
@@ -833,9 +839,9 @@ Version 0.66.3
 
 Version 0.66.2
 --------------
-* enabled data-pathes with quotes 
+* enabled data-pacthes with quotes 
 * enabled missing category labels
-* used tgdm for progress display
+* used tqdm for progress display
 
 Version 0.66.1
 --------------

{nkululeko-0.88.12 → nkululeko-0.89.1}/README.md

@@ -160,7 +160,7 @@ All of them take *--config <my_config.ini>* as an argument.
   * *configurations*: which experiments to combine
   * *--method* (optional): majority_voting, mean (default), max, sum, uncertainty, uncertainty_weighted, confidence_weighted, performance_weighted  
   * *--threshold*: uncertainty threshold (1.0 means no threshold)
-  * *--weightes*: weights for performance_weighted method (could be from previous UAR, ACC)
+  * *--weights*: weights for performance_weighted method (could be from previous UAR, ACC)
   * *--outfile* (optional): name of CSV file for output (default: ensemble_result.csv)
   * *--no_labels* (optional): indicate that no ground truth is given 
 * **nkululeko.multidb**: do [multiple experiments](http://blog.syntheticspeech.de/2024/01/02/nkululeko-compare-several-databases/), comparing several databases cross and in itself
@@ -176,14 +176,11 @@ All of them take *--config <my_config.ini>* as an argument.
 * **nkululeko.predict**: [predict features](http://blog.syntheticspeech.de/2023/08/16/nkululeko-how-to-predict-labels-for-your-data-from-existing-models-and-check-them/) like SNR, MOS, arousal/valence, age/gender, with DNN models
 * **nkululeko.segment**: [segment a database](http://blog.syntheticspeech.de/2023/07/14/nkululeko-segmenting-a-database/) based on VAD (voice activity detection)
 * **nkululeko.resample**: check on all [sampling rates and change](http://blog.syntheticspeech.de/2023/08/31/how-to-fix-different-sampling-rates-in-a-dataset-with-nkululeko/) to 16kHz 
-* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line.
-  * usage: nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET]
-                    [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
- 
-
-
-
+* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line. Usage:
 
+  ```bash
+  $ python -m nkululeko.nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET] [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
+  ```
 
 There's my [blog](http://blog.syntheticspeech.de/?s=nkululeko) with tutorials:
 * [Introduction](http://blog.syntheticspeech.de/2021/08/04/machine-learning-experiment-framework/)

{nkululeko-0.88.12 → nkululeko-0.89.1}/nkululeko/constants.py

@@ -1,2 +1,2 @@
-VERSION="0.88.12"
+VERSION="0.89.1"
 SAMPLING_RATE = 16000

{nkululeko-0.88.12 → nkululeko-0.89.1}/nkululeko/ensemble.py

@@ -26,7 +26,11 @@ from pathlib import Path
 
 import numpy as np
 import pandas as pd
-from sklearn.metrics import balanced_accuracy_score
+from sklearn.metrics import(
+    balanced_accuracy_score, 
+    classification_report,
+    f1_score
+)
 
 from nkululeko.constants import VERSION
 from nkululeko.experiment import Experiment
@@ -284,6 +288,10 @@ def ensemble_predictions(
     predicted = ensemble_preds["predicted"]
     uar = balanced_accuracy_score(truth, predicted)
     acc = (truth == predicted).mean()
+    # print classification report
+    Util("ensemble").debug(f"\n {classification_report(truth, predicted)}")
+    # f1 = f1_score(truth, predicted, pos_label='p')
+    # Util("ensemble").debug(f"F1: {f1:.3f}")
     Util("ensemble").debug(f"{method}: UAR: {uar:.3f}, ACC: {acc:.3f}")
 
     return ensemble_preds

{nkululeko-0.88.12 → nkululeko-0.89.1}/nkululeko/feat_extract/feats_analyser.py

@@ -139,7 +139,7 @@ class FeatureAnalyser:
                 elif model_s == "svm":
                     from sklearn.svm import SVC
 
-                    c = float(self.util.config_val("MODEL", "C_val", "0.001"))
+                    c = float(self.util.config_val("MODEL", "C_val", "1.0"))
                     model = SVC(kernel="linear", C=c, gamma="scale")
                     result_importances[model_s] = self._get_importance(
                         model, permutation
@@ -205,7 +205,7 @@ class FeatureAnalyser:
                         model, permutation
                     )
                 elif model_s == "xgr":
-                    from xgboost import XGBClassifier
+                    from xgboost import XGBRegressor
 
                     model = XGBRegressor()
                     result_importances[model_s] = self._get_importance(
@@ -270,12 +270,14 @@ class FeatureAnalyser:
             )
         )
 
+        # print feature importance values to file and debug and save to result
+        self.util.debug(f"Importance features from {model_name}: features = \n{df_imp['feats'].values.tolist()}")
         # result file
         res_dir = self.util.get_path("res_dir")
         filename = f"_EXPL_{model_name}"
         if permutation:
             filename += "_perm"
-        filename = f"{res_dir}{self.util.get_exp_name(only_data=True)}{filename}_{model_name}.txt"
+        filename = f"{res_dir}{self.util.get_exp_name(only_data=True)}{filename}_{max_feat_num}_fi.txt"
         with open(filename, "w") as text_file:
             text_file.write(
                 "features in order of decreasing importance according to model"
@@ -283,7 +285,8 @@ class FeatureAnalyser:
             )
 
         df_imp.to_csv(filename, mode="a")
-
+        self.util.debug(f"Saved feature importance values to {filename}")
+        
         # check if feature distributions should be plotted
         plot_feats = self.util.config_val("EXPL", "feature_distributions", False)
         if plot_feats:

{nkululeko-0.88.12 → nkululeko-0.89.1}/nkululeko/models/model_tree.py

@@ -12,4 +12,6 @@ class Tree_model(Model):
     def __init__(self, df_train, df_test, feats_train, feats_test):
         super().__init__(df_train, df_test, feats_train, feats_test)
         self.name = "tree"
-        self.clf = DecisionTreeClassifier()  # set up the classifier
+        self.clf = DecisionTreeClassifier(
+            random_state=42
+        )  # set up the classifier

{nkululeko-0.88.12 → nkululeko-0.89.1}/nkululeko/reporting/reporter.py

@@ -27,6 +27,7 @@ from sklearn.metrics import (
     r2_score,
     roc_auc_score,
     roc_curve,
+    RocCurveDisplay,
 )
 
 import nkululeko.glob_conf as glob_conf
@@ -75,6 +76,7 @@ class Reporter:
         self.result = Result(0, 0, 0, 0, "unknown")
         self.run = run
         self.epoch = epoch
+        self.model_type = self.util.get_model_type()
         self._set_metric()
         self.filenameadd = ""
         self.cont_to_cat = False
@@ -387,6 +389,7 @@ class Reporter:
             epoch = self.epoch
         """Print all evaluation values to text file."""
         res_dir = self.util.get_path("res_dir")
+        fig_dir = self.util.get_path("fig_dir")
         file_name = f"{res_dir}{self.util.get_exp_name()}_{epoch}{self.filenameadd}.txt"
         if self.util.exp_is_classification():
             labels = glob_conf.labels
@@ -397,6 +400,10 @@ class Reporter:
                     target_names=labels,
                     output_dict=True,
                 )
+                # print classifcation report in console
+                self.util.debug(
+                    f"\n {classification_report(self.truths, self.preds, target_names=labels)}"
+                )
             except ValueError as e:
                 self.util.debug(
                     "Reporter: caught a ValueError when trying to get"
@@ -415,6 +422,17 @@ class Reporter:
                 if len(np.unique(self.truths)) == 2:
                     fpr, tpr, _ = roc_curve(self.truths, self.preds)
                     auc_score = auc(fpr, tpr)
+                    display = RocCurveDisplay(
+                        fpr=fpr,
+                        tpr=tpr,
+                        roc_auc=auc_score,
+                        estimator_name=f"{self.model_type} estimator",
+                    )
+                    # save plot
+                    plot_path = f"{fig_dir}{self.util.get_exp_name()}_{epoch}{self.filenameadd}_roc.{self.format}"
+                    display.plot(ax=None)
+                    plt.savefig(plot_path)
+                    self.util.debug(f"Saved ROC curve to {plot_path}")
                     pauc_score = roc_auc_score(self.truths, self.preds, max_fpr=0.1)
                     auc_pauc = f"auc: {auc_score:.3f}, pauc: {pauc_score:.3f} from epoch: {epoch}"
                     self.util.debug(auc_pauc)

{nkululeko-0.88.12 → nkululeko-0.89.1}/nkululeko/utils/util.py

@@ -175,7 +175,7 @@ class Util:
         """Get the experiment directory."""
         root = os.path.join(self.config["EXP"]["root"], "")
         name = self.config["EXP"]["name"]
-        dir_name = f"{root}{name}"
+        dir_name = f"{root}/{name}"
         audeer.mkdir(dir_name)
         return dir_name
 

{nkululeko-0.88.12 → nkululeko-0.89.1/nkululeko.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nkululeko
-Version: 0.88.12
+Version: 0.89.1
 Summary: Machine learning audio prediction experiments based on templates
 Home-page: https://github.com/felixbur/nkululeko
 Author: Felix Burkhardt
@@ -204,7 +204,7 @@ All of them take *--config <my_config.ini>* as an argument.
   * *configurations*: which experiments to combine
   * *--method* (optional): majority_voting, mean (default), max, sum, uncertainty, uncertainty_weighted, confidence_weighted, performance_weighted  
   * *--threshold*: uncertainty threshold (1.0 means no threshold)
-  * *--weightes*: weights for performance_weighted method (could be from previous UAR, ACC)
+  * *--weights*: weights for performance_weighted method (could be from previous UAR, ACC)
   * *--outfile* (optional): name of CSV file for output (default: ensemble_result.csv)
   * *--no_labels* (optional): indicate that no ground truth is given 
 * **nkululeko.multidb**: do [multiple experiments](http://blog.syntheticspeech.de/2024/01/02/nkululeko-compare-several-databases/), comparing several databases cross and in itself
@@ -220,14 +220,11 @@ All of them take *--config <my_config.ini>* as an argument.
 * **nkululeko.predict**: [predict features](http://blog.syntheticspeech.de/2023/08/16/nkululeko-how-to-predict-labels-for-your-data-from-existing-models-and-check-them/) like SNR, MOS, arousal/valence, age/gender, with DNN models
 * **nkululeko.segment**: [segment a database](http://blog.syntheticspeech.de/2023/07/14/nkululeko-segmenting-a-database/) based on VAD (voice activity detection)
 * **nkululeko.resample**: check on all [sampling rates and change](http://blog.syntheticspeech.de/2023/08/31/how-to-fix-different-sampling-rates-in-a-dataset-with-nkululeko/) to 16kHz 
-* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line.
-  * usage: nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET]
-                    [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
- 
-
-
-
+* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line. Usage:
 
+  ```bash
+  $ python -m nkululeko.nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET] [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
+  ```
 
 There's my [blog](http://blog.syntheticspeech.de/?s=nkululeko) with tutorials:
 * [Introduction](http://blog.syntheticspeech.de/2021/08/04/machine-learning-experiment-framework/)
@@ -359,8 +356,17 @@ F. Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben and Björn Schulle
 Changelog
 =========
 
-Version 0.88.12
+Version 0.89.1
 --------------
+* print and save result of feature importance
+
+Version 0.89.0
+--------------
+* added Roc plots and classification report on Debug
+
+
+Version 0.88.12
+---------------
 * added n_jobs for sklearn processing
 * re_named num_workers n_jobs
 
@@ -833,9 +839,9 @@ Version 0.66.3
 
 Version 0.66.2
 --------------
-* enabled data-pathes with quotes 
+* enabled data-pacthes with quotes 
 * enabled missing category labels
-* used tgdm for progress display
+* used tqdm for progress display
 
 Version 0.66.1
 --------------