nkululeko 0.88.4__tar.gz → 0.88.6__tar.gz

{nkululeko-0.88.4 → nkululeko-0.88.6}/CHANGELOG.md

@@ -1,6 +1,16 @@
 Changelog
 =========
 
+Version 0.88.6
+--------------
+* added test speaker assign
+
+Version 0.88.5
+--------------
+* add a unique name to the uncertainty plot
+* fix error in speaker embedding (still need speechbrain < 1.0)
+* add get_target_name function in util
+
 Version 0.88.4
 --------------
 * added more ensemble methods, e.g. based on uncertainty

{nkululeko-0.88.4/nkululeko.egg-info → nkululeko-0.88.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nkululeko
-Version: 0.88.4
+Version: 0.88.6
 Summary: Machine learning audio prediction experiments based on templates
 Home-page: https://github.com/felixbur/nkululeko
 Author: Felix Burkhardt
@@ -68,7 +68,8 @@ A project to detect speaker characteristics by machine learning experiments with
 
 The idea is to have a framework (based on e.g. sklearn and torch) that can be used to rapidly and automatically analyse audio data and explore machine learning models based on that data.
 
-* NEW with nkululeko: [Finetune transformer-models](http://blog.syntheticspeech.de/2024/05/29/nkululeko-how-to-finetune-a-transformer-model/)
+* NEW with nkululek: [Ensemble learning](http://blog.syntheticspeech.de/2024/06/25/nkululeko-ensemble-classifiers-with-late-fusion/)
+* NEW: [Finetune transformer-models](http://blog.syntheticspeech.de/2024/05/29/nkululeko-how-to-finetune-a-transformer-model/)
 * The latest features can be seen in [the ini-file](./ini_file.md) options that are used to control Nkululeko
 * Below is a [Hello World example](#helloworld) that should set you up fastly, also on [Google Colab](https://colab.research.google.com/drive/1GYNBd5cdZQ1QC3Jm58qoeMaJg3UuPhjw?usp=sharing#scrollTo=4G_SjuF9xeQf), and [with Kaggle](https://www.kaggle.com/felixburk/nkululeko-hello-world-example)
 * [Here's a blog post on how to set up nkululeko on your computer.](http://blog.syntheticspeech.de/2021/08/30/how-to-set-up-your-first-nkululeko-project/)
@@ -203,7 +204,7 @@ All of them take *--config <my_config.ini>* as an argument.
 * **nkululeko.nkululeko**: do machine learning experiments combining features and learners
 * **nkululeko.ensemble**: [combine several nkululeko experiments](http://blog.syntheticspeech.de/2024/06/25/nkululeko-ensemble-classifiers-with-late-fusion/) and report on late fusion results
   * *configurations*: which experiments to combine
-  * *--method* (optional): mean, max, sum, max_class, uncertainty_threshold, uncertainty_weighted, confidence_weighted
+  * *--method* (optional): majority_voting, mean, max, sum, max_class, uncertainty_threshold, uncertainty_weighted, confidence_weighted
   * *--threshold*: uncertainty threshold (1.0 means no threshold)
   * *--outfile* (optional): name of CSV file for output
   * *--no_labels* (optional): indicate that no ground truth is given 
@@ -359,6 +360,16 @@ F. Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben and Björn Schulle
 Changelog
 =========
 
+Version 0.88.6
+--------------
+* added test speaker assign
+
+Version 0.88.5
+--------------
+* add a unique name to the uncertainty plot
+* fix error in speaker embedding (still need speechbrain < 1.0)
+* add get_target_name function in util
+
 Version 0.88.4
 --------------
 * added more ensemble methods, e.g. based on uncertainty

{nkululeko-0.88.4 → nkululeko-0.88.6}/README.md

@@ -24,7 +24,8 @@ A project to detect speaker characteristics by machine learning experiments with
 
 The idea is to have a framework (based on e.g. sklearn and torch) that can be used to rapidly and automatically analyse audio data and explore machine learning models based on that data.
 
-* NEW with nkululeko: [Finetune transformer-models](http://blog.syntheticspeech.de/2024/05/29/nkululeko-how-to-finetune-a-transformer-model/)
+* NEW with nkululek: [Ensemble learning](http://blog.syntheticspeech.de/2024/06/25/nkululeko-ensemble-classifiers-with-late-fusion/)
+* NEW: [Finetune transformer-models](http://blog.syntheticspeech.de/2024/05/29/nkululeko-how-to-finetune-a-transformer-model/)
 * The latest features can be seen in [the ini-file](./ini_file.md) options that are used to control Nkululeko
 * Below is a [Hello World example](#helloworld) that should set you up fastly, also on [Google Colab](https://colab.research.google.com/drive/1GYNBd5cdZQ1QC3Jm58qoeMaJg3UuPhjw?usp=sharing#scrollTo=4G_SjuF9xeQf), and [with Kaggle](https://www.kaggle.com/felixburk/nkululeko-hello-world-example)
 * [Here's a blog post on how to set up nkululeko on your computer.](http://blog.syntheticspeech.de/2021/08/30/how-to-set-up-your-first-nkululeko-project/)
@@ -159,7 +160,7 @@ All of them take *--config <my_config.ini>* as an argument.
 * **nkululeko.nkululeko**: do machine learning experiments combining features and learners
 * **nkululeko.ensemble**: [combine several nkululeko experiments](http://blog.syntheticspeech.de/2024/06/25/nkululeko-ensemble-classifiers-with-late-fusion/) and report on late fusion results
   * *configurations*: which experiments to combine
-  * *--method* (optional): mean, max, sum, max_class, uncertainty_threshold, uncertainty_weighted, confidence_weighted
+  * *--method* (optional): majority_voting, mean, max, sum, max_class, uncertainty_threshold, uncertainty_weighted, confidence_weighted
   * *--threshold*: uncertainty threshold (1.0 means no threshold)
   * *--outfile* (optional): name of CSV file for output
   * *--no_labels* (optional): indicate that no ground truth is given 

{nkululeko-0.88.4 → nkululeko-0.88.6}/nkululeko/constants.py

@@ -1,2 +1,2 @@
-VERSION="0.88.4"
+VERSION="0.88.6"
 SAMPLING_RATE = 16000

{nkululeko-0.88.4 → nkululeko-0.88.6}/nkululeko/data/dataset.py

@@ -423,6 +423,9 @@ class Dataset:
             self.util.debug(f"{self.name}: trying to reuse data splits")
             self.df_test = pd.read_pickle(storage_test)
             self.df_train = pd.read_pickle(storage_train)
+        elif isinstance(ast.literal_eval(split_strategy), list):
+            # treat this as a list of test speakers
+            self.assign_speakers(ast.literal_eval(split_strategy))
         else:
             self.util.error(f"unknown split strategy: {split_strategy}")
 
@@ -515,6 +518,19 @@ class Dataset:
         # because this generates new train/test sample quantaties, the feature extraction has to be done again
         glob_conf.config["FEATS"]["needs_feature_extraction"] = "True"
 
+    def assign_speakers(self, speakers):
+        """One way to split train and eval sets: Specify test speaker names."""
+        self.df_test = self.df[self.df.speaker.isin(speakers)]
+        if len(self.df_test) == 0:
+            self.util.error(f"no speakers found in {speakers}")
+        self.df_train = self.df[~self.df.index.isin(self.df_test.index)]
+        self.util.debug(
+            f"{self.name} (speakers assigned): [{self.df_train.shape[0]}/{self.df_test.shape[0]}]"
+            " samples in train/test"
+        )
+        # because this generates new train/test sample quantaties, the feature extraction has to be done again
+        glob_conf.config["FEATS"]["needs_feature_extraction"] = "True"
+
     def split_speakers(self):
         """One way to split train and eval sets: Specify percentage of evaluation speakers"""
         test_percent = int(self.util.config_val_data(self.name, "test_size", 20))

{nkululeko-0.88.4 → nkululeko-0.88.6}/nkululeko/feat_extract/feats_agender.py

@@ -1,13 +1,14 @@
 # feats_agender.py
+
 from nkululeko.feat_extract.featureset import Featureset
 import os
-import pandas as pd
+# import pandas as pd
 import audeer
 import nkululeko.glob_conf as glob_conf
 import audonnx
 import numpy as np
 import audinterface
-
+import torch
 
 class AgenderSet(Featureset):
     """
@@ -32,7 +33,8 @@ class AgenderSet(Featureset):
             archive_path = audeer.download_url(
                 model_url, cache_root, verbose=True)
             audeer.extract_archive(archive_path, model_root)
-        device = self.util.config_val("MODEL", "device", "cpu")
+        cuda = "cuda" if torch.cuda.is_available() else "cpu"
+        device = self.util.config_val("MODEL", "device", cuda)
         self.model = audonnx.load(model_root, device=device)
         self.util.debug(f"initialized agender model")
         self.model_loaded = True

{nkululeko-0.88.4 → nkululeko-0.88.6}/nkululeko/feat_extract/feats_spkrec.py

@@ -24,7 +24,7 @@ class Spkrec(Featureset):
     def __init__(self, name, data_df, feat_type):
         """Constructor. is_train is needed to distinguish from test/dev sets,
         because they use the codebook from the training"""
-        super().__init__(name, data_df)
+        super().__init__(name, data_df, feat_type)
         # check if device is not set, use cuda if available
         cuda = "cuda" if torch.cuda.is_available() else "cpu"
         self.device = self.util.config_val("MODEL", "device", cuda)

{nkululeko-0.88.4 → nkululeko-0.88.6}/nkululeko/reporting/reporter.py

@@ -2,28 +2,32 @@ import ast
 import glob
 import json
 import math
-import os
 
-from confidence_intervals import evaluate_with_conf_int
 import matplotlib.pyplot as plt
 import numpy as np
+
+# from torch import is_tensor
+from audmetric import (
+    accuracy,
+    concordance_cc,
+    mean_absolute_error,
+    mean_squared_error,
+    unweighted_average_recall,
+)
+
+# import os
+from confidence_intervals import evaluate_with_conf_int
 from scipy.special import softmax
-from scipy.stats import entropy
-from scipy.stats import pearsonr
-from sklearn.metrics import ConfusionMatrixDisplay
-from sklearn.metrics import auc
-from sklearn.metrics import classification_report
-from sklearn.metrics import confusion_matrix
-from sklearn.metrics import r2_score
-from sklearn.metrics import roc_auc_score
-from sklearn.metrics import roc_curve
-from torch import is_tensor
-
-from audmetric import accuracy
-from audmetric import concordance_cc
-from audmetric import mean_absolute_error
-from audmetric import mean_squared_error
-from audmetric import unweighted_average_recall
+from scipy.stats import entropy, pearsonr
+from sklearn.metrics import (
+    ConfusionMatrixDisplay,
+    auc,
+    classification_report,
+    confusion_matrix,
+    r2_score,
+    roc_auc_score,
+    roc_curve,
+)
 
 import nkululeko.glob_conf as glob_conf
 from nkululeko.plots import Plots
@@ -167,7 +171,7 @@ class Reporter:
             probas["uncertainty"] = uncertainty
             probas["correct"] = probas.predicted == probas.truth
             sp = self.util.get_pred_name()
-            
+
             self.probas = probas
             probas.to_csv(sp)
             self.util.debug(f"Saved probabilities to {sp}")
@@ -175,7 +179,13 @@ class Reporter:
             ax, caption = plots.plotcatcont(
                 probas, "correct", "uncertainty", "uncertainty", "correct"
             )
-            plots.save_plot(ax, caption, "Uncertainty", "uncertainty", "samples")
+            plots.save_plot(
+                ax,
+                caption,
+                "Uncertainty",
+                "uncertainty_samples",
+                self.util.get_exp_name(),
+            )
 
     def set_id(self, run, epoch):
         """Make the report identifiable with run and epoch index."""

{nkululeko-0.88.4 → nkululeko-0.88.6}/nkululeko/utils/util.py

@@ -157,8 +157,9 @@ class Util:
 
     def get_pred_name(self):
         store = self.get_path("store")
+        target = self.get_target_name()
         pred_name = self.get_model_description()
-        return f"{store}/pred_{pred_name}.csv"
+        return f"{store}/pred_{target}_{pred_name}.csv"
 
     def is_categorical(self, pd_series):
         """Check if a dataframe column is categorical"""
@@ -217,9 +218,14 @@ class Util:
         return_string = f"{ds}"
         if not only_data:
             mt = self.get_model_description()
-            return_string = return_string + "_" + mt
+            target = self.get_target_name()
+            return_string = return_string + "_" + target + "_" + mt
         return return_string.replace("__", "_")
 
+    def get_target_name(self):
+        """Get a string as name from all target sets that are used."""
+        return self.config["DATA"]["target"]
+
     def get_model_type(self):
         return self.config["MODEL"]["type"]
 

{nkululeko-0.88.4 → nkululeko-0.88.6/nkululeko.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nkululeko
-Version: 0.88.4
+Version: 0.88.6
 Summary: Machine learning audio prediction experiments based on templates
 Home-page: https://github.com/felixbur/nkululeko
 Author: Felix Burkhardt
@@ -68,7 +68,8 @@ A project to detect speaker characteristics by machine learning experiments with
 
 The idea is to have a framework (based on e.g. sklearn and torch) that can be used to rapidly and automatically analyse audio data and explore machine learning models based on that data.
 
-* NEW with nkululeko: [Finetune transformer-models](http://blog.syntheticspeech.de/2024/05/29/nkululeko-how-to-finetune-a-transformer-model/)
+* NEW with nkululek: [Ensemble learning](http://blog.syntheticspeech.de/2024/06/25/nkululeko-ensemble-classifiers-with-late-fusion/)
+* NEW: [Finetune transformer-models](http://blog.syntheticspeech.de/2024/05/29/nkululeko-how-to-finetune-a-transformer-model/)
 * The latest features can be seen in [the ini-file](./ini_file.md) options that are used to control Nkululeko
 * Below is a [Hello World example](#helloworld) that should set you up fastly, also on [Google Colab](https://colab.research.google.com/drive/1GYNBd5cdZQ1QC3Jm58qoeMaJg3UuPhjw?usp=sharing#scrollTo=4G_SjuF9xeQf), and [with Kaggle](https://www.kaggle.com/felixburk/nkululeko-hello-world-example)
 * [Here's a blog post on how to set up nkululeko on your computer.](http://blog.syntheticspeech.de/2021/08/30/how-to-set-up-your-first-nkululeko-project/)
@@ -203,7 +204,7 @@ All of them take *--config <my_config.ini>* as an argument.
 * **nkululeko.nkululeko**: do machine learning experiments combining features and learners
 * **nkululeko.ensemble**: [combine several nkululeko experiments](http://blog.syntheticspeech.de/2024/06/25/nkululeko-ensemble-classifiers-with-late-fusion/) and report on late fusion results
   * *configurations*: which experiments to combine
-  * *--method* (optional): mean, max, sum, max_class, uncertainty_threshold, uncertainty_weighted, confidence_weighted
+  * *--method* (optional): majority_voting, mean, max, sum, max_class, uncertainty_threshold, uncertainty_weighted, confidence_weighted
   * *--threshold*: uncertainty threshold (1.0 means no threshold)
   * *--outfile* (optional): name of CSV file for output
   * *--no_labels* (optional): indicate that no ground truth is given 
@@ -359,6 +360,16 @@ F. Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben and Björn Schulle
 Changelog
 =========
 
+Version 0.88.6
+--------------
+* added test speaker assign
+
+Version 0.88.5
+--------------
+* add a unique name to the uncertainty plot
+* fix error in speaker embedding (still need speechbrain < 1.0)
+* add get_target_name function in util
+
 Version 0.88.4
 --------------
 * added more ensemble methods, e.g. based on uncertainty

{nkululeko-0.88.4 → nkululeko-0.88.6}/nkululeko.egg-info/SOURCES.txt

@@ -6,8 +6,6 @@ setup.cfg
 setup.py
 data/aesdd/process_database.py
 data/androids/process_database.py
-data/androids_orig/process_database.py
-data/androids_test/process_database.py
 data/ased/process_database.py
 data/asvp-esd/process_database.py
 data/baved/process_database.py

nkululeko-0.88.4/data/androids_orig/process_database.py

@@ -1,93 +0,0 @@
-"""
-This folder is to import the 
-Androids-corpus depression
-database to nkululeko.
-
-I used the version downloadable from [github](https://github.com/CheyneyComputerScience/CREMA-D)
-
-downloaded April 27th 2023
-
-I used the version downloadable from [Dropbox, mentioned in this github page](https://github.com/androidscorpus/data)
-
-Download and unzip the file Androids-corpus.zip to the current folder
-
-Usage: `python process_database.py`
-
-"""
-
-import pandas as pd
-import os
-import audeer
-
-dataset_name = 'androids'
-data_root = './Androids-Corpus/'
-
-# read in the fold list 
-fold_dict = {}
-df_fold = pd.read_csv(f'{data_root}fold-lists.csv')
-for i in range(df_fold.shape[0]): 
-    for j in range(df_fold.shape[1]): 
-        try:
-            value = df_fold.iloc[i, j].replace('\'', '')
-        except AttributeError:
-            value = 'na' 
-        if j in range(7, 12):
-            fold_dict[value] = j - 7
-        else:
-            fold_dict[value] = j 
-directory_list = audeer.list_file_names(data_root, filetype='wav', recursive=True, basenames=True)
-
-depressions, speakers, educations, genders, ages, tasks, folds = [], [], [], [], [], [], []
-file_paths = []
-print(len(directory_list))
-gender_map = {'F':'female', 'M':'male'}
-depression_map = {'P':'depressed', 'C':'control'}
-
-for file in directory_list:
-    # storing file paths
-    # file = file.replace(os.getcwd(), '.')
-    file_paths.append(data_root+file)
-    # storing file emotions
-    fn = audeer.basename_wo_ext(file)
-
-    # The naming convention of the audio files is as follows:
-    # nn_XGmm_t.wav
-    # where nn is a unique integer identifier such that, in a given group, files with the same nn contain the voice of the same speaker (there is a trailing 0 for numbers lower than 10), X is an alphabetic character corresponding to the speaker’s condition (P for depression patient and C for control), G is an alphabetic character that stands for the speaker’s gender (M for male and F for female), mm is a two-digits integer number corresponding to the speaker’s age, and t is an integer number between 1 and 4 accounting for the education level (1 corresponds to primary school and 4 corresponds to university). The letter X was used for the 2 participants who did not provide information about this aspect. There is no indication of the task because recordings corresponding to RT and IT are stored in different directories.
-    if 'Reading-Task'in file:
-        task = 'reading'
-    elif 'Interview-Task' in file:
-        task = 'interview'
-    else:
-        print('ERROR: task undefined')
-        exit(-1)
- 
-    part = fn.split('_')
-    dir_name = f'{part[0]}_{part[1]}_{part[2]}'
-    depression = part[1][0]
-    speaker = f'{depression}_{part[0]}'
-    gender = part[1][1]
-    age = part[1][2:4]
-    education = part[2]
-    depressions.append(depression_map[depression])
-    speakers.append(speaker)
-    genders.append(gender_map[gender])
-    ages.append(age)
-    tasks.append(task)
-    educations.append(education)
-    folds.append(fold_dict[dir_name])
-#    print(f'{file} {speaker}')
-
-
-# dataframe for emotion of files
-df = pd.DataFrame({'file':file_paths, 
-                   'speaker':speakers, 
-                   'gender':genders, 
-                   'age':ages, 
-                   'task':tasks, 
-                   'depression':depressions, 
-                   'education':educations, 
-                   'fold':folds})
-
-df = df.set_index('file')
-df.head()
-df.to_csv(f'{dataset_name}.csv')

nkululeko-0.88.4/data/androids_test/process_database.py

@@ -1,93 +0,0 @@
-"""
-This folder is to import the 
-Androids-corpus depression
-database to nkululeko.
-
-I used the version downloadable from [github](https://github.com/CheyneyComputerScience/CREMA-D)
-
-downloaded April 27th 2023
-
-I used the version downloadable from [Dropbox, mentioned in this github page](https://github.com/androidscorpus/data)
-
-Download and unzip the file Androids-corpus.zip to the current folder
-
-Usage: `python process_database.py`
-
-"""
-
-import pandas as pd
-import os
-import audeer
-
-dataset_name = 'androids'
-data_root = './Androids-Corpus/'
-
-# read in the fold list 
-fold_dict = {}
-df_fold = pd.read_csv(f'{data_root}fold-lists.csv')
-for i in range(df_fold.shape[0]): 
-    for j in range(df_fold.shape[1]): 
-        try:
-            value = df_fold.iloc[i, j].replace('\'', '')
-        except AttributeError:
-            value = 'na' 
-        if j in range(7, 12):
-            fold_dict[value] = j - 7
-        else:
-            fold_dict[value] = j 
-directory_list = audeer.list_file_names(data_root, filetype='wav', recursive=True, basenames=True)
-
-depressions, speakers, educations, genders, ages, tasks, folds = [], [], [], [], [], [], []
-file_paths = []
-print(len(directory_list))
-gender_map = {'F':'female', 'M':'male'}
-depression_map = {'P':'depressed', 'C':'control'}
-
-for file in directory_list:
-    # storing file paths
-    # file = file.replace(os.getcwd(), '.')
-    file_paths.append(data_root+file)
-    # storing file emotions
-    fn = audeer.basename_wo_ext(file)
-
-    # The naming convention of the audio files is as follows:
-    # nn_XGmm_t.wav
-    # where nn is a unique integer identifier such that, in a given group, files with the same nn contain the voice of the same speaker (there is a trailing 0 for numbers lower than 10), X is an alphabetic character corresponding to the speaker’s condition (P for depression patient and C for control), G is an alphabetic character that stands for the speaker’s gender (M for male and F for female), mm is a two-digits integer number corresponding to the speaker’s age, and t is an integer number between 1 and 4 accounting for the education level (1 corresponds to primary school and 4 corresponds to university). The letter X was used for the 2 participants who did not provide information about this aspect. There is no indication of the task because recordings corresponding to RT and IT are stored in different directories.
-    if 'Reading-Task'in file:
-        task = 'reading'
-    elif 'Interview-Task' in file:
-        task = 'interview'
-    else:
-        print('ERROR: task undefined')
-        exit(-1)
- 
-    part = fn.split('_')
-    dir_name = f'{part[0]}_{part[1]}_{part[2]}'
-    depression = part[1][0]
-    speaker = f'{depression}_{part[0]}'
-    gender = part[1][1]
-    age = part[1][2:4]
-    education = part[2]
-    depressions.append(depression_map[depression])
-    speakers.append(speaker)
-    genders.append(gender_map[gender])
-    ages.append(age)
-    tasks.append(task)
-    educations.append(education)
-    folds.append(fold_dict[dir_name])
-#    print(f'{file} {speaker}')
-
-
-# dataframe for emotion of files
-df = pd.DataFrame({'file':file_paths, 
-                   'speaker':speakers, 
-                   'gender':genders, 
-                   'age':ages, 
-                   'task':tasks, 
-                   'depression':depressions, 
-                   'education':educations, 
-                   'fold':folds})
-
-df = df.set_index('file')
-df.head()
-df.to_csv(f'{dataset_name}.csv')