nkululeko 0.79.5__tar.gz → 0.80.1__tar.gz

{nkululeko-0.79.5 → nkululeko-0.80.1}/CHANGELOG.md

@@ -1,6 +1,16 @@
 Changelog
 =========
 
+Version 0.80.1
+--------------
+* added support for string value in import_features
++ added support for multiple extra training databases when doing multi-db experiments
+
+Version 0.80.0
+--------------
+* fixed bug no feature import
+* add support for multiple import feature files
+
 Version 0.79.5
 --------------
 * fixed bug on demo without in- or output
@@ -8,7 +18,7 @@ Version 0.79.5
 
 Version 0.79.4
 --------------
-* added funcionality in demo for regression
+* added functionality in demo for regression
 
 Version 0.79.3
 --------------

{nkululeko-0.79.5/nkululeko.egg-info → nkululeko-0.80.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nkululeko
-Version: 0.79.5
+Version: 0.80.1
 Summary: Machine learning audio prediction experiments based on templates
 Home-page: https://github.com/felixbur/nkululeko
 Author: Felix Burkhardt
@@ -188,16 +188,22 @@ combine_per_speaker = mode
 Read the [Hello World example](#hello-world-example) for initial usage with Emo-DB dataset.
 
 Here is an overview of the interfaces/modules:
+
+All of them take *--config <my_config.ini>* as an argument.
+
 * **nkululeko.nkululeko**: do machine learning experiments combining features and learners
-* **nkululeko.multidb**: do multiple experiments, comparing several databases cross and in itself
-* **nkululeko.demo**: demo the current best model on the command line
-* **nkululeko.test**: predict a series of files with the current best model
-* **nkululeko.explore**: perform data exploration
-* **nkululeko.augment**: augment the current training data
-* **nkululeko.aug_train**: augment the current training data and do a training including this data
-* **nkululeko.predict**: predict features like SNR, MOS, arousal/valence, age/gender, with DNN models
-* **nkululeko.segment**: segment a database based on VAD (voice activity detection)
-* **nkululeko.resample**: check on all sampling rates and change to 16kHz 
+* **nkululeko.multidb**: do [multiple experiments](http://blog.syntheticspeech.de/2024/01/02/nkululeko-compare-several-databases/), comparing several databases cross and in itself
+* **nkululeko.demo**: [demo the current best model](http://blog.syntheticspeech.de/2022/09/01/nkululeko-how-to-evaluate-a-test-set-with-a-given-best-model/) on the command line
+  * *--list* (optional) list of input files
+  * *--file* (optional) name of input file
+  * *--outfile* (optional) name of CSV file for output
+* **nkululeko.test**: predict a [given data set](http://blog.syntheticspeech.de/2022/09/01/nkululeko-how-to-evaluate-a-test-set-with-a-given-best-model/) with the current best model
+* **nkululeko.explore**: perform [data exploration](http://blog.syntheticspeech.de/2023/05/11/nkululeko-how-to-visualize-your-data-distribution/)
+* **nkululeko.augment**: [augment](http://blog.syntheticspeech.de/2023/03/13/nkululeko-how-to-augment-the-training-set/) the current training data
+* **nkululeko.aug_train**: augment the current training data [and do a training](http://blog.syntheticspeech.de/2023/03/13/nkululeko-how-to-augment-the-training-set/) including this data
+* **nkululeko.predict**: [predict features](http://blog.syntheticspeech.de/2023/08/16/nkululeko-how-to-predict-labels-for-your-data-from-existing-models-and-check-them/) like SNR, MOS, arousal/valence, age/gender, with DNN models
+* **nkululeko.segment**: [segment a database](http://blog.syntheticspeech.de/2023/07/14/nkululeko-segmenting-a-database/) based on VAD (voice activity detection)
+* **nkululeko.resample**: check on all [sampling rates and change](http://blog.syntheticspeech.de/2023/08/31/how-to-fix-different-sampling-rates-in-a-dataset-with-nkululeko/) to 16kHz 
 
 There's my [blog](http://blog.syntheticspeech.de/?s=nkululeko) with tutorials:
 * [Introduction](http://blog.syntheticspeech.de/2021/08/04/machine-learning-experiment-framework/)
@@ -317,6 +323,16 @@ F. Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben and Björn Schulle
 Changelog
 =========
 
+Version 0.80.1
+--------------
+* added support for string value in import_features
++ added support for multiple extra training databases when doing multi-db experiments
+
+Version 0.80.0
+--------------
+* fixed bug no feature import
+* add support for multiple import feature files
+
 Version 0.79.5
 --------------
 * fixed bug on demo without in- or output
@@ -324,7 +340,7 @@ Version 0.79.5
 
 Version 0.79.4
 --------------
-* added funcionality in demo for regression
+* added functionality in demo for regression
 
 Version 0.79.3
 --------------

{nkululeko-0.79.5 → nkululeko-0.80.1}/README.md

@@ -142,16 +142,22 @@ combine_per_speaker = mode
 Read the [Hello World example](#hello-world-example) for initial usage with Emo-DB dataset.
 
 Here is an overview of the interfaces/modules:
+
+All of them take *--config <my_config.ini>* as an argument.
+
 * **nkululeko.nkululeko**: do machine learning experiments combining features and learners
-* **nkululeko.multidb**: do multiple experiments, comparing several databases cross and in itself
-* **nkululeko.demo**: demo the current best model on the command line
-* **nkululeko.test**: predict a series of files with the current best model
-* **nkululeko.explore**: perform data exploration
-* **nkululeko.augment**: augment the current training data
-* **nkululeko.aug_train**: augment the current training data and do a training including this data
-* **nkululeko.predict**: predict features like SNR, MOS, arousal/valence, age/gender, with DNN models
-* **nkululeko.segment**: segment a database based on VAD (voice activity detection)
-* **nkululeko.resample**: check on all sampling rates and change to 16kHz 
+* **nkululeko.multidb**: do [multiple experiments](http://blog.syntheticspeech.de/2024/01/02/nkululeko-compare-several-databases/), comparing several databases cross and in itself
+* **nkululeko.demo**: [demo the current best model](http://blog.syntheticspeech.de/2022/09/01/nkululeko-how-to-evaluate-a-test-set-with-a-given-best-model/) on the command line
+  * *--list* (optional) list of input files
+  * *--file* (optional) name of input file
+  * *--outfile* (optional) name of CSV file for output
+* **nkululeko.test**: predict a [given data set](http://blog.syntheticspeech.de/2022/09/01/nkululeko-how-to-evaluate-a-test-set-with-a-given-best-model/) with the current best model
+* **nkululeko.explore**: perform [data exploration](http://blog.syntheticspeech.de/2023/05/11/nkululeko-how-to-visualize-your-data-distribution/)
+* **nkululeko.augment**: [augment](http://blog.syntheticspeech.de/2023/03/13/nkululeko-how-to-augment-the-training-set/) the current training data
+* **nkululeko.aug_train**: augment the current training data [and do a training](http://blog.syntheticspeech.de/2023/03/13/nkululeko-how-to-augment-the-training-set/) including this data
+* **nkululeko.predict**: [predict features](http://blog.syntheticspeech.de/2023/08/16/nkululeko-how-to-predict-labels-for-your-data-from-existing-models-and-check-them/) like SNR, MOS, arousal/valence, age/gender, with DNN models
+* **nkululeko.segment**: [segment a database](http://blog.syntheticspeech.de/2023/07/14/nkululeko-segmenting-a-database/) based on VAD (voice activity detection)
+* **nkululeko.resample**: check on all [sampling rates and change](http://blog.syntheticspeech.de/2023/08/31/how-to-fix-different-sampling-rates-in-a-dataset-with-nkululeko/) to 16kHz 
 
 There's my [blog](http://blog.syntheticspeech.de/?s=nkululeko) with tutorials:
 * [Introduction](http://blog.syntheticspeech.de/2021/08/04/machine-learning-experiment-framework/)

{nkululeko-0.79.5 → nkululeko-0.80.1}/nkululeko/constants.py

@@ -1,2 +1,2 @@
-VERSION="0.79.5"
+VERSION="0.80.1"
 SAMPLING_RATE = 16000

nkululeko-0.80.1/nkululeko/demo_feats.py

@@ -0,0 +1,71 @@
+# demo_feats.py
+# Test some features extracted
+
+from nkululeko.experiment import Experiment
+import configparser
+from nkululeko.utils.util import Util
+from nkululeko.constants import VERSION
+import argparse
+import os
+
+
+def main(src_dir):
+    parser = argparse.ArgumentParser(description="Call the nkululeko framework.")
+    parser.add_argument("--config", default="exp.ini", help="The base configuration")
+    parser.add_argument(
+        "--file", help="A file that should be processed (16kHz mono wav)"
+    )
+    parser.add_argument(
+        "--list",
+        help=(
+            "A file with a list of files, one per line, that should be"
+            " processed (16kHz mono wav)"
+        ),
+        nargs="?",
+        default=None,
+    )
+    parser.add_argument(
+        "--outfile",
+        help=("A filename to store the results in CSV"),
+        nargs="?",
+        default=None,
+    )
+    args = parser.parse_args()
+    if args.config is not None:
+        config_file = args.config
+    else:
+        config_file = f"{src_dir}/exp.ini"
+
+    # test if the configuration file exists
+    if not os.path.isfile(config_file):
+        print(f"ERROR: no such file: {config_file}")
+        exit()
+
+    # load one configuration per experiment
+    config = configparser.ConfigParser()
+    config.read(config_file)
+
+    # create a new experiment
+    expr = Experiment(config)
+    module = "demo_feats"
+    expr.set_module(module)
+    util = Util(module)
+    util.debug(
+        f"running {expr.name} from config {config_file}, nkululeko version"
+        f" {VERSION}"
+    )
+
+    if args.file is None and args.list is None:
+        expr.demo_feats(None, False, args.outfile)
+    else:
+        if args.list is None:
+            expr.demo_feats(args.file, False, args.outfile)
+        else:
+            expr.demo_feats(args.list, True, args.outfile)
+
+    print("DONE")
+
+
+if __name__ == "__main__":
+    cwd = os.path.dirname(os.path.abspath(__file__))
+    main(cwd)  # use this if you want to state the config file path on command line

{nkululeko-0.79.5 → nkululeko-0.80.1}/nkululeko/experiment.py

@@ -357,7 +357,7 @@ class Experiment:
             self.df_train = self.df_train[
                 self.df_train.index.isin(self.feats_train.index)
             ]
-            self.util.warn(f"mew train labels shape: {self.df_train.shape[0]}")
+            self.util.warn(f"new train labels shape: {self.df_train.shape[0]}")
         if self.feats_test.shape[0] < self.df_test.shape[0]:
             self.util.warn(
                 f"test feats ({self.feats_test.shape[0]}) != test labels"

nkululeko-0.80.1/nkululeko/feat_extract/feats_import.py

@@ -0,0 +1,43 @@
+# feats_import.py
+
+import os
+import ast
+import audformat
+import pandas as pd
+from nkululeko.utils.util import Util
+from nkululeko.feat_extract.featureset import Featureset
+
+
+class Importset(Featureset):
+    """Class to import features that have been compiled elsewhere"""
+
+    def __init__(self, name, data_df):
+        super().__init__(name, data_df)
+
+    def extract(self):
+        """Import the features."""
+        self.util.debug(f"importing features for {self.name}")
+        try:
+            feat_import_files = self.util.config_val("FEATS", "import_file", False)
+            feat_import_files = ast.literal_eval(feat_import_files)
+        except ValueError as e:
+            self.util.error(
+                "feature type == import needs import_file = ['file1', 'filex']"
+            )
+        except SyntaxError as se:
+            if type(feat_import_files) == str:
+                feat_import_files = [feat_import_files]
+            else:
+                self.util.error(f"import_file is wrong: {feat_import_files}")
+
+        feat_df = pd.DataFrame()
+        for feat_import_file in feat_import_files:
+            if not os.path.isfile(feat_import_file):
+                self.util.error(f"no import file: {feat_import_file}")
+            df = audformat.utils.read_csv(feat_import_file)
+            df = df[df.index.isin(self.data_df.index)]
+            feat_df = pd.concat([feat_df, df])
+        if feat_df.shape[0] == 0:
+            self.util.error(f"Imported features for data set {self.name} not found!")
+        # and assign to be the "official" feature set
+        self.df = feat_df

{nkululeko-0.79.5 → nkululeko-0.80.1}/nkululeko/multidb.py

@@ -9,7 +9,6 @@ import matplotlib.pyplot as plt
 import matplotlib.cm as cm
 import numpy as np
 import os
-import audeer
 from nkululeko.experiment import Experiment
 import configparser
 from nkululeko.utils.util import Util
@@ -41,9 +40,9 @@ def main(src_dir):
     last_epochs = np.zeros(dim * dim).reshape([dim, dim])
     # check if some data should be added to training
     try:
-        extra_train = config["CROSSDB"]["train_extra"]
+        extra_trains = config["CROSSDB"]["train_extra"]
     except KeyError:
-        extra_train = False
+        extra_trains = False
 
     for i in range(dim):
         for j in range(dim):
@@ -54,9 +53,12 @@ def main(src_dir):
             if i == j:
                 dataset = datasets[i]
                 print(f"running {dataset}")
-                if extra_train:
-                    config["DATA"]["databases"] = f"['{dataset}', '{extra_train}']"
-                    config["DATA"][f"{extra_train}.split_strategy"] = "train"
+                if extra_trains:
+                    extra_trains_1 = extra_trains.removeprefix("[").removesuffix("]")
+                    config["DATA"]["databases"] = f"['{dataset}', {extra_trains_1}]"
+                    extra_trains_2 = ast.literal_eval(extra_trains)
+                    for extra_train in extra_trains_2:
+                        config["DATA"][f"{extra_train}.split_strategy"] = "train"
                 else:
                     config["DATA"]["databases"] = f"['{dataset}']"
                 config["EXP"]["name"] = dataset
@@ -64,13 +66,16 @@ def main(src_dir):
                 train = datasets[i]
                 test = datasets[j]
                 print(f"running train: {train}, test: {test}")
-                if extra_train:
+                if extra_trains:
+                    extra_trains_1 = extra_trains.removeprefix("[").removesuffix("]")
                     config["DATA"][
                         "databases"
-                    ] = f"['{train}', '{test}', '{extra_train}']"
+                    ] = f"['{train}', '{test}', {extra_trains_1}]"
                     config["DATA"][f"{test}.split_strategy"] = "test"
                     config["DATA"][f"{train}.split_strategy"] = "train"
-                    config["DATA"][f"{extra_train}.split_strategy"] = "train"
+                    extra_trains_2 = ast.literal_eval(extra_trains)
+                    for extra_train in extra_trains_2:
+                        config["DATA"][f"{extra_train}.split_strategy"] = "train"
                 else:
                     config["DATA"]["databases"] = f"['{train}', '{test}']"
                     config["DATA"][f"{test}.split_strategy"] = "test"

{nkululeko-0.79.5 → nkululeko-0.80.1/nkululeko.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nkululeko
-Version: 0.79.5
+Version: 0.80.1
 Summary: Machine learning audio prediction experiments based on templates
 Home-page: https://github.com/felixbur/nkululeko
 Author: Felix Burkhardt
@@ -188,16 +188,22 @@ combine_per_speaker = mode
 Read the [Hello World example](#hello-world-example) for initial usage with Emo-DB dataset.
 
 Here is an overview of the interfaces/modules:
+
+All of them take *--config <my_config.ini>* as an argument.
+
 * **nkululeko.nkululeko**: do machine learning experiments combining features and learners
-* **nkululeko.multidb**: do multiple experiments, comparing several databases cross and in itself
-* **nkululeko.demo**: demo the current best model on the command line
-* **nkululeko.test**: predict a series of files with the current best model
-* **nkululeko.explore**: perform data exploration
-* **nkululeko.augment**: augment the current training data
-* **nkululeko.aug_train**: augment the current training data and do a training including this data
-* **nkululeko.predict**: predict features like SNR, MOS, arousal/valence, age/gender, with DNN models
-* **nkululeko.segment**: segment a database based on VAD (voice activity detection)
-* **nkululeko.resample**: check on all sampling rates and change to 16kHz 
+* **nkululeko.multidb**: do [multiple experiments](http://blog.syntheticspeech.de/2024/01/02/nkululeko-compare-several-databases/), comparing several databases cross and in itself
+* **nkululeko.demo**: [demo the current best model](http://blog.syntheticspeech.de/2022/09/01/nkululeko-how-to-evaluate-a-test-set-with-a-given-best-model/) on the command line
+  * *--list* (optional) list of input files
+  * *--file* (optional) name of input file
+  * *--outfile* (optional) name of CSV file for output
+* **nkululeko.test**: predict a [given data set](http://blog.syntheticspeech.de/2022/09/01/nkululeko-how-to-evaluate-a-test-set-with-a-given-best-model/) with the current best model
+* **nkululeko.explore**: perform [data exploration](http://blog.syntheticspeech.de/2023/05/11/nkululeko-how-to-visualize-your-data-distribution/)
+* **nkululeko.augment**: [augment](http://blog.syntheticspeech.de/2023/03/13/nkululeko-how-to-augment-the-training-set/) the current training data
+* **nkululeko.aug_train**: augment the current training data [and do a training](http://blog.syntheticspeech.de/2023/03/13/nkululeko-how-to-augment-the-training-set/) including this data
+* **nkululeko.predict**: [predict features](http://blog.syntheticspeech.de/2023/08/16/nkululeko-how-to-predict-labels-for-your-data-from-existing-models-and-check-them/) like SNR, MOS, arousal/valence, age/gender, with DNN models
+* **nkululeko.segment**: [segment a database](http://blog.syntheticspeech.de/2023/07/14/nkululeko-segmenting-a-database/) based on VAD (voice activity detection)
+* **nkululeko.resample**: check on all [sampling rates and change](http://blog.syntheticspeech.de/2023/08/31/how-to-fix-different-sampling-rates-in-a-dataset-with-nkululeko/) to 16kHz 
 
 There's my [blog](http://blog.syntheticspeech.de/?s=nkululeko) with tutorials:
 * [Introduction](http://blog.syntheticspeech.de/2021/08/04/machine-learning-experiment-framework/)
@@ -317,6 +323,16 @@ F. Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben and Björn Schulle
 Changelog
 =========
 
+Version 0.80.1
+--------------
+* added support for string value in import_features
++ added support for multiple extra training databases when doing multi-db experiments
+
+Version 0.80.0
+--------------
+* fixed bug no feature import
+* add support for multiple import feature files
+
 Version 0.79.5
 --------------
 * fixed bug on demo without in- or output
@@ -324,7 +340,7 @@ Version 0.79.5
 
 Version 0.79.4
 --------------
-* added funcionality in demo for regression
+* added functionality in demo for regression
 
 Version 0.79.3
 --------------

{nkululeko-0.79.5 → nkululeko-0.80.1}/nkululeko.egg-info/SOURCES.txt

@@ -60,6 +60,7 @@ nkululeko/augment.py
 nkululeko/cacheddataset.py
 nkululeko/constants.py
 nkululeko/demo.py
+nkululeko/demo_feats.py
 nkululeko/demo_predictor.py
 nkululeko/experiment.py
 nkululeko/explore.py

nkululeko-0.79.5/nkululeko/feat_extract/feats_import.py

@@ -1,45 +0,0 @@
-# feats_import.py
-
-from nkululeko.utils.util import Util
-from nkululeko.feat_extract.featureset import Featureset
-import os
-import pandas as pd
-import audformat
-
-
-class Importset(Featureset):
-    """Class to import features that have been compiled elsewhere"""
-
-    def __init__(self, name, data_df):
-        super().__init__(name, data_df)
-
-    def extract(self):
-        """Import the features or load them from disk if present."""
-        store = self.util.get_path("store")
-        storage = f"{store}{self.name}.pkl"
-        extract = eval(self.util.config_val("FEATS", "needs_feature_extraction", False))
-        no_reuse = eval(self.util.config_val("FEATS", "no_reuse", "False"))
-        feat_import_file = self.util.config_val("FEATS", "import_file", False)
-        if not os.path.isfile(feat_import_file):
-            self.util.warn(f"no import file: {feat_import_file}")
-        if extract or no_reuse or not os.path.isfile(storage):
-            self.util.debug(f"importing features for {self.name}")
-            # df = pd.read_csv(feat_import_file, sep=',', header=0,
-            #     index_col=['file', 'start', 'end'])
-            df = audformat.utils.read_csv(feat_import_file)
-            # scale features before use?
-            # from sklearn.preprocessing import StandardScaler
-            # scaler = StandardScaler()
-            # scaled_features = scaler.fit_transform(df.values)
-            # df = pd.DataFrame(scaled_features, index=df.index, columns=df.columns)
-            # use only the rows from the data index
-            # df = self.data_df.join(df).drop(columns=self.data_df.columns)
-            df = df.loc[self.data_df.index]
-            # df = pd.concat([self.data_df, df], axis=1, join="inner").drop(columns=self.data_df.columns)
-            # in any case, store to disk for later use
-            df.to_pickle(storage)
-            # and assign to be the "official" feature set
-            self.df = df
-        else:
-            self.util.debug("reusing imported features.")
-            self.df = pd.read_pickle(storage)