aucurriculum 0.1.0__tar.gz

aucurriculum-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Simon Rampp, Manuel Milling, Andreas Triantafyllopoulos, Björn W. Schuller.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aucurriculum-0.1.0/PKG-INFO

@@ -0,0 +1,58 @@
+Metadata-Version: 2.1
+Name: aucurriculum
+Version: 0.1.0
+Summary: A Curriculum Learning Toolkit for Deep Learning Tasks built on top of autrainer.
+Home-page: https://github.com/autrainer/aucurriculum
+License: MIT
+Keywords: curriculum learning,autrainer,computer audition,deep learning,artificial intelligence,pytorch,hydra-core,command-line,yaml
+Author: Simon Rampp
+Author-email: simon.rampp@tum.de
+Requires-Python: >=3.9,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: autrainer (==0.4.0)
+Project-URL: Documentation, https://autrainer.github.io/aucurriculum/
+Project-URL: Repository, https://github.com/autrainer/aucurriculum
+Description-Content-Type: text/markdown
+
+<div align="center">
+  <img src="https://autrainer.github.io/aucurriculum/_images/logo_banner.png" alt="aucurriculum — A Curriculum Learning Toolkit for Deep Learning Tasks built on top of autrainer">
+</div>
+
+# aucurriculum
+
+[![aucurriculum PyPI Version](https://img.shields.io/pypi/v/aucurriculum?logo=pypi&logoColor=b4befe&color=b4befe)](https://pypi.org/project/aucurriculum/)
+[![aucurriculum Python Versions](https://img.shields.io/pypi/pyversions/aucurriculum?logo=python&logoColor=b4befe&color=b4befe)](https://pypi.org/project/aucurriculum/)
+[![aucurriculum GitHub License](https://img.shields.io/badge/license-MIT-b4befe?logo=c)](https://github.com/autrainer/aucurriculum/blob/main/LICENSE)
+
+A Curriculum Learning Toolkit for Deep Learning Tasks built on top of [autrainer](https://github.com/autrainer/autrainer).
+
+## Installation
+
+To install _aucurriculum_, first ensure that PyTorch (along with torchvision and torchaudio) version 2.0 or higher is installed.
+For installation instructions, refer to the [PyTorch website](https://pytorch.org/get-started/locally/).
+
+It is recommended to install _aucurriculum_ within a virtual environment.
+To create a new virtual environment, refer to the [Python venv documentation](https://docs.python.org/3/library/venv.html).
+
+Next, install _aucurriculum_ using _pip_.
+
+```bash
+pip install aucurriculum
+```
+
+To install _aucurriculum_ from source, refer to the [contribution guide](https://autrainer.github.io/aucurriculum/development/contributing.html).
+
+## Next Steps
+
+To get started using _aucurriculum_, the [quickstart guide](https://autrainer.github.io/aucurriculum/usage/quickstart.html) outlines the creation of a simple training configuration
+and [tutorials](https://autrainer.github.io/aucurriculum/usage/tutorials.html) provide examples for implementing custom scoring and pacing functions including their configurations.
+
+For a complete list of available CLI commands, refer to the [CLI reference](https://autrainer.github.io/aucurriculum/usage/cli_reference.html) or the [CLI wrapper](https://autrainer.github.io/aucurriculum/usage/cli_wrapper.html).
+

aucurriculum-0.1.0/README.md

@@ -0,0 +1,34 @@
+<div align="center">
+  <img src="https://autrainer.github.io/aucurriculum/_images/logo_banner.png" alt="aucurriculum — A Curriculum Learning Toolkit for Deep Learning Tasks built on top of autrainer">
+</div>
+
+# aucurriculum
+
+[![aucurriculum PyPI Version](https://img.shields.io/pypi/v/aucurriculum?logo=pypi&logoColor=b4befe&color=b4befe)](https://pypi.org/project/aucurriculum/)
+[![aucurriculum Python Versions](https://img.shields.io/pypi/pyversions/aucurriculum?logo=python&logoColor=b4befe&color=b4befe)](https://pypi.org/project/aucurriculum/)
+[![aucurriculum GitHub License](https://img.shields.io/badge/license-MIT-b4befe?logo=c)](https://github.com/autrainer/aucurriculum/blob/main/LICENSE)
+
+A Curriculum Learning Toolkit for Deep Learning Tasks built on top of [autrainer](https://github.com/autrainer/autrainer).
+
+## Installation
+
+To install _aucurriculum_, first ensure that PyTorch (along with torchvision and torchaudio) version 2.0 or higher is installed.
+For installation instructions, refer to the [PyTorch website](https://pytorch.org/get-started/locally/).
+
+It is recommended to install _aucurriculum_ within a virtual environment.
+To create a new virtual environment, refer to the [Python venv documentation](https://docs.python.org/3/library/venv.html).
+
+Next, install _aucurriculum_ using _pip_.
+
+```bash
+pip install aucurriculum
+```
+
+To install _aucurriculum_ from source, refer to the [contribution guide](https://autrainer.github.io/aucurriculum/development/contributing.html).
+
+## Next Steps
+
+To get started using _aucurriculum_, the [quickstart guide](https://autrainer.github.io/aucurriculum/usage/quickstart.html) outlines the creation of a simple training configuration
+and [tutorials](https://autrainer.github.io/aucurriculum/usage/tutorials.html) provide examples for implementing custom scoring and pacing functions including their configurations.
+
+For a complete list of available CLI commands, refer to the [CLI reference](https://autrainer.github.io/aucurriculum/usage/cli_reference.html) or the [CLI wrapper](https://autrainer.github.io/aucurriculum/usage/cli_wrapper.html).

aucurriculum-0.1.0/aucurriculum/__init__.py

@@ -0,0 +1,36 @@
+from autrainer.core.constants import (
+    ExportConstants,
+    NamingConstants,
+)
+
+from aucurriculum.core.constants import CurriculumConstants
+from aucurriculum.core.wrappers import main
+
+
+# always overlay the autrainer constants
+NamingConstants().CONFIG_DIRS = (
+    NamingConstants().CONFIG_DIRS + CurriculumConstants().CONFIG_DIRS
+)
+NamingConstants().NAMING_CONVENTION = NamingConstants().NAMING_CONVENTION[
+    :-1
+] + [
+    "curriculum",
+    "curriculum.sampling",
+    "curriculum.scoring",
+    "curriculum.pacing",
+    "curriculum.pacing.initial_size",
+    "curriculum.pacing.final_iteration",
+    "seed",
+]
+NamingConstants().VALID_AGGREGATIONS = list(
+    set(NamingConstants().NAMING_CONVENTION)
+    - set(NamingConstants().INVALID_AGGREGATIONS)
+)
+ExportConstants().LOGGING_DEPTH = 3
+ExportConstants().IGNORE_PARAMS = ExportConstants().IGNORE_PARAMS + [
+    "curriculum.short",
+    "curriculum.sampling.short",
+]
+
+__version__ = "0.1.0"
+__all__ = ["main"]

aucurriculum-0.1.0/aucurriculum/cli/__init__.py

@@ -0,0 +1,33 @@
+from autrainer.core.scripts import (
+    inference,
+    rm_failed,
+    rm_states,
+)
+
+from aucurriculum.core.scripts import (
+    create,
+    curriculum,
+    fetch,
+    group,
+    postprocess,
+    preprocess,
+    show,
+    train,
+)
+from aucurriculum.core.scripts import list_configs as list
+
+
+__all__ = [
+    "create",
+    "curriculum",
+    "fetch",
+    "group",
+    "inference",
+    "list",
+    "postprocess",
+    "preprocess",
+    "rm_failed",
+    "rm_states",
+    "show",
+    "train",
+]

aucurriculum-0.1.0/aucurriculum/core/constants/__init__.py

@@ -0,0 +1,4 @@
+from .curriculum_constants import CurriculumConstants
+
+
+__all__ = ["CurriculumConstants"]

aucurriculum-0.1.0/aucurriculum/core/constants/curriculum_constants.py

@@ -0,0 +1,26 @@
+from typing import List
+
+from autrainer.core.constants import AbstractConstants
+
+
+class CurriculumConstants(AbstractConstants):
+    """Singleton for managing the curriculum configurations of
+    `aucurriculum`."""
+
+    _config_dirs = [
+        "curriculum",
+        "curriculum/pacing",
+        "curriculum/sampling",
+        "curriculum/scoring",
+    ]
+
+    @property
+    def CONFIG_DIRS(self) -> List[str]:
+        """Get the curriculum configuration directories for Hydra
+        configurations. Defaults to :attr:`["curriculum", "curriculum/pacing",
+        "curriculum/sampling", "curriculum/scoring"]`.
+
+        Returns:
+            Curriculum configuration directories for Hydra configurations.
+        """
+        return self._config_dirs

aucurriculum-0.1.0/aucurriculum/core/filters/__init__.py

@@ -0,0 +1,4 @@
+from .filters import FilterPartialCurriculum, FilterPlaceholderScoringFunction
+
+
+__all__ = ["FilterPartialCurriculum", "FilterPlaceholderScoringFunction"]

aucurriculum-0.1.0/aucurriculum/core/filters/filters.py

@@ -0,0 +1,45 @@
+from hydra_filter_sweeper import AbstractFilter
+from omegaconf import DictConfig
+
+
+class FilterPlaceholderScoringFunction(AbstractFilter):  # pragma: no cover
+    def filter(self, config: DictConfig, directory: str) -> bool:
+        """Filter out run configurations with placeholder (None) scoring
+        functions.
+
+        Args:
+            config: Run configuration.
+            directory: Directory where the run configuration is stored.
+
+        Returns:
+            Whether to filter out the run configuration.
+        """
+        return config.curriculum.scoring.id == "None"
+
+
+class FilterPartialCurriculum(AbstractFilter):  # pragma: no cover
+    def filter(self, config: DictConfig, directory: str) -> bool:
+        """Filter out run configurations with partial curricula.
+        A partial curriculum is one where the id is not "None" but one or more
+        of the sampling, scoring, and pacing ids are "None".
+
+        Args:
+            config: Run configuration.
+            directory: Directory where the run configuration is stored.
+
+        Returns:
+            Whether to filter out the run configuration.
+        """
+        c = config.curriculum
+        if c.id == "None":
+            if any(
+                id != "None"
+                for id in [c.sampling.id, c.scoring.id, c.pacing.id]
+            ):
+                return True
+            return False
+        if any(
+            id == "None" for id in [c.sampling.id, c.scoring.id, c.pacing.id]
+        ):
+            return True
+        return False

aucurriculum-0.1.0/aucurriculum/core/plotting/__init__.py

@@ -0,0 +1,4 @@
+from .plot_curriculum import CurriculumPlots
+
+
+__all__ = ["CurriculumPlots"]

aucurriculum-0.1.0/aucurriculum/core/plotting/plot_curriculum.py

@@ -0,0 +1,274 @@
+import os
+import warnings
+
+from autrainer.core.plotting import PlotBase
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import seaborn as sns
+
+
+class CurriculumPlots(PlotBase):
+    def __init__(
+        self,
+        output_directory: str,
+        training_type: str,
+        figsize: tuple,
+        latex: bool,
+        filetypes: list,
+        pickle: bool,
+        context: str,
+        palette: str,
+        replace_none: bool,
+        add_titles: bool,
+        add_xlabels: bool,
+        add_ylabels: bool,
+        rcParams: dict,
+    ) -> None:
+        """Plot the sample difficulty scores and pace of curricula.
+
+        Args:
+            output_directory: Output directory to save plots to.
+            training_type: Type of training in ["Epoch", "Step"].
+            figsize: Figure size in inches.
+            latex: Whether to use LaTeX in plots. Requires the `latex` package.
+                To install all necessary dependencies, run:
+                `pip install autrainer[latex]`.
+            filetypes: Filetypes to save plots as.
+            pickle: Whether to save additional pickle files of the plots.
+            context: Context for seaborn plots.
+            palette: Color palette for seaborn plots.
+            replace_none: Whether to replace "None" in labels with "~".
+            add_titles: Whether to add titles to plots.
+            add_xlabels: Whether to add x-labels to plots.
+            add_ylabels: Whether to add y-labels to plots.
+            rcParams: Additional Matplotlib rcParams to set.
+        """
+        super().__init__(
+            output_directory,
+            training_type,
+            figsize,
+            latex,
+            filetypes,
+            pickle,
+            context,
+            palette,
+            replace_none,
+            add_titles,
+            add_xlabels,
+            add_ylabels,
+            rcParams,
+        )
+
+    def plot_score(self, df: pd.DataFrame, score_id: str, num_bins=20) -> None:
+        """Plot the sample difficulty scores by creating histograms,
+        facet grids, and ridge plots visualizing the global and per class
+        sample difficulty distributions.
+
+        Args:
+            df: DataFrame containing the sample difficulty scores ("mean",
+            "ranks") and corresponding labels ("decoded").
+            score_id: ID of the scoring function.
+            num_bins: Number of bins for the histogram. Defaults to 20.
+        """
+        df["decoded"] = df["decoded"].astype(str)
+        order = sorted(df["decoded"].unique())
+        sample_counts = df["decoded"].value_counts().sort_index().values
+
+        self._plot_fn(
+            df=df,
+            score_id=score_id,
+            name="histogram",
+            x="mean",
+            y_label="count",
+            fn=sns.histplot,
+            fn_kargs={"bins": num_bins},
+        )
+        self._plot_facet_grid(
+            df=df,
+            score_id=score_id,
+            name="Count per Class",
+            y_label="count",
+            fn_map=sns.histplot,
+            fn_kargs={
+                "bins": np.arange(0, 1 + (1 / num_bins), 1 / num_bins),
+                "stat": "count",
+            },
+            col_order=order,
+        )
+        percent_char = "\\%" if self.latex else "%"
+        self._plot_facet_grid(
+            df=df,
+            score_id=score_id,
+            name="Percent per Class",
+            y_label=f"ratio [{percent_char}]",
+            fn_map=sns.histplot,
+            fn_kargs={
+                "bins": np.arange(0, 1 + (1 / num_bins), 1 / num_bins),
+                "stat": "percent",
+            },
+            col_order=order,
+            sample_counts=sample_counts,
+        )
+        self._plot_ridge(
+            df=df, score_id=score_id, name="Ridge per Class", row_order=order
+        )
+
+    def _plot_facet_grid(
+        self,
+        df: pd.DataFrame,
+        score_id: str,
+        name: str,
+        y_label: str,
+        fn_map: callable,
+        fn_kargs: dict,
+        col_order: list = None,
+        sample_counts: list = None,
+    ) -> None:
+        fg = sns.FacetGrid(
+            df,
+            col="decoded",
+            col_order=col_order,
+            hue="decoded",
+            col_wrap=5,
+            height=self.figsize[1] / 2,
+            aspect=self.figsize[0] / (self.figsize[1] * 2),
+        )
+        fg.set_titles(col_template="{col_name}")
+        if sample_counts is not None:
+            for ax, ct in zip(fg.axes.flat, sample_counts):
+                ax.set_title(f"{ax.get_title()} ({ct} samples)")
+        fg.map_dataframe(fn_map, x="ranks", **fn_kargs)
+        if self.add_xlabels:
+            fg.set_xlabels("ranks")
+        else:
+            fg.set_xlabels("")
+        if self.add_ylabels:
+            fg.set_ylabels(y_label)
+        else:
+            fg.set_ylabels("")
+        if self.add_titles:
+            fg.figure.suptitle(f"{score_id} {name}", y=0.975)
+        else:
+            fg.figure.suptitle("")
+        path = os.path.join("_plots", score_id)
+        self.save_plot(fg.figure, name.lower().replace(" ", "_"), path)
+
+    def _plot_fn(
+        self,
+        df: pd.DataFrame,
+        score_id: str,
+        name: str,
+        x: str,
+        y_label: str,
+        fn: callable,
+        fn_kargs: dict,
+    ) -> None:
+        fig = plt.figure(figsize=self.figsize)
+        fn(data=df, x=x, **fn_kargs)
+        self._add_label(fig.gca(), x, y_label)
+        path = os.path.join("_plots", score_id)
+        self.save_plot(fig, name.lower().replace(" ", "_"), path)
+
+    def _plot_ridge(
+        self, df: pd.DataFrame, score_id: str, name: str, row_order: list
+    ) -> None:
+        _old_warn = warnings.filters[:]
+        warnings.filterwarnings("ignore", category=UserWarning)
+        height = (self.figsize[1] / 10) * 1.25
+        fg = sns.FacetGrid(
+            df,
+            row="decoded",
+            row_order=row_order,
+            hue="decoded",
+            height=height,
+            aspect=(self.figsize[0] * 1.25) / height,
+        )
+        fg.map_dataframe(
+            sns.kdeplot,
+            x="ranks",
+            bw_adjust=0.5,
+            fill=True,
+        )
+        for ax in fg.axes.flat:
+            ax.patch.set_facecolor("none")
+            ax.xaxis.grid(False)
+        fg.set_titles("")
+        fg.set(yticks=[], ylabel="")
+
+        def label(x, color, label):
+            ax = plt.gca()
+            ax.text(
+                x=0.04,
+                y=0,
+                s=label,
+                ha="right",
+                va="bottom",
+                transform=ax.transAxes,
+            )
+
+        fg.map(label, "ranks")
+        fg.despine(left=True)
+        if self.add_titles:
+            fg.figure.suptitle(f"{score_id} {name}", y=0.975)
+        else:
+            fg.figure.suptitle("")
+
+        path = os.path.join("_plots", score_id)
+        self.save_plot(fg.figure, name.lower().replace(" ", "_"), path)
+        warnings.filters = _old_warn
+
+    def plot_correlation_matrix(self, df: pd.DataFrame, name: str) -> None:
+        """Create and plot the correlation matrix of the scoring functions.
+
+        Args:
+            df: DataFrame containing multiple scoring functions per column.
+            name: Name of the correlation matrix plot.
+        """
+        df.sort_index(axis="columns", inplace=True)
+        fig_height = int(1 + len(df.columns))
+        fig = plt.figure(
+            figsize=(max(self.figsize[0], fig_height), fig_height)
+        )
+        correlation_matrix = df.corr(method="spearman")
+        correlation_matrix.to_csv(
+            os.path.join(self.output_directory, f"{name}.csv")
+        )
+        percent_char = "\\%" if self.latex else "%"
+
+        def escape_percent(x):
+            return f"{(x*100):.2f}{percent_char}"
+
+        sns.heatmap(
+            correlation_matrix,
+            annot=correlation_matrix.map(escape_percent),
+            fmt="",
+            cbar=False,
+            cmap="crest",
+        )
+        fig.gca().set_xticklabels(df.columns, rotation=25)
+        self.save_plot(fig, name)
+
+    def plot_run(self, df: pd.DataFrame) -> None:
+        """Plot the dynamic dataset size determined by the curriculum.
+
+        Args:
+            df: DataFrame containing the run metrics and dataset sizes.
+        """
+        fig = plt.figure(figsize=self.figsize)
+        sns.lineplot(
+            data=df["dataset_size"], dashes=False, drawstyle="steps-post"
+        )
+        self._add_label(fig.gca(), self.training_type, "dataset size")
+        self.save_plot(fig, "pace")
+
+    def plot_pace(self, df: pd.DataFrame) -> None:
+        """Plot the dynamic dataset size of multiple runs.
+
+        Args:
+            df: DataFrame containing multiple runs and their dataset sizes.
+        """
+        fig = plt.figure(figsize=self.figsize)
+        sns.lineplot(data=df, dashes=False, drawstyle="steps-post")
+        self._add_label(fig.gca(), self.training_type, "dataset size")
+        self.save_plot(fig, "pace")

aucurriculum-0.1.0/aucurriculum/core/scripts/__init__.py

@@ -0,0 +1,28 @@
+from .cli_wrapper import fetch, preprocess
+from .create_script import CreateCurriculumScript, create
+from .curriculum_script import CurriculumScript, curriculum
+from .group_script import GroupCurriculumScript, group
+from .list_script import ListCurriculumScript, list_configs
+from .postprocess_script import PostprocessCurriculumScript, postprocess
+from .show_script import ShowCurriculumScript, show
+from .train_script import TrainCurriculumScript, train
+
+
+__all__ = [
+    "create",
+    "CreateCurriculumScript",
+    "curriculum",
+    "CurriculumScript",
+    "fetch",
+    "group",
+    "GroupCurriculumScript",
+    "list_configs",
+    "ListCurriculumScript",
+    "postprocess",
+    "PostprocessCurriculumScript",
+    "preprocess",
+    "show",
+    "ShowCurriculumScript",
+    "train",
+    "TrainCurriculumScript",
+]

aucurriculum-0.1.0/aucurriculum/core/scripts/cli.py

@@ -0,0 +1,79 @@
+import argparse
+from typing import List
+
+from autrainer.core.scripts import (
+    AbstractScript,
+    CommandLineError,
+    DeleteFailedScript,
+    DeleteStatesScript,
+    FetchScript,
+    InferenceScript,
+    PreprocessScript,
+)
+from autrainer.core.scripts.cli import CLI as AutrainerCLI
+
+import aucurriculum
+from aucurriculum.core.scripts import (
+    CreateCurriculumScript,
+    CurriculumScript,
+    GroupCurriculumScript,
+    ListCurriculumScript,
+    PostprocessCurriculumScript,
+    ShowCurriculumScript,
+    TrainCurriculumScript,
+)
+
+
+class CLI(AutrainerCLI):
+    def __init__(self) -> None:
+        super().__init__()
+        self._update_metadata()
+
+    def _init_scripts(self) -> List[AbstractScript]:
+        return [
+            # configuration management
+            CreateCurriculumScript(),
+            ListCurriculumScript(),
+            ShowCurriculumScript(),
+            # preprocessing
+            FetchScript(),
+            PreprocessScript(),
+            # training
+            TrainCurriculumScript(),
+            # inference
+            InferenceScript(),
+            # postprocessing
+            PostprocessCurriculumScript(),
+            DeleteFailedScript(),
+            DeleteStatesScript(),
+            GroupCurriculumScript(),
+            # curriculum learning
+            CurriculumScript(),
+        ]
+
+    def _update_metadata(self) -> None:
+        self.parser.prog = "aucurriculum"
+        self.parser.description = (
+            "A Curriculum Learning Toolkit for Deep Learning Tasks "
+            "built on top of autrainer."
+        )
+        for action in self.parser._actions:
+            if isinstance(action, argparse._VersionAction):
+                action.version = f"%(prog)s {aucurriculum.__version__}"
+
+
+def main() -> None:
+    cli = CLI()
+    try:
+        cli.main()
+    except CommandLineError as e:
+        e.handle()
+
+
+def get_parser() -> argparse.ArgumentParser:
+    cli = CLI()  # pragma: no cover
+    return cli.parser  # pragma: no cover
+
+
+if __name__ == "__main__":
+    main()