junifer 0.0.7.dev105__tar.gz → 0.0.7.dev121__tar.gz

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: junifer
-Version: 0.0.7.dev105
+Version: 0.0.7.dev121
 Summary: JUelich NeuroImaging FEature extractoR
 Author-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
 Maintainer-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>

junifer-0.0.7.dev121/docs/changes/newsfragments/429.change

@@ -0,0 +1 @@
+:func:`.read_transform` has a new parameter ``nan_policy`` to handle NaN values when transforming feature data by `Fede Raimondo`_

junifer-0.0.7.dev121/docs/changes/newsfragments/429.enh

@@ -0,0 +1 @@
+Implement a NaN policy for :func:`.read_transform` by `Fede Raimondo`_

junifer-0.0.7.dev121/docs/changes/newsfragments/452.doc

@@ -0,0 +1 @@
+Add documentation on :class:`.ConfigManager`, dumping data object and extending data dump assets by `Synchon Mandal`_

junifer-0.0.7.dev121/docs/changes/newsfragments/452.feature

@@ -0,0 +1 @@
+Allow pipeline data object to be dumped by introducing :class:`.DataObjectDumper` and available dumping / loading assets to be extended by introducing :func:`.register_data_dump_asset` decorator by `Synchon Mandal`_

junifer-0.0.7.dev121/docs/extending/data_dump_asset.rst

@@ -0,0 +1,46 @@
+.. include:: ../links.inc
+
+.. _adding_data_dump_assets:
+
+Adding Data Dump Assets
+=======================
+
+``junifer`` supports dumping ``nibabel.Nifti1Image`` and ``pandas.DataFrame`` which should cover most fMRI use cases. But, in case you work with other modalities like EEG, you can register your own data dump asset.
+
+How to add a data dump asset
+----------------------------
+
+This example shows how to create a data dump asset for ``mne.io.Raw``.
+
+#. Check :ref:`extending junifer <extending_extension>` on how to create a
+   *junifer extension* if you have not done so.
+#. Create the data registry in the *extension script* like so:
+
+   .. code-block:: python
+
+    from pathlib import Path
+
+    from junifer.api.decorators import register_data_dump_asset
+    from junifer.pipeline import BaseDataDumpAsset
+
+    import mne
+
+    @register_data_dump_asset([mne.io.Raw], [".fif", ".fif.gz"])
+    class RawAsset(BaseDataDumpAsset):
+        """Class for ``mne.io.Raw`` dumper."""
+
+        def dump(self) -> None:
+            self.data.save(self.path_without_ext.with_suffix(".raw.fif.gz"))
+
+        @classmethod
+        def load(cls: "RawAsset", path: Path) -> mne.io.Raw:
+            return mne.io.Raw(path)
+
+   * :func:`.register_data_dump_asset` registers a class. The first argument is
+     a list of types that the class is responsible for saving and the second
+     argument is a list of file extensions that the class is responsible for
+     loading.
+   * Inheriting from ``junifer.pipeline.BaseDumpAsset`` takes care of the class
+     acting as a data dump asset.
+   * Method ``junifer.pipeline.BaseDumpAsset.dump`` and class method
+     ``junifer.pipeline.BaseDumpAsset.load`` need to be implemented.

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/docs/extending/index.rst

@@ -33,3 +33,4 @@ DataGrabbers, Preprocessors, Markers, etc.,  following the *junifer* way.
    plugins
    data_registries
    data_types
+   data_dump_asset

junifer-0.0.7.dev121/docs/using/configuring.rst

@@ -0,0 +1,65 @@
+.. include:: ../links.inc
+
+.. _configuring:
+
+Configuring Pipeline Behaviour
+==============================
+
+It is also possible to configure some internal :ref:`pipeline <pipeline>` behaviour via :obj:`.ConfigManager`.
+It can be done either via the command-line interface (CLI) or the application programming
+interface (API).
+
+To use via the CLI, one would do:
+
+.. code-block:: bash
+
+   <CONFIG-KEY>=<CONFIG-VAL> junifer run ...
+
+and via the API like so:
+
+.. code-block:: python
+
+    from junifer.utils import config
+
+    # Add config
+    config.set(key="<config-key>", val=<config-val>)
+
+    # do feature extraction
+    ...
+
+    # Remove config
+    config.delete("<config-key>")
+
+.. _available_configurations:
+
+Available Configurations
+------------------------
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - CLI Key
+     - API Key
+     - Value
+     - Description
+   * - ``JUNIFER_DATA_LOCATION``
+     - ``data.location``
+     - str
+     - Alternative location for ``junifer-data``
+   * - ``JUNIFER_DATAGRABBER_SKIPIDCHECK``
+     - ``datagrabber.skipidcheck``
+     - bool
+     - Skip DataLad-based DataGrabber's ID check
+   * - ``JUNIFER_DATAGRABBER_SKIPDIRTYCHECK``
+     - ``datagrabber.skipdirtycheck``
+     - bool
+     - Skip Git "dirty" check for a DataLad dataset clone of a DataGrabber
+   * - ``JUNIFER_PREPROCESSING_DUMP_LOCATION``
+     - ``preprocessing.dump.location``
+     - str
+     - Dump location of pre-processed data for debugging purposes
+   * - ``JUNIFER_PREPROCESSING_DUMP_GRANULARITY``
+     - ``preprocessing.dump.granularity``
+     - "full" or "final"
+     - Dump all pre-processing steps or just the final pre-processed data

junifer-0.0.7.dev121/docs/using/dumping.rst

@@ -0,0 +1,18 @@
+.. include:: ../links.inc
+
+.. _dumping:
+
+Dumping Pipeline Data
+=====================
+
+It is usually not required to check the :ref:`data object <data_object>` but
+for debugging purposes one can do that. Since, there is no direct way to interact
+with it, ``junifer`` provides a way to dump the object before, in between and / or after
+pre-processing. It is controlled by the ``JUNIFER_PREPROCESSING_DUMP_LOCATION`` and
+``JUNIFER_PREPROCESSING_DUMP_GRANULARITY`` :ref:`configuration options <available_configurations>`.
+
+``junifer`` dumps the ``"data"`` attribute of the data object as proper files in their respective formats,
+for example, ``nibabel.Nifti1Image`` gets dumped as a ``.nii.gz`` file. As of now, ``junifer`` can dump
+``nibabel.Nifti1Image`` and ``pandas.DataFrame`` (confound files) file formats. In case you
+:ref:`add custom data types <adding_data_types>` which support different file formats, you can create and
+register a :ref:`custom dumper <adding_data_dump_assets>`.

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/docs/using/index.rst

@@ -18,6 +18,8 @@ to interact with HPC and HTC systems.
    codeless
    running
    queueing
+   configuring
+   dumping
 
 
 .. _using_components:

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/junifer/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.0.7.dev105'
-__version_tuple__ = version_tuple = (0, 0, 7, 'dev105')
+__version__ = version = '0.0.7.dev121'
+__version_tuple__ = version_tuple = (0, 0, 7, 'dev121')

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/junifer/api/decorators.py

@@ -6,8 +6,13 @@
 # License: AGPL
 
 from ..data import DataDispatcher
-from ..pipeline import PipelineComponentRegistry
+from ..pipeline import (
+    AssetDumperDispatcher,
+    AssetLoaderDispatcher,
+    PipelineComponentRegistry,
+)
 from ..typing import (
+    DataDumpAssetLike,
     DataGrabberLike,
     DataRegistryLike,
     MarkerLike,
@@ -17,6 +22,7 @@ from ..typing import (
 
 
 __all__ = [
+    "register_data_dump_asset",
     "register_data_registry",
     "register_datagrabber",
     "register_datareader",
@@ -184,3 +190,49 @@ def register_data_registry(name: str) -> DataRegistryLike:
         return klass
 
     return decorator
+
+
+def register_data_dump_asset(
+    types: list[type], exts: list[str]
+) -> DataDumpAssetLike:
+    """Asset registration decorator.
+
+    Registers the data dump asset for ``types`` with ``exts``.
+
+    Parameters
+    ----------
+    types : list of class
+        The classes to dump.
+    exts : list of str
+        The extensions to load.
+
+    Returns
+    -------
+    class
+        The unmodified input class.
+
+    """
+
+    def decorator(klass: DataDumpAssetLike) -> DataDumpAssetLike:
+        """Actual decorator.
+
+        Parameters
+        ----------
+        klass : class
+            The class of the data dump asset to register.
+
+        Returns
+        -------
+        class
+            The unmodified input class.
+
+        """
+        # Add asset dumper
+        for t in types:
+            AssetDumperDispatcher()[t] = klass
+        # Add asset loader
+        for e in exts:
+            AssetLoaderDispatcher()[e] = klass
+        return klass
+
+    return decorator

junifer-0.0.7.dev121/junifer/api/tests/test_decorators.py

@@ -0,0 +1,78 @@
+"""Provide tests for public decorators."""
+
+# Authors: Synchon Mandal <s.mandal@fz-juelich.de>
+# License: AGPL
+
+import pickle
+
+from junifer.api.decorators import (
+    register_data_dump_asset,
+    register_data_registry,
+)
+from junifer.data import BasePipelineDataRegistry, DataDispatcher
+from junifer.pipeline import (
+    AssetDumperDispatcher,
+    AssetLoaderDispatcher,
+    BaseDataDumpAsset,
+)
+
+
+def test_register_data_registry() -> None:
+    """Test data registry registration."""
+
+    @register_data_registry("dumb")
+    class DumDum(BasePipelineDataRegistry):
+        def __init__(self):
+            super().__init__()
+
+        def register(self):
+            pass
+
+        def deregister(self):
+            pass
+
+        def load(self):
+            pass
+
+        def get(self):
+            pass
+
+    assert "dumb" in DataDispatcher()
+    _ = DataDispatcher().pop("dumb")
+    assert "dumb" not in DataDispatcher()
+
+
+def test_register_data_dump_asset() -> None:
+    """Test data dump asset registration."""
+
+    class Int(int): ...
+
+    class Float(float): ...
+
+    @register_data_dump_asset([Int, Float], [".int", ".float"])
+    class DumAsset(BaseDataDumpAsset):
+        def dump(self):
+            suffix = ""
+            if isinstance(self.data, Int):
+                suffix = ".int"
+            else:
+                suffix = ".float"
+            pickle.dump(self.data, self.path_without_ext.with_suffix(suffix))
+
+        @classmethod
+        def load(cls, path):
+            return pickle.load(path)
+
+    assert Int in AssetDumperDispatcher()
+    assert Float in AssetDumperDispatcher()
+    _ = AssetDumperDispatcher().pop(Int)
+    _ = AssetDumperDispatcher().pop(Float)
+    assert Int not in AssetDumperDispatcher()
+    assert Float not in AssetDumperDispatcher()
+
+    assert ".int" in AssetLoaderDispatcher()
+    assert ".float" in AssetLoaderDispatcher()
+    _ = AssetLoaderDispatcher().pop(".int")
+    _ = AssetLoaderDispatcher().pop(".float")
+    assert ".int" not in AssetLoaderDispatcher()
+    assert ".float" not in AssetLoaderDispatcher()

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/junifer/onthefly/read_transform.py

@@ -5,6 +5,7 @@
 
 from typing import Optional
 
+import numpy as np
 import pandas as pd
 
 from ..typing import StorageLike
@@ -19,6 +20,7 @@ def read_transform(
     transform: str,
     feature_name: Optional[str] = None,
     feature_md5: Optional[str] = None,
+    nan_policy: Optional[str] = "bypass",
     transform_args: Optional[tuple] = None,
     transform_kw_args: Optional[dict] = None,
 ) -> pd.DataFrame:
@@ -35,6 +37,16 @@ def read_transform(
         Name of the feature to read (default None).
     feature_md5 : str, optional
         MD5 hash of the feature to read (default None).
+    nan_policy : str, optional
+        The policy to handle NaN values (default "ignore").
+        Options are:
+
+        * "bypass": Do nothing and pass NaN values to the transform function.
+        * "drop_element": Drop (skip) elements with NaN values.
+        * "drop_rows": Drop (skip) rows with NaN values.
+        * "drop_columns": Drop (skip) columns with NaN values.
+        * "drop_symmetric": Drop (skip) symmetric pairs with NaN values.
+
     transform_args : tuple, optional
         The positional arguments for the callable of ``transform``
         (default None).
@@ -47,6 +59,18 @@ def read_transform(
     pandas.DataFrame
         The transformed feature as a dataframe.
 
+    Raises
+    ------
+    ValueError
+        If ``nan_policy`` is invalid or
+        if *package* is invalid.
+    RuntimeError
+        If *package* is ``bctpy`` and stored data kind is not ``matrix``.
+    ImportError
+        If ``bctpy`` cannot be imported.
+    AttributeError
+        If *function* to be invoked in invalid.
+
     Notes
     -----
     This function has been only tested for:
@@ -63,6 +87,18 @@ def read_transform(
     transform_args = transform_args or ()
     transform_kw_args = transform_kw_args or {}
 
+    if nan_policy not in [
+        "bypass",
+        "drop_element",
+        "drop_rows",
+        "drop_columns",
+        "drop_symmetric",
+    ]:
+        raise_error(
+            f"Unknown nan_policy: {nan_policy}",
+            klass=ValueError,
+        )
+
     # Read storage
     stored_data = storage.read(
         feature_name=feature_name, feature_md5=feature_md5
@@ -106,22 +142,52 @@ def read_transform(
         except AttributeError as err:
             raise_error(msg=str(err), klass=AttributeError)
 
-        # Apply function and store subject-wise
+        # Apply function and store element-wise
         output_list = []
+        element_list = []
         logger.debug(
             f"Computing '{package}.{func_str}' for feature "
             f"{feature_name or feature_md5} ..."
         )
-        for subject in range(stored_data["data"].shape[2]):
+        for i_element, element in enumerate(stored_data["element"]):
+            t_data = stored_data["data"][:, :, i_element]
+            has_nan = np.isnan(np.min(t_data))
+            if nan_policy == "drop_element" and has_nan:
+                logger.debug(
+                    f"Skipping element {element} due to NaN values ..."
+                )
+                continue
+            elif nan_policy == "drop_rows" and has_nan:
+                logger.debug(
+                    f"Skipping rows with NaN values in element {element} ..."
+                )
+                t_data = t_data[~np.isnan(t_data).any(axis=1)]
+            elif nan_policy == "drop_columns" and has_nan:
+                logger.debug(
+                    f"Skipping columns with NaN values in element {element} "
+                    "..."
+                )
+                t_data = t_data[:, ~np.isnan(t_data).any(axis=0)]
+            elif nan_policy == "drop_symmetric":
+                logger.debug(
+                    f"Skipping pairs of rows/columns with NaN values in "
+                    f"element {element}..."
+                )
+                good_rows = ~np.isnan(t_data).any(axis=1)
+                good_columns = ~np.isnan(t_data).any(axis=0)
+                good_idx = np.logical_and(good_rows, good_columns)
+                t_data = t_data[good_idx][:, good_idx]
+
             output = func(
-                stored_data["data"][:, :, subject],
+                t_data,
                 *transform_args,
                 **transform_kw_args,
             )
             output_list.append(output)
+            element_list.append(element)
 
         # Create dataframe for index
-        idx_df = pd.DataFrame(data=stored_data["element"])
+        idx_df = pd.DataFrame(data=element_list)
         # Create multiindex from dataframe
         logger.debug(
             "Generating pandas.MultiIndex for feature "

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/junifer/onthefly/tests/test_read_transform.py

@@ -64,6 +64,36 @@ def matrix_storage(tmp_path: Path) -> HDF5FeatureStorage:
     return storage
 
 
+@pytest.fixture
+def matrix_storage_with_nan(tmp_path: Path) -> HDF5FeatureStorage:
+    """Return a HDF5FeatureStorage with matrix data.
+
+    Parameters
+    ----------
+    tmp_path : pathlib.Path
+        The path to the test directory.
+
+    """
+    storage = HDF5FeatureStorage(tmp_path / "matrix_store_nan.hdf5")
+    data = np.arange(36).reshape(3, 3, 4).astype(float)
+    data[1, 1, 2] = np.nan
+    data[1, 2, 2] = np.nan
+    for i in range(4):
+        storage.store(
+            kind="matrix",
+            meta={
+                "element": {"subject": f"test{i + 1}"},
+                "dependencies": [],
+                "marker": {"name": "matrix"},
+                "type": "BOLD",
+            },
+            data=data[:, :, i],
+            col_names=["f1", "f2", "f3"],
+            row_names=["g1", "g2", "g3"],
+        )
+    return storage
+
+
 def test_incorrect_package(matrix_storage: HDF5FeatureStorage) -> None:
     """Test error check for incorrect package name.
 
@@ -176,3 +206,57 @@ def test_bctpy_function(
         )
         assert "Computing" in caplog.text
         assert "Generating" in caplog.text
+
+
+@pytest.mark.parametrize(
+    "nan_policy, error_msg",
+    [
+        ("drop_element", None),
+        ("drop_rows", "square"),
+        ("drop_columns", "square"),
+        ("drop_symmetric", None),
+        ("bypass", "NaNs"),
+        ("wrong", "Unknown"),
+    ],
+)
+def test_bctpy_nans(
+    matrix_storage_with_nan: HDF5FeatureStorage,
+    caplog: pytest.LogCaptureFixture,
+    nan_policy: str,
+    error_msg: str,
+) -> None:
+    """Test working function of bctpy.
+
+    Parameters
+    ----------
+    matrix_storage_with_nan : HDF5FeatureStorage
+        The HDF5FeatureStorage with matrix data, as fixture.
+    caplog : pytest.LogCaptureFixture
+        The pytest.LogCaptureFixture object.
+    nan_policy : str
+        The NAN policy to test.
+    error_msg : str
+        The expected error message snippet. If None, no error should be raised.
+
+    """
+    # Skip test if import fails
+    pytest.importorskip("bct")
+
+    with caplog.at_level(logging.DEBUG):
+        if error_msg is None:
+            read_transform(
+                storage=matrix_storage_with_nan,  # type: ignore
+                feature_name="BOLD_matrix",
+                transform="bctpy_eigenvector_centrality_und",
+                nan_policy=nan_policy,
+            )
+            assert "Computing" in caplog.text
+            assert "Generating" in caplog.text
+        else:
+            with pytest.raises(ValueError, match=error_msg):
+                read_transform(
+                    storage=matrix_storage_with_nan,  # type: ignore
+                    feature_name="BOLD_matrix",
+                    transform="bctpy_eigenvector_centrality_und",
+                    nan_policy=nan_policy,
+                )

{junifer-0.0.7.dev105 → junifer-0.0.7.dev121}/junifer/pipeline/__init__.pyi

@@ -1,4 +1,8 @@
 __all__ = [
+    "AssetDumperDispatcher",
+    "AssetLoaderDispatcher",
+    "BaseDataDumpAsset",
+    "DataObjectDumper",
     "PipelineComponentRegistry",
     "PipelineStepMixin",
     "UpdateMetaMixin",
@@ -6,6 +10,12 @@ __all__ = [
     "MarkerCollection",
 ]
 
+from ._data_object_dumper import (
+    AssetDumperDispatcher,
+    AssetLoaderDispatcher,
+    BaseDataDumpAsset,
+    DataObjectDumper,
+)
 from .pipeline_component_registry import PipelineComponentRegistry
 from .pipeline_step_mixin import PipelineStepMixin
 from .update_meta_mixin import UpdateMetaMixin