junifer 0.0.7.dev111__tar.gz → 0.0.7.dev123__tar.gz

{junifer-0.0.7.dev111 → junifer-0.0.7.dev123}/.github/workflows/docker-ci-publish.yml

@@ -59,7 +59,7 @@ jobs:
       # https://github.com/docker/login-action
       - name: Log into registry ${{ env.REGISTRY }}
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
+        uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
@@ -69,7 +69,7 @@ jobs:
       # https://github.com/docker/metadata-action
       - name: Extract Docker metadata
         id: meta
-        uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
+        uses: docker/metadata-action@c1e51972afc2121e065aed6d45c65596fe445f3f # v5.8.0
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
 

{junifer-0.0.7.dev111 → junifer-0.0.7.dev123}/.github/workflows/docker-docs-publish.yml

@@ -56,7 +56,7 @@ jobs:
       # https://github.com/docker/login-action
       - name: Log into registry ${{ env.REGISTRY }}
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
+        uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
@@ -66,7 +66,7 @@ jobs:
       # https://github.com/docker/metadata-action
       - name: Extract Docker metadata
         id: meta
-        uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
+        uses: docker/metadata-action@c1e51972afc2121e065aed6d45c65596fe445f3f # v5.8.0
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
 

{junifer-0.0.7.dev111 → junifer-0.0.7.dev123}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: junifer
-Version: 0.0.7.dev111
+Version: 0.0.7.dev123
 Summary: JUelich NeuroImaging FEature extractoR
 Author-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
 Maintainer-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>

junifer-0.0.7.dev123/docs/changes/newsfragments/452.doc

@@ -0,0 +1 @@
+Add documentation on :class:`.ConfigManager`, dumping data object and extending data dump assets by `Synchon Mandal`_

junifer-0.0.7.dev123/docs/changes/newsfragments/452.feature

@@ -0,0 +1 @@
+Allow pipeline data object to be dumped by introducing :class:`.DataObjectDumper` and available dumping / loading assets to be extended by introducing :func:`.register_data_dump_asset` decorator by `Synchon Mandal`_

junifer-0.0.7.dev123/docs/extending/data_dump_asset.rst

@@ -0,0 +1,46 @@
+.. include:: ../links.inc
+
+.. _adding_data_dump_assets:
+
+Adding Data Dump Assets
+=======================
+
+``junifer`` supports dumping ``nibabel.Nifti1Image`` and ``pandas.DataFrame`` which should cover most fMRI use cases. But, in case you work with other modalities like EEG, you can register your own data dump asset.
+
+How to add a data dump asset
+----------------------------
+
+This example shows how to create a data dump asset for ``mne.io.Raw``.
+
+#. Check :ref:`extending junifer <extending_extension>` on how to create a
+   *junifer extension* if you have not done so.
+#. Create the data registry in the *extension script* like so:
+
+   .. code-block:: python
+
+    from pathlib import Path
+
+    from junifer.api.decorators import register_data_dump_asset
+    from junifer.pipeline import BaseDataDumpAsset
+
+    import mne
+
+    @register_data_dump_asset([mne.io.Raw], [".fif", ".fif.gz"])
+    class RawAsset(BaseDataDumpAsset):
+        """Class for ``mne.io.Raw`` dumper."""
+
+        def dump(self) -> None:
+            self.data.save(self.path_without_ext.with_suffix(".raw.fif.gz"))
+
+        @classmethod
+        def load(cls: "RawAsset", path: Path) -> mne.io.Raw:
+            return mne.io.Raw(path)
+
+   * :func:`.register_data_dump_asset` registers a class. The first argument is
+     a list of types that the class is responsible for saving and the second
+     argument is a list of file extensions that the class is responsible for
+     loading.
+   * Inheriting from ``junifer.pipeline.BaseDumpAsset`` takes care of the class
+     acting as a data dump asset.
+   * Method ``junifer.pipeline.BaseDumpAsset.dump`` and class method
+     ``junifer.pipeline.BaseDumpAsset.load`` need to be implemented.

{junifer-0.0.7.dev111 → junifer-0.0.7.dev123}/docs/extending/index.rst

@@ -33,3 +33,4 @@ DataGrabbers, Preprocessors, Markers, etc.,  following the *junifer* way.
    plugins
    data_registries
    data_types
+   data_dump_asset

junifer-0.0.7.dev123/docs/using/configuring.rst

@@ -0,0 +1,65 @@
+.. include:: ../links.inc
+
+.. _configuring:
+
+Configuring Pipeline Behaviour
+==============================
+
+It is also possible to configure some internal :ref:`pipeline <pipeline>` behaviour via :obj:`.ConfigManager`.
+It can be done either via the command-line interface (CLI) or the application programming
+interface (API).
+
+To use via the CLI, one would do:
+
+.. code-block:: bash
+
+   <CONFIG-KEY>=<CONFIG-VAL> junifer run ...
+
+and via the API like so:
+
+.. code-block:: python
+
+    from junifer.utils import config
+
+    # Add config
+    config.set(key="<config-key>", val=<config-val>)
+
+    # do feature extraction
+    ...
+
+    # Remove config
+    config.delete("<config-key>")
+
+.. _available_configurations:
+
+Available Configurations
+------------------------
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - CLI Key
+     - API Key
+     - Value
+     - Description
+   * - ``JUNIFER_DATA_LOCATION``
+     - ``data.location``
+     - str
+     - Alternative location for ``junifer-data``
+   * - ``JUNIFER_DATAGRABBER_SKIPIDCHECK``
+     - ``datagrabber.skipidcheck``
+     - bool
+     - Skip DataLad-based DataGrabber's ID check
+   * - ``JUNIFER_DATAGRABBER_SKIPDIRTYCHECK``
+     - ``datagrabber.skipdirtycheck``
+     - bool
+     - Skip Git "dirty" check for a DataLad dataset clone of a DataGrabber
+   * - ``JUNIFER_PREPROCESSING_DUMP_LOCATION``
+     - ``preprocessing.dump.location``
+     - str
+     - Dump location of pre-processed data for debugging purposes
+   * - ``JUNIFER_PREPROCESSING_DUMP_GRANULARITY``
+     - ``preprocessing.dump.granularity``
+     - "full" or "final"
+     - Dump all pre-processing steps or just the final pre-processed data

junifer-0.0.7.dev123/docs/using/dumping.rst

@@ -0,0 +1,18 @@
+.. include:: ../links.inc
+
+.. _dumping:
+
+Dumping Pipeline Data
+=====================
+
+It is usually not required to check the :ref:`data object <data_object>` but
+for debugging purposes one can do that. Since, there is no direct way to interact
+with it, ``junifer`` provides a way to dump the object before, in between and / or after
+pre-processing. It is controlled by the ``JUNIFER_PREPROCESSING_DUMP_LOCATION`` and
+``JUNIFER_PREPROCESSING_DUMP_GRANULARITY`` :ref:`configuration options <available_configurations>`.
+
+``junifer`` dumps the ``"data"`` attribute of the data object as proper files in their respective formats,
+for example, ``nibabel.Nifti1Image`` gets dumped as a ``.nii.gz`` file. As of now, ``junifer`` can dump
+``nibabel.Nifti1Image`` and ``pandas.DataFrame`` (confound files) file formats. In case you
+:ref:`add custom data types <adding_data_types>` which support different file formats, you can create and
+register a :ref:`custom dumper <adding_data_dump_assets>`.

{junifer-0.0.7.dev111 → junifer-0.0.7.dev123}/docs/using/index.rst

@@ -18,6 +18,8 @@ to interact with HPC and HTC systems.
    codeless
    running
    queueing
+   configuring
+   dumping
 
 
 .. _using_components:

junifer-0.0.7.dev123/junifer/_version.py

@@ -0,0 +1,34 @@
+# file generated by setuptools-scm
+# don't change, don't track in version control
+
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
+
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Tuple
+    from typing import Union
+
+    VERSION_TUPLE = Tuple[Union[int, str], ...]
+    COMMIT_ID = Union[str, None]
+else:
+    VERSION_TUPLE = object
+    COMMIT_ID = object
+
+version: str
+__version__: str
+__version_tuple__: VERSION_TUPLE
+version_tuple: VERSION_TUPLE
+commit_id: COMMIT_ID
+__commit_id__: COMMIT_ID
+
+__version__ = version = '0.0.7.dev123'
+__version_tuple__ = version_tuple = (0, 0, 7, 'dev123')
+
+__commit_id__ = commit_id = 'g77932fce8'

{junifer-0.0.7.dev111 → junifer-0.0.7.dev123}/junifer/api/decorators.py

@@ -6,8 +6,13 @@
 # License: AGPL
 
 from ..data import DataDispatcher
-from ..pipeline import PipelineComponentRegistry
+from ..pipeline import (
+    AssetDumperDispatcher,
+    AssetLoaderDispatcher,
+    PipelineComponentRegistry,
+)
 from ..typing import (
+    DataDumpAssetLike,
     DataGrabberLike,
     DataRegistryLike,
     MarkerLike,
@@ -17,6 +22,7 @@ from ..typing import (
 
 
 __all__ = [
+    "register_data_dump_asset",
     "register_data_registry",
     "register_datagrabber",
     "register_datareader",
@@ -184,3 +190,49 @@ def register_data_registry(name: str) -> DataRegistryLike:
         return klass
 
     return decorator
+
+
+def register_data_dump_asset(
+    types: list[type], exts: list[str]
+) -> DataDumpAssetLike:
+    """Asset registration decorator.
+
+    Registers the data dump asset for ``types`` with ``exts``.
+
+    Parameters
+    ----------
+    types : list of class
+        The classes to dump.
+    exts : list of str
+        The extensions to load.
+
+    Returns
+    -------
+    class
+        The unmodified input class.
+
+    """
+
+    def decorator(klass: DataDumpAssetLike) -> DataDumpAssetLike:
+        """Actual decorator.
+
+        Parameters
+        ----------
+        klass : class
+            The class of the data dump asset to register.
+
+        Returns
+        -------
+        class
+            The unmodified input class.
+
+        """
+        # Add asset dumper
+        for t in types:
+            AssetDumperDispatcher()[t] = klass
+        # Add asset loader
+        for e in exts:
+            AssetLoaderDispatcher()[e] = klass
+        return klass
+
+    return decorator

junifer-0.0.7.dev123/junifer/api/tests/test_decorators.py

@@ -0,0 +1,78 @@
+"""Provide tests for public decorators."""
+
+# Authors: Synchon Mandal <s.mandal@fz-juelich.de>
+# License: AGPL
+
+import pickle
+
+from junifer.api.decorators import (
+    register_data_dump_asset,
+    register_data_registry,
+)
+from junifer.data import BasePipelineDataRegistry, DataDispatcher
+from junifer.pipeline import (
+    AssetDumperDispatcher,
+    AssetLoaderDispatcher,
+    BaseDataDumpAsset,
+)
+
+
+def test_register_data_registry() -> None:
+    """Test data registry registration."""
+
+    @register_data_registry("dumb")
+    class DumDum(BasePipelineDataRegistry):
+        def __init__(self):
+            super().__init__()
+
+        def register(self):
+            pass
+
+        def deregister(self):
+            pass
+
+        def load(self):
+            pass
+
+        def get(self):
+            pass
+
+    assert "dumb" in DataDispatcher()
+    _ = DataDispatcher().pop("dumb")
+    assert "dumb" not in DataDispatcher()
+
+
+def test_register_data_dump_asset() -> None:
+    """Test data dump asset registration."""
+
+    class Int(int): ...
+
+    class Float(float): ...
+
+    @register_data_dump_asset([Int, Float], [".int", ".float"])
+    class DumAsset(BaseDataDumpAsset):
+        def dump(self):
+            suffix = ""
+            if isinstance(self.data, Int):
+                suffix = ".int"
+            else:
+                suffix = ".float"
+            pickle.dump(self.data, self.path_without_ext.with_suffix(suffix))
+
+        @classmethod
+        def load(cls, path):
+            return pickle.load(path)
+
+    assert Int in AssetDumperDispatcher()
+    assert Float in AssetDumperDispatcher()
+    _ = AssetDumperDispatcher().pop(Int)
+    _ = AssetDumperDispatcher().pop(Float)
+    assert Int not in AssetDumperDispatcher()
+    assert Float not in AssetDumperDispatcher()
+
+    assert ".int" in AssetLoaderDispatcher()
+    assert ".float" in AssetLoaderDispatcher()
+    _ = AssetLoaderDispatcher().pop(".int")
+    _ = AssetLoaderDispatcher().pop(".float")
+    assert ".int" not in AssetLoaderDispatcher()
+    assert ".float" not in AssetLoaderDispatcher()

{junifer-0.0.7.dev111 → junifer-0.0.7.dev123}/junifer/pipeline/__init__.pyi

@@ -1,4 +1,8 @@
 __all__ = [
+    "AssetDumperDispatcher",
+    "AssetLoaderDispatcher",
+    "BaseDataDumpAsset",
+    "DataObjectDumper",
     "PipelineComponentRegistry",
     "PipelineStepMixin",
     "UpdateMetaMixin",
@@ -6,6 +10,12 @@ __all__ = [
     "MarkerCollection",
 ]
 
+from ._data_object_dumper import (
+    AssetDumperDispatcher,
+    AssetLoaderDispatcher,
+    BaseDataDumpAsset,
+    DataObjectDumper,
+)
 from .pipeline_component_registry import PipelineComponentRegistry
 from .pipeline_step_mixin import PipelineStepMixin
 from .update_meta_mixin import UpdateMetaMixin