junifer 0.0.5.dev219__tar.gz → 0.0.5.dev242__tar.gz

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: junifer
-Version: 0.0.5.dev219
+Version: 0.0.5.dev242
 Summary: JUelich NeuroImaging FEature extractoR
 Author-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
 Maintainer-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/docs/whats_new.rst

@@ -8,6 +8,127 @@ What's new
 
 .. towncrier release notes start
 
+Junifer 0.0.5 (2024-07-22)
+--------------------------
+
+Bugfixes
+^^^^^^^^
+
+- Remove extra dimension from parcellations warped via ANTs to other template
+  spaces by `Synchon Mandal`_ (:gh:`324`)
+- Remove extra dimension from parcellations warped via ANTs when merging
+  parcellations by `Synchon Mandal`_ (:gh:`331`)
+- Fix ``junifer reset`` to properly delete storage directory and handle
+  ``junifer_jobs`` deletion if not empty by `Synchon Mandal`_ (:gh:`332`)
+- Fix validation failure of multiple Preprocessors with different input data
+  types requirement by `Synchon Mandal`_ (:gh:`339`)
+- Remove extra dimension from masks warped via ANTs by `Synchon Mandal`_
+  (:gh:`340`)
+
+
+API Changes
+^^^^^^^^^^^
+
+- For :class:`.CrossParcellationFC`, ``aggregation_method`` and
+  ``correlation_method`` have been renamed to ``agg_method`` and
+  ``corr_method`` respectively and ``agg_method_params`` has been added; for
+  ``FunctionalConnectivityBase``, :class:`.FunctionalConnectivityParcels`,
+  :class:`.FunctionalConnectivitySpheres`, :class:`.EdgeCentricFCParcels` and
+  :class:`.EdgeCentricFCSpheres`, ``cor_method`` and ``cor_method_params`` have
+  been renamed to ``conn_method`` and ``conn_method_params`` by `Synchon
+  Mandal`_ (:gh:`348`)
+- ``fractional`` parameter for ``ALFFBase``, :class:`.ALFFParcels` and
+  :class:`.ALFFSpheres` have been removed in favour of returning both ALFF and
+  fALFF by `Synchon Mandal`_ (:gh:`349`)
+- Add ``partial_pattern_ok`` argument to :class:`.PatternDataGrabber` to not
+  raise error on missing mandatory key checks for data types by `Synchon
+  Mandal`_ (:gh:`351`)
+
+
+Improved Documentation
+^^^^^^^^^^^^^^^^^^^^^^
+
+- Improve sub-package level docstrings to better reflect their purposes by
+  `Synchon Mandal`_ (:gh:`115`)
+- Update "Queueing Jobs (HPC, HTC)" section by `Synchon Mandal`_ (:gh:`327`)
+- Add missing DataGrabber ``types`` options in respective docstrings of
+  :class:`.DataladAOMICID1000`, :class:`.DataladAOMICPIOP1`,
+  :class:`.DataladAOMICPIOP2` and :class:`.DMCC13Benchmark` by `Synchon
+  Mandal`_ (:gh:`330`)
+
+
+Enhancements
+^^^^^^^^^^^^
+
+- Add test to be sure that :class:`.JuniferNiftiSpheresMasker` with mean
+  aggregation function behaves exactly as
+  :class:`nilearn.maskers.NiftiSpheresMasker` by `Synchon Mandal`_ (:gh:`136`)
+- Refactor DataGrabber ``patterns`` to make helper types like ``*_mask`` as
+  "nested types" of the actual data type by `Synchon Mandal`_ (:gh:`341`)
+- Adapt :class:`.DataladAOMICID1000`, :class:`.DataladAOMICPIOP1` and
+  :class:`.DataladAOMICPIOP2` to support ``FreeSurfer`` data type by `Synchon
+  Mandal`_ (:gh:`346`)
+- ``FunctionalConnectivity``-family Markers now use
+  :class:`sklearn.covariance.EmpiricalCovariance` as the default covariance
+  estimator and ``correlation`` as the default connecivity matrix kind by
+  `Synchon Mandal`_ (:gh:`348`)
+- Enable Markers to output multiple features by `Synchon Mandal`_ (:gh:`349`)
+- Add support for ``UKB_15K_GM`` mask by `Synchon Mandal`_ (:gh:`350`)
+- Adapt :class:`.MultipleDataGrabber` to handle "nested types" introduced in
+  :gh:`341` by `Synchon Mandal`_ (:gh:`351`)
+
+
+Features
+^^^^^^^^
+
+- Introduce :class:`.Smoothing` for smoothing / blurring images as a
+  preprocessing step by `Synchon Mandal`_ (:gh:`161`)
+- Add support for choosing between ``bash`` and ``zsh`` shells when queueing by
+  `Synchon Mandal`_ (:gh:`273`)
+- Add ``junifer list-elements`` to list out available elements for a
+  DataGrabber based on filtering via ``--element`` by `Synchon Mandal`_
+  (:gh:`323`)
+- Introduce new storage type ``scalar_table`` and adapt
+  :class:`.HDF5FeatureStorage` to support it by `Synchon Mandal`_ (:gh:`343`)
+- Add support for `BrainPrint
+  <https://github.com/Deep-MI/BrainPrint?tab=readme-ov-file>`_ marker by
+  `Synchon Mandal`_ (:gh:`344`)
+- Allow Unix path expansion directives to be used in
+  :class:`.PatternDataGrabber` ``patterns`` by `Synchon Mandal`_ (:gh:`345`)
+- Add support for ``FreeSurfer`` data type for :class:`.PatternDataGrabber` by
+  `Synchon Mandal`_ (:gh:`346`)
+- Introduce :class:`.JuniferConnectivityMeasure` for customising functional
+  connectivity matrix kinds and measurements by `Synchon Mandal`_ (:gh:`348`)
+- Introduce :class:`.PatternValidationMixin` to simplify validation for
+  pattern-based DataGrabbers and :func:`.deep_update` for updating dictionary
+  with varying width and depth by `Synchon Mandal`_ (:gh:`351`)
+
+
+Miscellaneous
+^^^^^^^^^^^^^
+
+- Improve CI to allow external tool installation to fail gracefully and update
+  necessary dependency version and conditional checks by `Synchon Mandal`_
+  (:gh:`318`)
+- Update ``pre-commit`` dependency versions, add ``blacken-docs`` to
+  ``pre-commit``, add ``__all__`` for modules, sub-packages and package, update
+  ``ruff`` and ``pytest`` configs in ``pyproject.toml`` by `Synchon Mandal`_
+  (:gh:`337`)
+- Add support for accessing FreeSurfer via Docker wrapper along with
+  ``mri_binarize``, ``mri_pretess``, ``mri_mc`` and ``mris_convert`` by
+  `Synchon Mandal`_ (:gh:`342`)
+- Update core and docs dependencies by `Synchon Mandal`_ (:gh:`347`)
+- Integrate ``warnings`` with ``logging`` respecting filters by `Fede
+  Raimondo`_ (:gh:`351`)
+
+
+Deprecations and Removals
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Remove ``Power`` coordinates, ``fetch_icbm152_brain_gm_mask`` mask,
+  ``BOLDWarper`` Preprocessor by `Synchon Mandal`_ (:gh:`336`)
+
+
 Junifer 0.0.4 (2024-04-05)
 --------------------------
 

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/junifer/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.0.5.dev219'
-__version_tuple__ = version_tuple = (0, 0, 5, 'dev219')
+__version__ = version = '0.0.5.dev242'
+__version_tuple__ = version_tuple = (0, 0, 5, 'dev242')

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/junifer/datagrabber/__init__.py

@@ -17,6 +17,7 @@ from .hcp1200 import HCP1200, DataladHCP1200
 from .multiple import MultipleDataGrabber
 from .dmcc13_benchmark import DMCC13Benchmark
 
+from .pattern_validation_mixin import PatternValidationMixin
 
 __all__ = [
     "BaseDataGrabber",
@@ -30,4 +31,5 @@ __all__ = [
     "DataladHCP1200",
     "MultipleDataGrabber",
     "DMCC13Benchmark",
+    "PatternValidationMixin",
 ]

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/junifer/datagrabber/base.py

@@ -11,7 +11,6 @@ from typing import Dict, Iterator, List, Tuple, Union
 
 from ..pipeline import UpdateMetaMixin
 from ..utils import logger, raise_error
-from .utils import validate_types
 
 
 __all__ = ["BaseDataGrabber"]
@@ -30,16 +29,21 @@ class BaseDataGrabber(ABC, UpdateMetaMixin):
     datadir : str or pathlib.Path
         The directory where the data is / will be stored.
 
-    Attributes
-    ----------
-    datadir : pathlib.Path
-        The directory where the data is / will be stored.
+    Raises
+    ------
+    TypeError
+        If ``types`` is not a list or if the values are not string.
 
     """
 
     def __init__(self, types: List[str], datadir: Union[str, Path]) -> None:
         # Validate types
-        validate_types(types)
+        if not isinstance(types, list):
+            raise_error(msg="`types` must be a list", klass=TypeError)
+        if any(not isinstance(x, str) for x in types):
+            raise_error(
+                msg="`types` must be a list of strings", klass=TypeError
+            )
         self.types = types
 
         # Convert str to Path

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/junifer/datagrabber/hcp1200/hcp1200.py

@@ -10,8 +10,8 @@ from pathlib import Path
 from typing import Dict, List, Union
 
 from ...api.decorators import register_datagrabber
+from ...utils import raise_error
 from ..pattern import PatternDataGrabber
-from ..utils import raise_error
 
 
 __all__ = ["HCP1200"]

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/junifer/datagrabber/multiple.py

@@ -7,13 +7,15 @@
 
 from typing import Dict, List, Tuple, Union
 
-from ..utils import raise_error
+from ..api.decorators import register_datagrabber
+from ..utils import deep_update, raise_error
 from .base import BaseDataGrabber
 
 
 __all__ = ["MultipleDataGrabber"]
 
 
+@register_datagrabber
 class MultipleDataGrabber(BaseDataGrabber):
     """Concrete implementation for multi sourced data fetching.
 
@@ -27,19 +29,53 @@ class MultipleDataGrabber(BaseDataGrabber):
     **kwargs
         Keyword arguments passed to superclass.
 
+    Raises
+    ------
+    RuntimeError
+        If ``datagrabbers`` have different element keys or
+        overlapping data types or nested data types.
+
     """
 
     def __init__(self, datagrabbers: List[BaseDataGrabber], **kwargs) -> None:
         # Check datagrabbers consistency
-        # 1) same element keys
+        # Check for same element keys
         first_keys = datagrabbers[0].get_element_keys()
         for dg in datagrabbers[1:]:
             if dg.get_element_keys() != first_keys:
-                raise_error("DataGrabbers have different element keys.")
-        # 2) no overlapping types
+                raise_error(
+                    msg="DataGrabbers have different element keys",
+                    klass=RuntimeError,
+                )
+        # Check for no overlapping types (and nested data types)
         types = [x for dg in datagrabbers for x in dg.get_types()]
         if len(types) != len(set(types)):
-            raise_error("DataGrabbers have overlapping types.")
+            if all(hasattr(dg, "patterns") for dg in datagrabbers):
+                first_patterns = datagrabbers[0].patterns
+                for dg in datagrabbers[1:]:
+                    for data_type in set(types):
+                        dtype_pattern = dg.patterns.get(data_type)
+                        if dtype_pattern is None:
+                            continue
+                        # Check if first-level keys of data type are same
+                        if (
+                            dtype_pattern.keys()
+                            == first_patterns[data_type].keys()
+                        ):
+                            raise_error(
+                                msg=(
+                                    "DataGrabbers have overlapping mandatory "
+                                    "and / or optional key(s) for data type: "
+                                    f"`{data_type}`"
+                                ),
+                                klass=RuntimeError,
+                            )
+            else:
+                # Can't check further
+                raise_error(
+                    msg="DataGrabbers have overlapping types",
+                    klass=RuntimeError,
+                )
         self._datagrabbers = datagrabbers
 
     def __getitem__(self, element: Union[str, Tuple]) -> Dict:
@@ -65,7 +101,7 @@ class MultipleDataGrabber(BaseDataGrabber):
         metas = []
         for dg in self._datagrabbers:
             t_out = dg[element]
-            out.update(t_out)
+            deep_update(out, t_out)
             # Now get the meta for this datagrabber
             t_meta = {}
             dg.update_meta(t_meta, "datagrabber")

{junifer-0.0.5.dev219 → junifer-0.0.5.dev242}/junifer/datagrabber/pattern.py

@@ -15,7 +15,7 @@ import numpy as np
 from ..api.decorators import register_datagrabber
 from ..utils import logger, raise_error
 from .base import BaseDataGrabber
-from .utils import validate_patterns, validate_replacements
+from .pattern_validation_mixin import PatternValidationMixin
 
 
 __all__ = ["PatternDataGrabber"]
@@ -26,7 +26,7 @@ _CONFOUNDS_FORMATS = ("fmriprep", "adhoc")
 
 
 @register_datagrabber
-class PatternDataGrabber(BaseDataGrabber):
+class PatternDataGrabber(BaseDataGrabber, PatternValidationMixin):
     """Concrete implementation for pattern-based data fetching.
 
     Implements a DataGrabber that understands patterns to grab data.
@@ -142,6 +142,13 @@ class PatternDataGrabber(BaseDataGrabber):
         The directory where the data is / will be stored.
     confounds_format : {"fmriprep", "adhoc"} or None, optional
         The format of the confounds for the dataset (default None).
+    partial_pattern_ok : bool, optional
+        Whether to raise error if partial pattern for a data type is found.
+        This allows to bypass mandatory key check and issue a warning
+        instead of raising error. This allows one to have a DataGrabber
+        with data types without the corresponding mandatory keys and is
+        powerful when used with :class:`.MultipleDataGrabber`
+        (default True).
 
     Raises
     ------
@@ -157,17 +164,21 @@ class PatternDataGrabber(BaseDataGrabber):
         replacements: Union[List[str], str],
         datadir: Union[str, Path],
         confounds_format: Optional[str] = None,
+        partial_pattern_ok: bool = False,
     ) -> None:
-        # Validate patterns
-        validate_patterns(types=types, patterns=patterns)
-        self.patterns = patterns
-
         # Convert replacements to list if not already
         if not isinstance(replacements, list):
             replacements = [replacements]
-        # Validate replacements
-        validate_replacements(replacements=replacements, patterns=patterns)
+        # Validate patterns
+        self.validate_patterns(
+            types=types,
+            replacements=replacements,
+            patterns=patterns,
+            partial_pattern_ok=partial_pattern_ok,
+        )
         self.replacements = replacements
+        self.patterns = patterns
+        self.partial_pattern_ok = partial_pattern_ok
 
         # Validate confounds format
         if (
@@ -436,14 +447,26 @@ class PatternDataGrabber(BaseDataGrabber):
         for t_idx in reversed(order):
             t_type = self.types[t_idx]
             types_element = set()
-            # Get the pattern
+
+            # Get the pattern dict
             t_pattern = self.patterns[t_type]
+            # Conditional fetch of base pattern for getting elements
+            pattern = None
+            # Try for data type pattern
+            pattern = t_pattern.get("pattern")
+            # Try for nested data type pattern
+            if pattern is None and self.partial_pattern_ok:
+                for v in t_pattern.values():
+                    if isinstance(v, dict) and "pattern" in v:
+                        pattern = v["pattern"]
+                        break
+
             # Replace the pattern
             (
                 re_pattern,
                 glob_pattern,
                 t_replacements,
-            ) = self._replace_patterns_regex(t_pattern["pattern"])
+            ) = self._replace_patterns_regex(pattern)
             for fname in self.datadir.glob(glob_pattern):
                 suffix = fname.relative_to(self.datadir).as_posix()
                 m = re.match(re_pattern, suffix)