junifer 0.0.8.dev306__tar.gz → 0.0.8.dev314__tar.gz

{junifer-0.0.8.dev306/junifer.egg-info → junifer-0.0.8.dev314}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: junifer
-Version: 0.0.8.dev306
+Version: 0.0.8.dev314
 Summary: JUelich NeuroImaging FEature extractoR
 Author-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
 Maintainer-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>

junifer-0.0.8.dev314/docs/changes/newsfragments/524.bugfix

@@ -0,0 +1 @@
+Fix partial element selectors (e.g., only the subject) always being reported as invalid by :func:`.run`, after computing their elements by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.bugfix.1

@@ -0,0 +1 @@
+Fix a relative ``datadir`` of the DataGrabber in the YAML being relative to the current directory (e.g., the job directory when queueing) instead of the YAML file by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.change

@@ -0,0 +1 @@
+Use only the replacements of the data types to grab as element keys in :class:`.PatternDataGrabber` (e.g., only ``subject`` with VBM data), so the data that does not depend on, e.g., the task is not processed once for each task by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.change.1

@@ -0,0 +1 @@
+Find the elements of :class:`.DataladAOMICPIOP1`, :class:`.DataladAOMICPIOP2`, :class:`.HCP1200` and :class:`.DMCC13Benchmark` in the dataset, so only the elements with data are listed, and return absolute paths in :class:`.PatternDataGrabber` by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.change.2

@@ -0,0 +1 @@
+Require the element selectors to have the same number of values, and each data type to be grabbed by only one of the DataGrabbers of :class:`.MultipleDataGrabber` by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.doc

@@ -0,0 +1 @@
+Document the elements of the DataGrabbers, combining datasets with :class:`.MultipleDataGrabber`, complete and partial elements, and restricting the values of the replacements by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.enh

@@ -0,0 +1 @@
+Run complete elements (with a value for each element key, e.g., in each queued job) without searching the dataset for its elements by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.feature

@@ -0,0 +1 @@
+Allow restricting the values of the replacements to grab in :class:`.PatternDataGrabber`, with ``replacements`` as a dictionary or with fields linked with ``_REPLACEMENT_FIELDS`` (e.g., ``tasks`` in the built-in DataGrabbers), and using the ``*`` and ``?`` wildcards in the patterns to find the elements by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/524.feature.1

@@ -0,0 +1 @@
+Allow :class:`.MultipleDataGrabber` to combine DataGrabbers with different element keys, joining their elements on the common keys, and to replace nested data types (e.g., the confounds of ``BOLD``) with a DataGrabber that only grabs them by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.bugfix

@@ -0,0 +1 @@
+Fix the same features being stored more than once: for each value of the element keys that the data does not depend on (e.g., the VBM features of a subject once per task), when running an element again, and when collecting again (:class:`.SQLiteFeatureStorage` also collected its own output file) by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.bugfix.1

@@ -0,0 +1 @@
+Fix the metadata of the preprocessors being replaced by the one of the last preprocessor by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change

@@ -0,0 +1 @@
+Add the ``processed_element`` argument to the ``store_*`` methods of :class:`.BaseFeatureStorage`, the element being processed, so the concrete storages can name the file of each element; custom storages must accept it by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.1

@@ -0,0 +1 @@
+Store the metadata of the preprocessors as a list, in order, in ``meta["preprocess"]`` by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.2

@@ -0,0 +1 @@
+Use ``upsert="ignore"`` by default in :class:`.SQLiteFeatureStorage`, so the elements already stored are not stored again by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.3

@@ -0,0 +1 @@
+Add the version of ``junifer`` to the MD5 of the features, as for the dependencies of the markers and steps, for reproducibility: features computed with different versions of ``junifer`` are different features, so the differences between features (e.g., between datasets) can only come from the data. Running again with a new version stores the features again, with the same name, next to the ones computed with the previous version by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.4

@@ -0,0 +1 @@
+Raise an error in the ``collect`` method of :class:`.HDF5FeatureStorage` and :class:`.SQLiteFeatureStorage` if a file to collect has more than one element, as ``collect`` only compacts the files of single elements by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.doc

@@ -0,0 +1 @@
+Add the "Implementation details" page with the elements, the fields of the metadata and the storage, and document the data types with different element keys when extending DataGrabbers by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.enh

@@ -0,0 +1 @@
+Store the features for the element with only the keys that the data type depends on (e.g., only ``subject`` for VBM data, when the elements also have ``task``), and skip the elements already stored by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.feature

@@ -0,0 +1 @@
+Add :meth:`.BaseDataGrabber.get_type_element_keys` to get the element keys that the data of a data type depends on by `Fede Raimondo`_

{junifer-0.0.8.dev306 → junifer-0.0.8.dev314}/docs/extending/datagrabber.rst

@@ -48,6 +48,13 @@ acquired for each task, but once in the entire session. So in this case, the
 ``T1w`` image for the element (``sub001``, ``ses1``, ``rest``) will be the same
 as the ``T1w`` image for the element (``sub001``, ``ses1``, ``stroop``).
 
+When the DataGrabber only grabs some of the data types, the elements only have
+the items that these data types need. For example, if only the ``T1w`` image is
+grabbed, the elements are (``subject``, ``session``), e.g., (``sub001``,
+``ses1``), so that the same ``T1w`` image is not processed once for each task.
+For the :class:`.PatternDataGrabber` (see below), the items that the data types
+need are the replacements in their patterns.
+
 We will now continue this section using as an example, a dataset in BIDS format
 in which 9 subjects (``sub-01`` to ``sub-09``) were scanned each during 3
 sessions (``ses-01``, ``ses-02``, ``ses-03``) and each session included a
@@ -200,6 +207,102 @@ to set the ``datadir``.
          kind: ExampleBIDSDataGrabber
          datadir: /data/project/test/data
 
+The DataGrabber finds the elements by searching the dataset for the files that
+match the pattern of each data type. An element is only available if it has the
+files of all the data types to grab. In our BIDS example, ``ses-03`` only has
+anatomical data, so the elements with ``ses-03`` are only available when only
+the ``T1w`` data type is grabbed.
+
+
+.. _extending_datagrabbers_replacement_values:
+
+Restricting the values of the replacements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, the DataGrabber grabs all the values of the replacements found in
+the dataset. To only grab some of them, ``replacements`` can be a dictionary
+with the values to grab for each replacement (or ``None`` to grab all of them).
+For example, to only grab the first two sessions of our BIDS example with the
+:class:`.PatternDataGrabber` in the YAML file:
+
+.. code-block:: yaml
+
+      datagrabber:
+         kind: PatternDataGrabber
+         datadir: /data/project/test/data
+         types:
+            - T1w
+            - BOLD
+         patterns:
+            T1w:
+               pattern: "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
+               space: native
+            BOLD:
+               pattern: "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz"
+               space: MNI152NLin6Asym
+         replacements:
+            subject: null
+            session:
+               - ses-01
+               - ses-02
+
+The elements with other values are not listed, and grabbing them raises an
+error. The values are ignored if the data types to grab do not use the
+replacement, e.g., the ``session`` values with a data type that is the same for
+all the sessions.
+
+A DataGrabber can also take the values to grab from one of its fields, e.g.,
+``sessions``, by linking the field to the replacement with
+``_REPLACEMENT_FIELDS``. The values of the field can be strings or enumerations
+(a :class:`enum.StrEnum` is recommended, so its members are also the strings),
+and the replacement cannot have values in ``replacements`` too:
+
+.. code-block:: python
+
+    from enum import StrEnum
+    from typing import ClassVar
+
+    from junifer.api.decorators import register_datagrabber
+    from junifer.datagrabber import DataType, PatternDataGrabber
+    from junifer.typing import DataGrabberPatterns
+
+
+    class Sessions(StrEnum):
+        ses_01 = "ses-01"
+        ses_02 = "ses-02"
+        ses_03 = "ses-03"
+
+
+    @register_datagrabber
+    class ExampleBIDSDataGrabber(PatternDataGrabber):
+
+        types: list[DataType] = [DataType.T1w, DataType.BOLD]
+        patterns: DataGrabberPatterns = {
+            "T1w": {
+                "pattern": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
+                "space": "native",
+            },
+            "BOLD": {
+                "pattern": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
+                "space": "MNI152NLin6Asym",
+            },
+        }
+        replacements: list[str] = ["subject", "session"]
+        sessions: list[Sessions] = list(Sessions)
+        # Only grab the sessions in the `sessions` field
+        _REPLACEMENT_FIELDS: ClassVar[dict[str, str]] = {"session": "sessions"}
+
+Then, the sessions to grab can be set in the YAML file:
+
+.. code-block:: yaml
+
+      datagrabber:
+         kind: ExampleBIDSDataGrabber
+         datadir: /data/project/test/data
+         sessions:
+            - ses-01
+            - ses-02
+
 
 Optional: Using datalad
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -296,20 +399,38 @@ This approach can be used directly from the YAML, like so:
      uri: "https://cerebra.fz-juelich.de/junifer/datalad-example-bids-ses.git"
      rootdir: example_bids_ses
 
+.. _extending_datagrabbers_path_expansion:
+
 Advanced: Using Unix-like path expansion directives
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 It is also possible to use some advanced Unix-like path expansion tricks to
 define our patterns.
 
-A very common thing would be to use ``*`` to match any number of
-characters but we cannot use it right after a replacement like:
+A very common thing would be to use ``*`` to match any number of characters
+or ``?`` to match a single character, both to find the elements and to grab
+their data. For example, if the acquisition of the BOLD images depends on the
+task (e.g., ``sub-01_task-rest_acq-mb3_bold.nii.gz`` and
+``sub-01_task-stroop_acq-seq_bold.nii.gz``), the pattern can match any
+acquisition, so the task values are ``rest`` and ``stroop``:
 
 .. code-block:: python
 
-    "derivatives/freesurfer/{subject}*"
+    "{subject}/func/{subject}_task-{task}_acq-*_bold.nii.gz"
+
+Similarly, ``?`` can match the first character of ``rfMRI`` (resting-state)
+and ``tfMRI`` (task) in the HCP dataset:
+
+.. code-block:: python
 
-or if there are multiple files or no files which can be globbed.
+    "{subject}/MNINonLinear/Results/?fMRI_{task}_{phase_encoding}/?fMRI_{task}_{phase_encoding}.nii.gz"
+
+The pattern must match exactly one file for each element, and we cannot use
+``*`` right after a replacement like:
+
+.. code-block:: python
+
+    "derivatives/freesurfer/{subject}*"
 
 We can also use ``[]`` and ``[!]`` to glob certain tricky files like with the
 case of FreeSurfer derivatives. The file structure seen in a typical
@@ -357,7 +478,13 @@ need to implement the following methods:
 
 - ``get_item``: to get a single item from the dataset.
 - ``get_elements``: to get the list of all elements present in the dataset
-- ``get_element_keys``: to get the keys of the elements in the dataset.
+- ``get_element_keys``: to get the keys of the elements in the dataset. They
+  also give the order of the values in each element, which must match the
+  parameters of ``get_item``.
+
+Optionally, if the data of some data types does not depend on all the keys of
+the element, we can also implement ``get_type_element_keys`` (see
+:ref:`extending_datagrabbers_type_keys`).
 
 .. note::
 
@@ -396,17 +523,15 @@ need to remember that for session *ses-03* there is no BOLD data.
    from itertools import product
 
 
-   def get_elements(self) -> list[str]:
+   def get_elements(self) -> list[tuple[str, str]]:
        subjects = ["sub-01", "sub-02", "sub-03"]
        sessions = ["ses-01", "ses-02"]
 
        # If we are not working on BOLD data, we can add "ses-03"
        if "BOLD" not in self.types:
            sessions.append("ses-03")
-       elements = []
-       for subject, element in product(subjects, sessions):
-           elements.append({"subject": subject, "session": session})
-       return elements
+       # The values of each element, in the order of the element keys
+       return list(product(subjects, sessions))
 
 
 And finally, we can implement the ``get_element_keys`` method. This method needs
@@ -419,11 +544,39 @@ method, in the same order.
    def get_element_keys(self) -> list[str]:
        return ["subject", "session"]
 
+.. _extending_datagrabbers_type_keys:
+
+Data types with different keys
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, the data of every data type depends on all the element keys. If
+the data of a data type only depends on some of them, the DataGrabber can say
+so by implementing ``get_type_element_keys``, which returns the element keys
+that the data of a data type depends on. For example, if the ``T1w`` image of
+our dataset was acquired once for each subject, instead of once for each
+session:
+
+.. code-block:: python
+
+   def get_type_element_keys(self, data_type: str) -> list[str]:
+       if data_type == "T1w":
+           return ["subject"]
+       return ["subject", "session"]
+
+Then, the ``T1w`` data (and the features computed from it) is stored once for
+each subject, instead of once for each session (see :ref:`internals`). The
+element keys must still be all the keys that the data types to grab depend on:
+if only ``T1w`` was grabbed, ``get_element_keys`` should return
+``["subject"]`` (and ``get_elements`` and ``get_item`` should only use the
+subject).
+
 
 So, to summarise, our DataGrabber will look like this:
 
 .. code-block:: python
 
+   from itertools import product
+
    from junifer.api.decorators import register_datagrabber
    from junifer.datagrabber import BaseDataGrabber
 
@@ -445,18 +598,15 @@ So, to summarise, our DataGrabber will look like this:
            }
            return out
 
-       def get_elements(self) -> list[str]:
+       def get_elements(self) -> list[tuple[str, str]]:
            subjects = ["sub-01", "sub-02", "sub-03"]
            sessions = ["ses-01", "ses-02"]
 
            # If we are not working on BOLD data, we can add "ses-03"
            if "BOLD" not in self.types:
                sessions.append("ses-03")
-           elements = []
-           for subject in subjects:
-               for session in sessions:
-                   elements.append({"subject": subject, "session": session})
-           return elements
+           # The values of each element, in the order of the element keys
+           return list(product(subjects, sessions))
 
        def get_element_keys(self) -> list[str]:
            return ["subject", "session"]

{junifer-0.0.8.dev306 → junifer-0.0.8.dev314}/docs/index.rst

@@ -33,6 +33,7 @@ enabling others to extend it easily.
    help
    contribution
    maintaining
+   internals
    faq
    whats_new
 

junifer-0.0.8.dev314/docs/internals.rst

@@ -0,0 +1,169 @@
+.. include:: links.inc
+
+.. _internals:
+
+Implementation details
+======================
+
+This section describes some of the assumptions and technical details of
+``junifer``, for developers. See the docstrings of the mentioned functions and
+classes for more details.
+
+Elements
+--------
+
+* The keys of the elements of a :class:`.PatternDataGrabber`
+  (:meth:`.PatternDataGrabber.get_element_keys`) are the replacements used by
+  the patterns (including the nested ones) of the data types to grab, in the
+  order of ``replacements``.
+* :meth:`.PatternDataGrabber.get_elements` finds the elements of each pattern
+  in the files (translating the Unix-like path expansion directives of the
+  patterns to a regular expression) and joins the elements of the patterns on
+  their common keys (``_join_elements``), so an element is available if it has
+  the files of all the data types, regardless of their order.
+* The values of the replacements to grab (:meth:`.PatternDataGrabber.get_replacement_values`)
+  are given by ``replacements`` (as a dictionary) or by the fields linked to
+  the replacements in ``_REPLACEMENT_FIELDS``, but not both. They filter the
+  elements found in the files, and grabbing an element with other values
+  raises an error (``_check_element``, called by
+  ``BaseDataGrabber.__getitem__``).
+* :meth:`.BaseDataGrabber.select_elements` selects the elements to run: the
+  selectors must have the same number of values. If they have a value for
+  each element key, they are the elements (the dataset is not searched),
+  otherwise they select the elements of the dataset that have their values.
+* :class:`.MultipleDataGrabber` joins the elements of its DataGrabbers on
+  their common keys, and grabs an element with each DataGrabber using its
+  keys. Each data type is grabbed by one DataGrabber, except for the nested
+  data types grabbed by a DataGrabber without the main file of the data type,
+  which replace the ones of the other DataGrabber.
+
+Metadata
+--------
+
+The ``meta`` of each data type in the :ref:`data object <data_object>` is
+built by the steps of the pipeline. Each step adds an entry with its class and
+parameters (``UpdateMetaMixin.update_meta``) and its dependencies:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - Key
+     - Added by
+     - Content
+     - In the MD5
+   * - ``datagrabber``
+     - DataGrabber
+     - The class and parameters of the DataGrabber, e.g., ``types``,
+       ``patterns`` and ``replacements``. For the DataLad-based ones, also the
+       dataset ID and commit, and whether the dataset has local changes
+       (``datalad_id``, ``datalad_commit_id`` and ``datalad_dirty``), so
+       different versions of the dataset give different hashes. For :class:`.MultipleDataGrabber`, also
+       ``datagrabbers`` with the entries of its DataGrabbers.
+     - Yes, except ``datadir`` for the DataLad-based DataGrabbers (which is
+       removed from the stored metadata too)
+   * - ``element``
+     - DataGrabber
+     - The element being processed (see below).
+     - No
+   * - ``_element_keys``
+     - DataGrabber
+     - The keys of the element that the data depends on (see below).
+     - Yes
+   * - ``datareader``
+     - DataReader
+     - The class and parameters of the DataReader.
+     - Yes
+   * - ``preprocess``
+     - Preprocessors
+     - The list of the classes and parameters of the preprocessors that
+       processed the data, in order.
+     - Yes
+   * - ``type``
+     - Marker
+     - The data type, e.g., ``BOLD``.
+     - Yes
+   * - ``marker``
+     - Marker
+     - The class and parameters of the marker, with ``name`` as the name of
+       the marker followed by the name of the feature, e.g., ``mean_mean``.
+     - Yes
+   * - ``dependencies``
+     - All the steps
+     - The names of the dependencies of the steps (their ``_DEPENDENCIES``).
+       When storing, they are replaced by their versions, with the version of
+       ``junifer``, e.g., ``{'junifer': '0.0.8', 'nilearn': '0.14.1'}``.
+     - Yes (the versions)
+   * - ``name``
+     - Storage (``process_meta``)
+     - The name of the feature: the data type and the marker name, e.g.,
+       ``BOLD_mean_mean``.
+     - Yes
+
+The DataGrabber adds two keys about the element to the ``meta`` of each data
+type (``BaseDataGrabber.__getitem__``):
+
+* ``element``: the element being processed, with all its keys. It is the same
+  for all the data types of the element.
+* ``_element_keys``: the keys of the element that the data of the data type
+  depends on (:meth:`.BaseDataGrabber.get_type_element_keys`). 
+  For a :class:`.PatternDataGrabber`, the replacements in the patterns of the
+  data type.
+
+For example, with ``BOLD`` data for each subject and task, and ``VBM_GM`` data
+for each subject, the element ``(sub-01, rest)`` gives:
+
+.. code-block::
+
+   {'BOLD': {'meta': {'_element_keys': ['subject', 'task'],
+                      'element': {'subject': 'sub-01', 'task': 'rest'},
+                      ...},
+             ...},
+    'VBM_GM': {'meta': {'_element_keys': ['subject'],
+                        'element': {'subject': 'sub-01', 'task': 'rest'},
+                        ...},
+               ...}}
+
+When storing the output of a marker, ``process_meta`` (in ``junifer.storage.utils``):
+
+* Removes ``element`` from the metadata and returns the element of the data,
+  with only the keys in ``_element_keys`` (all the keys of ``element`` if it
+  is not set).
+* Computes the MD5 hash of the metadata, which identifies the feature. It
+  includes ``_element_keys`` and the versions of the dependencies of the
+  steps and of ``junifer``, but not ``element``: the data of all the elements
+  of a feature have the same hash, and the features computed with different
+  versions of ``junifer`` (or of the dependencies) have different hashes.
+  This is by design, for reproducibility: the differences between features
+  with the same hash can only come from the data, not from different
+  versions of the algorithms. A feature computed again with another version
+  is a new feature, with the same name, so it must be read by its MD5.
+
+Storage
+-------
+
+* The data is stored for the element of the data (the element with only the
+  keys in ``_element_keys``), so the data that does not depend on a key (e.g.,
+  the ``VBM_GM`` data for the tasks of a subject) is stored once.
+* :meth:`.BaseFeatureStorage.store` passes the element being processed to
+  the methods storing the metadata and the data (``store_metadata``,
+  ``store_vector``, etc.) as ``processed_element``, besides the element of the
+  data (``element``). The storages that write a file for each element (with
+  ``single_output=False``) name it after the element being processed, e.g.,
+  ``element_sub-01_rest_out.hdf5``, so that the processes running different
+  elements in parallel never write to the same file. The storages without
+  files (e.g., databases) can ignore it, but their methods must accept it.
+* The data already stored for an element is not stored again (the storages
+  skip the existing elements, :class:`.SQLiteFeatureStorage` with
+  ``upsert="ignore"`` by default).
+* ``collect`` only collects the files of single elements (one element for
+  each feature, as written for each element with ``single_output=False``),
+  and raises an error otherwise. The data of an element can be in the files
+  of several elements being processed (e.g., the ``VBM_GM`` data of
+  ``sub-01`` in the files of each of its tasks), and it is only collected
+  once.
+* :meth:`.HDF5FeatureStorage.collect` works in two passes: the first one reads
+  the metadata and the elements of each file, to know which files to collect
+  for each feature (one for each element), and the second one collects the
+  data of each feature from these files, in chunks. Each file is read once in
+  the first pass, and once for each of its features in the second one.