junifer 0.0.8.dev311__tar.gz → 0.0.8.dev314__tar.gz

{junifer-0.0.8.dev311/junifer.egg-info → junifer-0.0.8.dev314}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: junifer
-Version: 0.0.8.dev311
+Version: 0.0.8.dev314
 Summary: JUelich NeuroImaging FEature extractoR
 Author-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
 Maintainer-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>

junifer-0.0.8.dev314/docs/changes/newsfragments/525.bugfix

@@ -0,0 +1 @@
+Fix the same features being stored more than once: for each value of the element keys that the data does not depend on (e.g., the VBM features of a subject once per task), when running an element again, and when collecting again (:class:`.SQLiteFeatureStorage` also collected its own output file) by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.bugfix.1

@@ -0,0 +1 @@
+Fix the metadata of the preprocessors being replaced by the one of the last preprocessor by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change

@@ -0,0 +1 @@
+Add the ``processed_element`` argument to the ``store_*`` methods of :class:`.BaseFeatureStorage`, the element being processed, so the concrete storages can name the file of each element; custom storages must accept it by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.1

@@ -0,0 +1 @@
+Store the metadata of the preprocessors as a list, in order, in ``meta["preprocess"]`` by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.2

@@ -0,0 +1 @@
+Use ``upsert="ignore"`` by default in :class:`.SQLiteFeatureStorage`, so the elements already stored are not stored again by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.3

@@ -0,0 +1 @@
+Add the version of ``junifer`` to the MD5 of the features, as for the dependencies of the markers and steps, for reproducibility: features computed with different versions of ``junifer`` are different features, so the differences between features (e.g., between datasets) can only come from the data. Running again with a new version stores the features again, with the same name, next to the ones computed with the previous version by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.change.4

@@ -0,0 +1 @@
+Raise an error in the ``collect`` method of :class:`.HDF5FeatureStorage` and :class:`.SQLiteFeatureStorage` if a file to collect has more than one element, as ``collect`` only compacts the files of single elements by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.doc

@@ -0,0 +1 @@
+Add the "Implementation details" page with the elements, the fields of the metadata and the storage, and document the data types with different element keys when extending DataGrabbers by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.enh

@@ -0,0 +1 @@
+Store the features for the element with only the keys that the data type depends on (e.g., only ``subject`` for VBM data, when the elements also have ``task``), and skip the elements already stored by `Fede Raimondo`_

junifer-0.0.8.dev314/docs/changes/newsfragments/525.feature

@@ -0,0 +1 @@
+Add :meth:`.BaseDataGrabber.get_type_element_keys` to get the element keys that the data of a data type depends on by `Fede Raimondo`_

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/docs/extending/datagrabber.rst

@@ -478,7 +478,13 @@ need to implement the following methods:
 
 - ``get_item``: to get a single item from the dataset.
 - ``get_elements``: to get the list of all elements present in the dataset
-- ``get_element_keys``: to get the keys of the elements in the dataset.
+- ``get_element_keys``: to get the keys of the elements in the dataset. They
+  also give the order of the values in each element, which must match the
+  parameters of ``get_item``.
+
+Optionally, if the data of some data types does not depend on all the keys of
+the element, we can also implement ``get_type_element_keys`` (see
+:ref:`extending_datagrabbers_type_keys`).
 
 .. note::
 
@@ -517,17 +523,15 @@ need to remember that for session *ses-03* there is no BOLD data.
    from itertools import product
 
 
-   def get_elements(self) -> list[str]:
+   def get_elements(self) -> list[tuple[str, str]]:
        subjects = ["sub-01", "sub-02", "sub-03"]
        sessions = ["ses-01", "ses-02"]
 
        # If we are not working on BOLD data, we can add "ses-03"
        if "BOLD" not in self.types:
            sessions.append("ses-03")
-       elements = []
-       for subject, element in product(subjects, sessions):
-           elements.append({"subject": subject, "session": session})
-       return elements
+       # The values of each element, in the order of the element keys
+       return list(product(subjects, sessions))
 
 
 And finally, we can implement the ``get_element_keys`` method. This method needs
@@ -540,11 +544,39 @@ method, in the same order.
    def get_element_keys(self) -> list[str]:
        return ["subject", "session"]
 
+.. _extending_datagrabbers_type_keys:
+
+Data types with different keys
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, the data of every data type depends on all the element keys. If
+the data of a data type only depends on some of them, the DataGrabber can say
+so by implementing ``get_type_element_keys``, which returns the element keys
+that the data of a data type depends on. For example, if the ``T1w`` image of
+our dataset was acquired once for each subject, instead of once for each
+session:
+
+.. code-block:: python
+
+   def get_type_element_keys(self, data_type: str) -> list[str]:
+       if data_type == "T1w":
+           return ["subject"]
+       return ["subject", "session"]
+
+Then, the ``T1w`` data (and the features computed from it) is stored once for
+each subject, instead of once for each session (see :ref:`internals`). The
+element keys must still be all the keys that the data types to grab depend on:
+if only ``T1w`` was grabbed, ``get_element_keys`` should return
+``["subject"]`` (and ``get_elements`` and ``get_item`` should only use the
+subject).
+
 
 So, to summarise, our DataGrabber will look like this:
 
 .. code-block:: python
 
+   from itertools import product
+
    from junifer.api.decorators import register_datagrabber
    from junifer.datagrabber import BaseDataGrabber
 
@@ -566,18 +598,15 @@ So, to summarise, our DataGrabber will look like this:
            }
            return out
 
-       def get_elements(self) -> list[str]:
+       def get_elements(self) -> list[tuple[str, str]]:
            subjects = ["sub-01", "sub-02", "sub-03"]
            sessions = ["ses-01", "ses-02"]
 
            # If we are not working on BOLD data, we can add "ses-03"
            if "BOLD" not in self.types:
                sessions.append("ses-03")
-           elements = []
-           for subject in subjects:
-               for session in sessions:
-                   elements.append({"subject": subject, "session": session})
-           return elements
+           # The values of each element, in the order of the element keys
+           return list(product(subjects, sessions))
 
        def get_element_keys(self) -> list[str]:
            return ["subject", "session"]

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/docs/index.rst

@@ -33,6 +33,7 @@ enabling others to extend it easily.
    help
    contribution
    maintaining
+   internals
    faq
    whats_new
 

junifer-0.0.8.dev314/docs/internals.rst

@@ -0,0 +1,169 @@
+.. include:: links.inc
+
+.. _internals:
+
+Implementation details
+======================
+
+This section describes some of the assumptions and technical details of
+``junifer``, for developers. See the docstrings of the mentioned functions and
+classes for more details.
+
+Elements
+--------
+
+* The keys of the elements of a :class:`.PatternDataGrabber`
+  (:meth:`.PatternDataGrabber.get_element_keys`) are the replacements used by
+  the patterns (including the nested ones) of the data types to grab, in the
+  order of ``replacements``.
+* :meth:`.PatternDataGrabber.get_elements` finds the elements of each pattern
+  in the files (translating the Unix-like path expansion directives of the
+  patterns to a regular expression) and joins the elements of the patterns on
+  their common keys (``_join_elements``), so an element is available if it has
+  the files of all the data types, regardless of their order.
+* The values of the replacements to grab (:meth:`.PatternDataGrabber.get_replacement_values`)
+  are given by ``replacements`` (as a dictionary) or by the fields linked to
+  the replacements in ``_REPLACEMENT_FIELDS``, but not both. They filter the
+  elements found in the files, and grabbing an element with other values
+  raises an error (``_check_element``, called by
+  ``BaseDataGrabber.__getitem__``).
+* :meth:`.BaseDataGrabber.select_elements` selects the elements to run: the
+  selectors must have the same number of values. If they have a value for
+  each element key, they are the elements (the dataset is not searched),
+  otherwise they select the elements of the dataset that have their values.
+* :class:`.MultipleDataGrabber` joins the elements of its DataGrabbers on
+  their common keys, and grabs an element with each DataGrabber using its
+  keys. Each data type is grabbed by one DataGrabber, except for the nested
+  data types grabbed by a DataGrabber without the main file of the data type,
+  which replace the ones of the other DataGrabber.
+
+Metadata
+--------
+
+The ``meta`` of each data type in the :ref:`data object <data_object>` is
+built by the steps of the pipeline. Each step adds an entry with its class and
+parameters (``UpdateMetaMixin.update_meta``) and its dependencies:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - Key
+     - Added by
+     - Content
+     - In the MD5
+   * - ``datagrabber``
+     - DataGrabber
+     - The class and parameters of the DataGrabber, e.g., ``types``,
+       ``patterns`` and ``replacements``. For the DataLad-based ones, also the
+       dataset ID and commit, and whether the dataset has local changes
+       (``datalad_id``, ``datalad_commit_id`` and ``datalad_dirty``), so
+       different versions of the dataset give different hashes. For :class:`.MultipleDataGrabber`, also
+       ``datagrabbers`` with the entries of its DataGrabbers.
+     - Yes, except ``datadir`` for the DataLad-based DataGrabbers (which is
+       removed from the stored metadata too)
+   * - ``element``
+     - DataGrabber
+     - The element being processed (see below).
+     - No
+   * - ``_element_keys``
+     - DataGrabber
+     - The keys of the element that the data depends on (see below).
+     - Yes
+   * - ``datareader``
+     - DataReader
+     - The class and parameters of the DataReader.
+     - Yes
+   * - ``preprocess``
+     - Preprocessors
+     - The list of the classes and parameters of the preprocessors that
+       processed the data, in order.
+     - Yes
+   * - ``type``
+     - Marker
+     - The data type, e.g., ``BOLD``.
+     - Yes
+   * - ``marker``
+     - Marker
+     - The class and parameters of the marker, with ``name`` as the name of
+       the marker followed by the name of the feature, e.g., ``mean_mean``.
+     - Yes
+   * - ``dependencies``
+     - All the steps
+     - The names of the dependencies of the steps (their ``_DEPENDENCIES``).
+       When storing, they are replaced by their versions, with the version of
+       ``junifer``, e.g., ``{'junifer': '0.0.8', 'nilearn': '0.14.1'}``.
+     - Yes (the versions)
+   * - ``name``
+     - Storage (``process_meta``)
+     - The name of the feature: the data type and the marker name, e.g.,
+       ``BOLD_mean_mean``.
+     - Yes
+
+The DataGrabber adds two keys about the element to the ``meta`` of each data
+type (``BaseDataGrabber.__getitem__``):
+
+* ``element``: the element being processed, with all its keys. It is the same
+  for all the data types of the element.
+* ``_element_keys``: the keys of the element that the data of the data type
+  depends on (:meth:`.BaseDataGrabber.get_type_element_keys`). 
+  For a :class:`.PatternDataGrabber`, the replacements in the patterns of the
+  data type.
+
+For example, with ``BOLD`` data for each subject and task, and ``VBM_GM`` data
+for each subject, the element ``(sub-01, rest)`` gives:
+
+.. code-block::
+
+   {'BOLD': {'meta': {'_element_keys': ['subject', 'task'],
+                      'element': {'subject': 'sub-01', 'task': 'rest'},
+                      ...},
+             ...},
+    'VBM_GM': {'meta': {'_element_keys': ['subject'],
+                        'element': {'subject': 'sub-01', 'task': 'rest'},
+                        ...},
+               ...}}
+
+When storing the output of a marker, ``process_meta`` (in ``junifer.storage.utils``):
+
+* Removes ``element`` from the metadata and returns the element of the data,
+  with only the keys in ``_element_keys`` (all the keys of ``element`` if it
+  is not set).
+* Computes the MD5 hash of the metadata, which identifies the feature. It
+  includes ``_element_keys`` and the versions of the dependencies of the
+  steps and of ``junifer``, but not ``element``: the data of all the elements
+  of a feature have the same hash, and the features computed with different
+  versions of ``junifer`` (or of the dependencies) have different hashes.
+  This is by design, for reproducibility: the differences between features
+  with the same hash can only come from the data, not from different
+  versions of the algorithms. A feature computed again with another version
+  is a new feature, with the same name, so it must be read by its MD5.
+
+Storage
+-------
+
+* The data is stored for the element of the data (the element with only the
+  keys in ``_element_keys``), so the data that does not depend on a key (e.g.,
+  the ``VBM_GM`` data for the tasks of a subject) is stored once.
+* :meth:`.BaseFeatureStorage.store` passes the element being processed to
+  the methods storing the metadata and the data (``store_metadata``,
+  ``store_vector``, etc.) as ``processed_element``, besides the element of the
+  data (``element``). The storages that write a file for each element (with
+  ``single_output=False``) name it after the element being processed, e.g.,
+  ``element_sub-01_rest_out.hdf5``, so that the processes running different
+  elements in parallel never write to the same file. The storages without
+  files (e.g., databases) can ignore it, but their methods must accept it.
+* The data already stored for an element is not stored again (the storages
+  skip the existing elements, :class:`.SQLiteFeatureStorage` with
+  ``upsert="ignore"`` by default).
+* ``collect`` only collects the files of single elements (one element for
+  each feature, as written for each element with ``single_output=False``),
+  and raises an error otherwise. The data of an element can be in the files
+  of several elements being processed (e.g., the ``VBM_GM`` data of
+  ``sub-01`` in the files of each of its tasks), and it is only collected
+  once.
+* :meth:`.HDF5FeatureStorage.collect` works in two passes: the first one reads
+  the metadata and the elements of each file, to know which files to collect
+  for each feature (one for each element), and the second one collects the
+  data of each feature from these files, in chunks. Each file is read once in
+  the first pass, and once for each of its features in the second one.

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/docs/understanding/storage.rst

@@ -30,6 +30,14 @@ your own ``Storage`` by providing a concrete implementation of
 :class:`.BaseFeatureStorage` or open an issue on `junifer Github`_ and we can
 help you out.
 
+The data of each element is stored only once: running the same elements again
+with the same configuration (e.g., after trying a few elements and then running
+all of them), or collecting the files of the elements again, does not change
+the data already stored, which is not stored again. The data that does not
+depend on all the keys of the element (e.g., the ``VBM_GM`` data of a subject,
+for elements with the subject and the task) is stored once for its keys (see
+:ref:`internals`).
+
 .. _storage_types:
 
 Storage Types

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/junifer/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.0.8.dev311'
-__version_tuple__ = version_tuple = (0, 0, 8, 'dev311')
+__version__ = version = '0.0.8.dev314'
+__version_tuple__ = version_tuple = (0, 0, 8, 'dev314')
 
-__commit_id__ = commit_id = 'gf37e90687'
+__commit_id__ = commit_id = 'g968a03e78'

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/junifer/api/tests/test_functions.py

@@ -9,23 +9,30 @@ import logging
 import sys
 import tempfile
 from contextlib import AbstractContextManager, nullcontext
+from copy import deepcopy
 from pathlib import Path
-from typing import Any
+from typing import Any, ClassVar
 
+import nibabel as nib
+import numpy as np
 import pytest
 from nibabel.filebasedimages import ImageFileError
 from ruamel.yaml import YAML
 
 import junifer.testing.registry  # noqa: F401
 from junifer.api import collect, list_elements, parse_yaml, queue, reset, run
-from junifer.datagrabber import PatternDataGrabber
+from junifer.api.decorators import register_marker
+from junifer.datagrabber import DataType, PatternDataGrabber
 from junifer.datagrabber.base import BaseDataGrabber
+from junifer.markers import BaseMarker
 from junifer.pipeline import (
     MarkerCollection,
     PipelineComponentRegistry,
     WorkDirManager,
 )
-from junifer.typing import Elements
+from junifer.storage import StorageType
+from junifer.storage.base import BaseFeatureStorage
+from junifer.typing import Dependencies, Elements, MarkerInOutMappings
 
 
 pytestmark = pytest.mark.external
@@ -1460,3 +1467,119 @@ def _run_selectors(
         elements=elements,
     )
     return n_searches
+
+
+@register_marker
+class _ImageMean(BaseMarker):
+    """Marker storing the mean of an image, for BOLD and VBM_GM."""
+
+    _DEPENDENCIES: ClassVar[Dependencies] = {"numpy"}
+    _MARKER_INOUT_MAPPINGS: ClassVar[MarkerInOutMappings] = {
+        DataType.BOLD: {"mean": StorageType.Vector},
+        DataType.VBM_GM: {"mean": StorageType.Vector},
+    }
+
+    def compute(self, input: dict, extra_input: dict | None = None) -> dict:
+        data = np.asarray(input["data"].dataobj, dtype=float)
+        return {
+            "mean": {"data": np.array([[data.mean()]]), "col_names": ["mean"]}
+        }
+
+
+@pytest.mark.parametrize("single_output", [True, False])
+@pytest.mark.parametrize(
+    "storage_kind", ["HDF5FeatureStorage", "SQLiteFeatureStorage"]
+)
+def test_run_data_types_element_keys(
+    tmp_path: Path, storage_kind: str, single_output: bool
+) -> None:
+    """Test run function storing data types with different element keys.
+
+    The BOLD data depends on the subject and the task, and the VBM data only
+    on the subject, so it is stored once for each subject. The elements are
+    run twice, as when trying a few elements and then running all of them.
+
+    Parameters
+    ----------
+    tmp_path : pathlib.Path
+        The path to the test directory.
+    storage_kind : str
+        The parametrized storage.
+    single_output : bool
+        The parametrized storage in a single file.
+
+    """
+    datadir = tmp_path / "data"
+    # BOLD: 10 * subject + task (1: rest, 2: movie); VBM: subject
+    for i, subject in enumerate(("sub-01", "sub-02"), start=1):
+        (datadir / subject).mkdir(parents=True)
+        nib.save(
+            nib.Nifti1Image(np.full((2, 2, 2), float(i)), np.eye(4)),
+            datadir / subject / f"{subject}_GM.nii",
+        )
+        for j, task in enumerate(("rest", "movie"), start=1):
+            nib.save(
+                nib.Nifti1Image(
+                    np.full((2, 2, 2, 3), 10.0 * i + j), np.eye(4)
+                ),
+                datadir / subject / f"{subject}_task-{task}_bold.nii",
+            )
+    datagrabber = {
+        "kind": "PatternDataGrabber",
+        "datadir": str(datadir),
+        "types": ["BOLD", "VBM_GM"],
+        "patterns": {
+            "BOLD": {
+                "pattern": "{subject}/{subject}_task-{task}_bold.nii",
+                "space": "native",
+            },
+            "VBM_GM": {
+                "pattern": "{subject}/{subject}_GM.nii",
+                "space": "native",
+            },
+        },
+        "replacements": ["subject", "task"],
+    }
+    uri = tmp_path / "out" / "features.db"
+    storage = {
+        "kind": storage_kind,
+        "uri": str(uri),
+        "single_output": single_output,
+    }
+    # A few elements, and then all of them
+    for elements in ([("sub-01", "rest")], None):
+        run(
+            workdir=tmp_path / "workdir",
+            datagrabber=deepcopy(datagrabber),
+            markers=[{"kind": "_ImageMean", "name": "mean"}],
+            storage=deepcopy(storage),
+            elements=elements,
+        )
+        if not single_output:
+            collect(deepcopy(storage))
+
+    reader = PipelineComponentRegistry().build_component_instance(
+        step="storage",
+        name=storage_kind,
+        baseclass=BaseFeatureStorage,
+        init_params={"uri": uri},
+    )
+    features = {
+        meta["name"]: md5 for md5, meta in reader.list_features().items()
+    }
+    bold = reader.read_df(feature_md5=features["BOLD_mean_mean"])
+    vbm = reader.read_df(feature_md5=features["VBM_GM_mean_mean"])
+    assert sorted(bold.index) == [
+        ("sub-01", "movie"),
+        ("sub-01", "rest"),
+        ("sub-02", "movie"),
+        ("sub-02", "rest"),
+    ]
+    assert sorted(bold["mean"]) == [11, 12, 21, 22]
+    # Once for each subject
+    assert vbm.index.names == ["subject"]
+    assert sorted(vbm.index.get_level_values("subject")) == [
+        "sub-01",
+        "sub-02",
+    ]
+    assert sorted(vbm["mean"]) == [1, 2]

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/junifer/datagrabber/base.py

@@ -183,18 +183,39 @@ class BaseDataGrabber(BaseModel, ABC, UpdateMetaMixin):
         self._check_element(named_element)
         # Fetch element
         out = self.get_item(**named_element)
-        # Update metadata
-        for _, t_val in out.items():
+        # Update metadata: the element being processed and the keys of the
+        # element that the data of each data type depends on
+        for t_type, t_val in out.items():
             self.update_meta(t_val, "datagrabber")
+            element_keys = self.get_type_element_keys(t_type)
             # Conditional for list dtype vals like Warp
-            if isinstance(t_val, list):
-                for entry in t_val:
-                    entry["meta"]["element"] = named_element
-            else:
-                t_val["meta"]["element"] = named_element
+            for entry in t_val if isinstance(t_val, list) else [t_val]:
+                entry["meta"]["element"] = named_element
+                entry["meta"]["_element_keys"] = element_keys
 
         return out
 
+    def get_type_element_keys(self, data_type: str) -> list[str]:
+        """Get the element keys that the data of a data type depends on.
+
+        The element keys (see :meth:`.get_element_keys`) are the keys that
+        the data types to grab depend on.
+
+        Parameters
+        ----------
+        data_type : str
+            The data type.
+
+        Returns
+        -------
+        list of str
+            The element keys, all of them unless a subclass knows which ones
+            the data type depends on (e.g., only the subject for data that is
+            the same for all the tasks of a subject).
+
+        """
+        return self.get_element_keys()
+
     def _check_element(self, element: dict) -> None:
         """Check the element can be grabbed.
 

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/junifer/datagrabber/multiple.py

@@ -190,12 +190,36 @@ class MultipleDataGrabber(BaseDataGrabber):
             to_update = out[kind]
             if not isinstance(to_update, list):
                 to_update = [to_update]
+            element_keys = self.get_type_element_keys(kind)
             for t_kind in to_update:
                 self.update_meta(t_kind, "datagrabber")
                 t_kind["meta"]["datagrabber"]["datagrabbers"] = metas
                 t_kind["meta"]["element"] = named_element
+                t_kind["meta"]["_element_keys"] = element_keys
         return out
 
+    def get_type_element_keys(self, data_type: str) -> list[str]:
+        """Get the element keys that the data of a data type depends on.
+
+        Parameters
+        ----------
+        data_type : str
+            The data type.
+
+        Returns
+        -------
+        list of str
+            The element keys that the data of the data type depends on in all
+            the DataGrabbers grabbing it (e.g., the one grabbing the main file
+            and the one grabbing only its nested data types).
+
+        """
+        keys = set()
+        for dg in self.datagrabbers:
+            if data_type in dg.get_types():
+                keys.update(dg.get_type_element_keys(data_type))
+        return [k for k in self.get_element_keys() if k in keys]
+
     def __enter__(self) -> "MultipleDataGrabber":
         """Implement context entry."""
         for dg in self.datagrabbers:

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/junifer/datagrabber/pattern.py

@@ -414,31 +414,47 @@ class PatternDataGrabber(BaseDataGrabber, PatternValidationMixin):
             )
         return patterns
 
-    def get_element_keys(self) -> list[str]:
-        """Get element keys.
+    def get_type_element_keys(self, data_type: str) -> list[str]:
+        """Get the element keys that the data of a data type depends on.
 
-        For each item in the "element" tuple, this functions returns the
-        corresponding key, that is, the ``replacements`` of the patterns of
-        the data types to grab (e.g., without ``task`` if only anatomical
-        data types are grabbed).
+        Parameters
+        ----------
+        data_type : str
+            The data type.
 
         Returns
         -------
         list of str
-            The element keys.
+            The replacements in the patterns of the data type (including the
+            nested ones), in the order of ``replacements``.
 
         """
-        patterns = [
-            pattern
-            for t_type in self.get_types()
-            for pattern in self._get_type_patterns(t_type)
-        ]
+        patterns = self._get_type_patterns(data_type)
         return [
             x
             for x in self.replacements
             if any(f"{{{x}}}" in pattern for pattern in patterns)
         ]
 
+    def get_element_keys(self) -> list[str]:
+        """Get element keys.
+
+        For each item in the "element" tuple, this functions returns the
+        corresponding key: the replacements in the patterns of the data
+        types to grab (see :meth:`.get_type_element_keys`), e.g., without
+        ``task`` if only anatomical data types are grabbed.
+
+        Returns
+        -------
+        list of str
+            The element keys, in the order of ``replacements``.
+
+        """
+        keys = {
+            k for t in self.get_types() for k in self.get_type_element_keys(t)
+        }
+        return [x for x in self.replacements if x in keys]
+
     def get_replacement_values(self) -> dict[str, list[str]]:
         """Get the values of the replacements to grab.
 

{junifer-0.0.8.dev311 → junifer-0.0.8.dev314}/junifer/datagrabber/tests/test_multiple.py

@@ -578,3 +578,11 @@ def test_MultipleDataGrabber_three_datasets(tmp_path: Path) -> None:
         tmp_path / "confounds/sub-02"
     )
     assert out["VBM_GM"]["path"].parent == tmp_path / "cat/sub-02/mri"
+    # The element being processed, and the keys that the data depends on
+    for data_type in ("BOLD", "VBM_GM"):
+        assert out[data_type]["meta"]["element"] == {
+            "subject": "sub-02",
+            "task": "rest",
+        }
+    assert out["BOLD"]["meta"]["_element_keys"] == ["subject", "task"]
+    assert out["VBM_GM"]["meta"]["_element_keys"] == ["subject"]