junifer 0.0.8.dev311__tar.gz → 0.0.8.dev318__tar.gz

{junifer-0.0.8.dev311/junifer.egg-info → junifer-0.0.8.dev318}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: junifer
-Version: 0.0.8.dev311
+Version: 0.0.8.dev318
 Summary: JUelich NeuroImaging FEature extractoR
 Author-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
 Maintainer-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>

junifer-0.0.8.dev318/docs/changes/newsfragments/525.bugfix

@@ -0,0 +1 @@
+Fix the same features being stored more than once: for each value of the element keys that the data does not depend on (e.g., the VBM features of a subject once per task), when running an element again, and when collecting again (:class:`.SQLiteFeatureStorage` also collected its own output file) by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.bugfix.1

@@ -0,0 +1 @@
+Fix the metadata of the preprocessors being replaced by the one of the last preprocessor by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.change

@@ -0,0 +1 @@
+Add the ``processed_element`` argument to the ``store_*`` methods of :class:`.BaseFeatureStorage`, the element being processed, so the concrete storages can name the file of each element; custom storages must accept it by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.change.1

@@ -0,0 +1 @@
+Store the metadata of the preprocessors as a list, in order, in ``meta["preprocess"]`` by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.change.2

@@ -0,0 +1 @@
+Use ``upsert="ignore"`` by default in :class:`.SQLiteFeatureStorage`, so the elements already stored are not stored again by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.change.3

@@ -0,0 +1 @@
+Add the version of ``junifer`` to the MD5 of the features, as for the dependencies of the markers and steps, for reproducibility: features computed with different versions of ``junifer`` are different features, so the differences between features (e.g., between datasets) can only come from the data. Running again with a new version stores the features again, with the same name, next to the ones computed with the previous version by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.change.4

@@ -0,0 +1 @@
+Raise an error in the ``collect`` method of :class:`.HDF5FeatureStorage` and :class:`.SQLiteFeatureStorage` if a file to collect has more than one element, as ``collect`` only compacts the files of single elements by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.doc

@@ -0,0 +1 @@
+Add the "Implementation details" page with the elements, the fields of the metadata and the storage, and document the data types with different element keys when extending DataGrabbers by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.enh

@@ -0,0 +1 @@
+Store the features for the element with only the keys that the data type depends on (e.g., only ``subject`` for VBM data, when the elements also have ``task``), and skip the elements already stored by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/525.feature

@@ -0,0 +1 @@
+Add :meth:`.BaseDataGrabber.get_type_element_keys` to get the element keys that the data of a data type depends on by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/528.bugfix

@@ -0,0 +1 @@
+Fix :class:`.DataladHCP1200` listing only the subjects already installed (the subjects are sub-datasets) and not finding the BOLD data of the subjects not installed, by listing all the subjects without searching the files and building the paths of the BOLD data without wildcards by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/528.change

@@ -0,0 +1 @@
+Raise an error when installing the dataset of a :class:`.PatternDataladDataGrabber` if a pattern searches the files of a sub-dataset, which are only there once the sub-dataset is installed, instead of not finding the elements by `Fede Raimondo`_

junifer-0.0.8.dev318/docs/changes/newsfragments/528.doc

@@ -0,0 +1 @@
+Document that the DataGrabbers with files in datalad sub-datasets must list their elements and build the paths without searching the files by `Fede Raimondo`_

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/docs/extending/datagrabber.rst

@@ -399,6 +399,17 @@ This approach can be used directly from the YAML, like so:
      uri: "https://cerebra.fz-juelich.de/junifer/datalad-example-bids-ses.git"
      rootdir: example_bids_ses
 
+.. important::
+
+   The elements are found by searching the files of the dataset, but the files
+   of a datalad *sub-dataset* are only there once the sub-dataset is installed.
+   If the patterns search the files of a sub-dataset (e.g., a dataset with a
+   sub-dataset for each subject), :class:`.PatternDataladDataGrabber` raises
+   an error when installing the dataset. These DataGrabbers must implement
+   ``get_elements`` without searching the files (e.g., from the names of the
+   sub-datasets) and build the paths in the sub-datasets without wildcards,
+   as :class:`.DataladHCP1200` does.
+
 .. _extending_datagrabbers_path_expansion:
 
 Advanced: Using Unix-like path expansion directives
@@ -478,7 +489,13 @@ need to implement the following methods:
 
 - ``get_item``: to get a single item from the dataset.
 - ``get_elements``: to get the list of all elements present in the dataset
-- ``get_element_keys``: to get the keys of the elements in the dataset.
+- ``get_element_keys``: to get the keys of the elements in the dataset. They
+  also give the order of the values in each element, which must match the
+  parameters of ``get_item``.
+
+Optionally, if the data of some data types does not depend on all the keys of
+the element, we can also implement ``get_type_element_keys`` (see
+:ref:`extending_datagrabbers_type_keys`).
 
 .. note::
 
@@ -517,17 +534,15 @@ need to remember that for session *ses-03* there is no BOLD data.
    from itertools import product
 
 
-   def get_elements(self) -> list[str]:
+   def get_elements(self) -> list[tuple[str, str]]:
        subjects = ["sub-01", "sub-02", "sub-03"]
        sessions = ["ses-01", "ses-02"]
 
        # If we are not working on BOLD data, we can add "ses-03"
        if "BOLD" not in self.types:
            sessions.append("ses-03")
-       elements = []
-       for subject, element in product(subjects, sessions):
-           elements.append({"subject": subject, "session": session})
-       return elements
+       # The values of each element, in the order of the element keys
+       return list(product(subjects, sessions))
 
 
 And finally, we can implement the ``get_element_keys`` method. This method needs
@@ -540,11 +555,39 @@ method, in the same order.
    def get_element_keys(self) -> list[str]:
        return ["subject", "session"]
 
+.. _extending_datagrabbers_type_keys:
+
+Data types with different keys
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, the data of every data type depends on all the element keys. If
+the data of a data type only depends on some of them, the DataGrabber can say
+so by implementing ``get_type_element_keys``, which returns the element keys
+that the data of a data type depends on. For example, if the ``T1w`` image of
+our dataset was acquired once for each subject, instead of once for each
+session:
+
+.. code-block:: python
+
+   def get_type_element_keys(self, data_type: str) -> list[str]:
+       if data_type == "T1w":
+           return ["subject"]
+       return ["subject", "session"]
+
+Then, the ``T1w`` data (and the features computed from it) is stored once for
+each subject, instead of once for each session (see :ref:`internals`). The
+element keys must still be all the keys that the data types to grab depend on:
+if only ``T1w`` was grabbed, ``get_element_keys`` should return
+``["subject"]`` (and ``get_elements`` and ``get_item`` should only use the
+subject).
+
 
 So, to summarise, our DataGrabber will look like this:
 
 .. code-block:: python
 
+   from itertools import product
+
    from junifer.api.decorators import register_datagrabber
    from junifer.datagrabber import BaseDataGrabber
 
@@ -566,18 +609,15 @@ So, to summarise, our DataGrabber will look like this:
            }
            return out
 
-       def get_elements(self) -> list[str]:
+       def get_elements(self) -> list[tuple[str, str]]:
            subjects = ["sub-01", "sub-02", "sub-03"]
            sessions = ["ses-01", "ses-02"]
 
            # If we are not working on BOLD data, we can add "ses-03"
            if "BOLD" not in self.types:
                sessions.append("ses-03")
-           elements = []
-           for subject in subjects:
-               for session in sessions:
-                   elements.append({"subject": subject, "session": session})
-           return elements
+           # The values of each element, in the order of the element keys
+           return list(product(subjects, sessions))
 
        def get_element_keys(self) -> list[str]:
            return ["subject", "session"]

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/docs/index.rst

@@ -33,6 +33,7 @@ enabling others to extend it easily.
    help
    contribution
    maintaining
+   internals
    faq
    whats_new
 

junifer-0.0.8.dev318/docs/internals.rst

@@ -0,0 +1,169 @@
+.. include:: links.inc
+
+.. _internals:
+
+Implementation details
+======================
+
+This section describes some of the assumptions and technical details of
+``junifer``, for developers. See the docstrings of the mentioned functions and
+classes for more details.
+
+Elements
+--------
+
+* The keys of the elements of a :class:`.PatternDataGrabber`
+  (:meth:`.PatternDataGrabber.get_element_keys`) are the replacements used by
+  the patterns (including the nested ones) of the data types to grab, in the
+  order of ``replacements``.
+* :meth:`.PatternDataGrabber.get_elements` finds the elements of each pattern
+  in the files (translating the Unix-like path expansion directives of the
+  patterns to a regular expression) and joins the elements of the patterns on
+  their common keys (``_join_elements``), so an element is available if it has
+  the files of all the data types, regardless of their order.
+* The values of the replacements to grab (:meth:`.PatternDataGrabber.get_replacement_values`)
+  are given by ``replacements`` (as a dictionary) or by the fields linked to
+  the replacements in ``_REPLACEMENT_FIELDS``, but not both. They filter the
+  elements found in the files, and grabbing an element with other values
+  raises an error (``_check_element``, called by
+  ``BaseDataGrabber.__getitem__``).
+* :meth:`.BaseDataGrabber.select_elements` selects the elements to run: the
+  selectors must have the same number of values. If they have a value for
+  each element key, they are the elements (the dataset is not searched),
+  otherwise they select the elements of the dataset that have their values.
+* :class:`.MultipleDataGrabber` joins the elements of its DataGrabbers on
+  their common keys, and grabs an element with each DataGrabber using its
+  keys. Each data type is grabbed by one DataGrabber, except for the nested
+  data types grabbed by a DataGrabber without the main file of the data type,
+  which replace the ones of the other DataGrabber.
+
+Metadata
+--------
+
+The ``meta`` of each data type in the :ref:`data object <data_object>` is
+built by the steps of the pipeline. Each step adds an entry with its class and
+parameters (``UpdateMetaMixin.update_meta``) and its dependencies:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - Key
+     - Added by
+     - Content
+     - In the MD5
+   * - ``datagrabber``
+     - DataGrabber
+     - The class and parameters of the DataGrabber, e.g., ``types``,
+       ``patterns`` and ``replacements``. For the DataLad-based ones, also the
+       dataset ID and commit, and whether the dataset has local changes
+       (``datalad_id``, ``datalad_commit_id`` and ``datalad_dirty``), so
+       different versions of the dataset give different hashes. For :class:`.MultipleDataGrabber`, also
+       ``datagrabbers`` with the entries of its DataGrabbers.
+     - Yes, except ``datadir`` for the DataLad-based DataGrabbers (which is
+       removed from the stored metadata too)
+   * - ``element``
+     - DataGrabber
+     - The element being processed (see below).
+     - No
+   * - ``_element_keys``
+     - DataGrabber
+     - The keys of the element that the data depends on (see below).
+     - Yes
+   * - ``datareader``
+     - DataReader
+     - The class and parameters of the DataReader.
+     - Yes
+   * - ``preprocess``
+     - Preprocessors
+     - The list of the classes and parameters of the preprocessors that
+       processed the data, in order.
+     - Yes
+   * - ``type``
+     - Marker
+     - The data type, e.g., ``BOLD``.
+     - Yes
+   * - ``marker``
+     - Marker
+     - The class and parameters of the marker, with ``name`` as the name of
+       the marker followed by the name of the feature, e.g., ``mean_mean``.
+     - Yes
+   * - ``dependencies``
+     - All the steps
+     - The names of the dependencies of the steps (their ``_DEPENDENCIES``).
+       When storing, they are replaced by their versions, with the version of
+       ``junifer``, e.g., ``{'junifer': '0.0.8', 'nilearn': '0.14.1'}``.
+     - Yes (the versions)
+   * - ``name``
+     - Storage (``process_meta``)
+     - The name of the feature: the data type and the marker name, e.g.,
+       ``BOLD_mean_mean``.
+     - Yes
+
+The DataGrabber adds two keys about the element to the ``meta`` of each data
+type (``BaseDataGrabber.__getitem__``):
+
+* ``element``: the element being processed, with all its keys. It is the same
+  for all the data types of the element.
+* ``_element_keys``: the keys of the element that the data of the data type
+  depends on (:meth:`.BaseDataGrabber.get_type_element_keys`). 
+  For a :class:`.PatternDataGrabber`, the replacements in the patterns of the
+  data type.
+
+For example, with ``BOLD`` data for each subject and task, and ``VBM_GM`` data
+for each subject, the element ``(sub-01, rest)`` gives:
+
+.. code-block::
+
+   {'BOLD': {'meta': {'_element_keys': ['subject', 'task'],
+                      'element': {'subject': 'sub-01', 'task': 'rest'},
+                      ...},
+             ...},
+    'VBM_GM': {'meta': {'_element_keys': ['subject'],
+                        'element': {'subject': 'sub-01', 'task': 'rest'},
+                        ...},
+               ...}}
+
+When storing the output of a marker, ``process_meta`` (in ``junifer.storage.utils``):
+
+* Removes ``element`` from the metadata and returns the element of the data,
+  with only the keys in ``_element_keys`` (all the keys of ``element`` if it
+  is not set).
+* Computes the MD5 hash of the metadata, which identifies the feature. It
+  includes ``_element_keys`` and the versions of the dependencies of the
+  steps and of ``junifer``, but not ``element``: the data of all the elements
+  of a feature have the same hash, and the features computed with different
+  versions of ``junifer`` (or of the dependencies) have different hashes.
+  This is by design, for reproducibility: the differences between features
+  with the same hash can only come from the data, not from different
+  versions of the algorithms. A feature computed again with another version
+  is a new feature, with the same name, so it must be read by its MD5.
+
+Storage
+-------
+
+* The data is stored for the element of the data (the element with only the
+  keys in ``_element_keys``), so the data that does not depend on a key (e.g.,
+  the ``VBM_GM`` data for the tasks of a subject) is stored once.
+* :meth:`.BaseFeatureStorage.store` passes the element being processed to
+  the methods storing the metadata and the data (``store_metadata``,
+  ``store_vector``, etc.) as ``processed_element``, besides the element of the
+  data (``element``). The storages that write a file for each element (with
+  ``single_output=False``) name it after the element being processed, e.g.,
+  ``element_sub-01_rest_out.hdf5``, so that the processes running different
+  elements in parallel never write to the same file. The storages without
+  files (e.g., databases) can ignore it, but their methods must accept it.
+* The data already stored for an element is not stored again (the storages
+  skip the existing elements, :class:`.SQLiteFeatureStorage` with
+  ``upsert="ignore"`` by default).
+* ``collect`` only collects the files of single elements (one element for
+  each feature, as written for each element with ``single_output=False``),
+  and raises an error otherwise. The data of an element can be in the files
+  of several elements being processed (e.g., the ``VBM_GM`` data of
+  ``sub-01`` in the files of each of its tasks), and it is only collected
+  once.
+* :meth:`.HDF5FeatureStorage.collect` works in two passes: the first one reads
+  the metadata and the elements of each file, to know which files to collect
+  for each feature (one for each element), and the second one collects the
+  data of each feature from these files, in chunks. Each file is read once in
+  the first pass, and once for each of its features in the second one.

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/docs/understanding/storage.rst

@@ -30,6 +30,14 @@ your own ``Storage`` by providing a concrete implementation of
 :class:`.BaseFeatureStorage` or open an issue on `junifer Github`_ and we can
 help you out.
 
+The data of each element is stored only once: running the same elements again
+with the same configuration (e.g., after trying a few elements and then running
+all of them), or collecting the files of the elements again, does not change
+the data already stored, which is not stored again. The data that does not
+depend on all the keys of the element (e.g., the ``VBM_GM`` data of a subject,
+for elements with the subject and the task) is stored once for its keys (see
+:ref:`internals`).
+
 .. _storage_types:
 
 Storage Types

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/junifer/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.0.8.dev311'
-__version_tuple__ = version_tuple = (0, 0, 8, 'dev311')
+__version__ = version = '0.0.8.dev318'
+__version_tuple__ = version_tuple = (0, 0, 8, 'dev318')
 
-__commit_id__ = commit_id = 'gf37e90687'
+__commit_id__ = commit_id = 'gd4a9dcd3a'

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/junifer/api/tests/test_functions.py

@@ -9,23 +9,30 @@ import logging
 import sys
 import tempfile
 from contextlib import AbstractContextManager, nullcontext
+from copy import deepcopy
 from pathlib import Path
-from typing import Any
+from typing import Any, ClassVar
 
+import nibabel as nib
+import numpy as np
 import pytest
 from nibabel.filebasedimages import ImageFileError
 from ruamel.yaml import YAML
 
 import junifer.testing.registry  # noqa: F401
 from junifer.api import collect, list_elements, parse_yaml, queue, reset, run
-from junifer.datagrabber import PatternDataGrabber
+from junifer.api.decorators import register_marker
+from junifer.datagrabber import DataType, PatternDataGrabber
 from junifer.datagrabber.base import BaseDataGrabber
+from junifer.markers import BaseMarker
 from junifer.pipeline import (
     MarkerCollection,
     PipelineComponentRegistry,
     WorkDirManager,
 )
-from junifer.typing import Elements
+from junifer.storage import StorageType
+from junifer.storage.base import BaseFeatureStorage
+from junifer.typing import Dependencies, Elements, MarkerInOutMappings
 
 
 pytestmark = pytest.mark.external
@@ -1460,3 +1467,119 @@ def _run_selectors(
         elements=elements,
     )
     return n_searches
+
+
+@register_marker
+class _ImageMean(BaseMarker):
+    """Marker storing the mean of an image, for BOLD and VBM_GM."""
+
+    _DEPENDENCIES: ClassVar[Dependencies] = {"numpy"}
+    _MARKER_INOUT_MAPPINGS: ClassVar[MarkerInOutMappings] = {
+        DataType.BOLD: {"mean": StorageType.Vector},
+        DataType.VBM_GM: {"mean": StorageType.Vector},
+    }
+
+    def compute(self, input: dict, extra_input: dict | None = None) -> dict:
+        data = np.asarray(input["data"].dataobj, dtype=float)
+        return {
+            "mean": {"data": np.array([[data.mean()]]), "col_names": ["mean"]}
+        }
+
+
+@pytest.mark.parametrize("single_output", [True, False])
+@pytest.mark.parametrize(
+    "storage_kind", ["HDF5FeatureStorage", "SQLiteFeatureStorage"]
+)
+def test_run_data_types_element_keys(
+    tmp_path: Path, storage_kind: str, single_output: bool
+) -> None:
+    """Test run function storing data types with different element keys.
+
+    The BOLD data depends on the subject and the task, and the VBM data only
+    on the subject, so it is stored once for each subject. The elements are
+    run twice, as when trying a few elements and then running all of them.
+
+    Parameters
+    ----------
+    tmp_path : pathlib.Path
+        The path to the test directory.
+    storage_kind : str
+        The parametrized storage.
+    single_output : bool
+        The parametrized storage in a single file.
+
+    """
+    datadir = tmp_path / "data"
+    # BOLD: 10 * subject + task (1: rest, 2: movie); VBM: subject
+    for i, subject in enumerate(("sub-01", "sub-02"), start=1):
+        (datadir / subject).mkdir(parents=True)
+        nib.save(
+            nib.Nifti1Image(np.full((2, 2, 2), float(i)), np.eye(4)),
+            datadir / subject / f"{subject}_GM.nii",
+        )
+        for j, task in enumerate(("rest", "movie"), start=1):
+            nib.save(
+                nib.Nifti1Image(
+                    np.full((2, 2, 2, 3), 10.0 * i + j), np.eye(4)
+                ),
+                datadir / subject / f"{subject}_task-{task}_bold.nii",
+            )
+    datagrabber = {
+        "kind": "PatternDataGrabber",
+        "datadir": str(datadir),
+        "types": ["BOLD", "VBM_GM"],
+        "patterns": {
+            "BOLD": {
+                "pattern": "{subject}/{subject}_task-{task}_bold.nii",
+                "space": "native",
+            },
+            "VBM_GM": {
+                "pattern": "{subject}/{subject}_GM.nii",
+                "space": "native",
+            },
+        },
+        "replacements": ["subject", "task"],
+    }
+    uri = tmp_path / "out" / "features.db"
+    storage = {
+        "kind": storage_kind,
+        "uri": str(uri),
+        "single_output": single_output,
+    }
+    # A few elements, and then all of them
+    for elements in ([("sub-01", "rest")], None):
+        run(
+            workdir=tmp_path / "workdir",
+            datagrabber=deepcopy(datagrabber),
+            markers=[{"kind": "_ImageMean", "name": "mean"}],
+            storage=deepcopy(storage),
+            elements=elements,
+        )
+        if not single_output:
+            collect(deepcopy(storage))
+
+    reader = PipelineComponentRegistry().build_component_instance(
+        step="storage",
+        name=storage_kind,
+        baseclass=BaseFeatureStorage,
+        init_params={"uri": uri},
+    )
+    features = {
+        meta["name"]: md5 for md5, meta in reader.list_features().items()
+    }
+    bold = reader.read_df(feature_md5=features["BOLD_mean_mean"])
+    vbm = reader.read_df(feature_md5=features["VBM_GM_mean_mean"])
+    assert sorted(bold.index) == [
+        ("sub-01", "movie"),
+        ("sub-01", "rest"),
+        ("sub-02", "movie"),
+        ("sub-02", "rest"),
+    ]
+    assert sorted(bold["mean"]) == [11, 12, 21, 22]
+    # Once for each subject
+    assert vbm.index.names == ["subject"]
+    assert sorted(vbm.index.get_level_values("subject")) == [
+        "sub-01",
+        "sub-02",
+    ]
+    assert sorted(vbm["mean"]) == [1, 2]

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/junifer/datagrabber/base.py

@@ -183,18 +183,39 @@ class BaseDataGrabber(BaseModel, ABC, UpdateMetaMixin):
         self._check_element(named_element)
         # Fetch element
         out = self.get_item(**named_element)
-        # Update metadata
-        for _, t_val in out.items():
+        # Update metadata: the element being processed and the keys of the
+        # element that the data of each data type depends on
+        for t_type, t_val in out.items():
             self.update_meta(t_val, "datagrabber")
+            element_keys = self.get_type_element_keys(t_type)
             # Conditional for list dtype vals like Warp
-            if isinstance(t_val, list):
-                for entry in t_val:
-                    entry["meta"]["element"] = named_element
-            else:
-                t_val["meta"]["element"] = named_element
+            for entry in t_val if isinstance(t_val, list) else [t_val]:
+                entry["meta"]["element"] = named_element
+                entry["meta"]["_element_keys"] = element_keys
 
         return out
 
+    def get_type_element_keys(self, data_type: str) -> list[str]:
+        """Get the element keys that the data of a data type depends on.
+
+        The element keys (see :meth:`.get_element_keys`) are the keys that
+        the data types to grab depend on.
+
+        Parameters
+        ----------
+        data_type : str
+            The data type.
+
+        Returns
+        -------
+        list of str
+            The element keys, all of them unless a subclass knows which ones
+            the data type depends on (e.g., only the subject for data that is
+            the same for all the tasks of a subject).
+
+        """
+        return self.get_element_keys()
+
     def _check_element(self, element: dict) -> None:
         """Check the element can be grabbed.
 

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/junifer/datagrabber/hcp1200/datalad_hcp1200.py

@@ -5,12 +5,14 @@
 #          Synchon Mandal <s.mandal@fz-juelich.de>
 # License: AGPL
 
+from itertools import product
 from pathlib import Path
 from typing import Annotated, Literal
 
 from pydantic import AnyUrl, BeforeValidator
 
 from ...api.decorators import register_datagrabber
+from ...typing import Elements
 from ...utils import ensure_list
 from ..base import DataType
 from ..datalad_base import DataladDataGrabber
@@ -67,3 +69,29 @@ class DataladHCP1200(DataladDataGrabber, HCP1200):
     def skip_file_check(self) -> bool:
         """Skip file check existence."""
         return True
+
+    def get_elements(self) -> Elements:
+        """Implement fetching list of elements in the dataset.
+
+        The subjects are sub-datasets, whose files are only there once they
+        are installed, so the files are not searched: the elements are all
+        the subjects (the directories of the dataset) with the tasks and
+        phase encodings to grab, even if some of them have no data.
+
+        Returns
+        -------
+        list
+            The list of elements that can be grabbed in the dataset.
+
+        """
+        subjects = sorted(
+            x.name
+            for x in self.fulldir.iterdir()
+            if x.is_dir() and not x.name.startswith(".")
+        )
+        values = {"subject": subjects, **self.get_replacement_values()}
+        element_keys = self.get_element_keys()
+        elements = list(product(*(values[k] for k in element_keys)))
+        if len(element_keys) == 1:
+            return [element[0] for element in elements]
+        return elements

{junifer-0.0.8.dev311 → junifer-0.0.8.dev318}/junifer/datagrabber/hcp1200/hcp1200.py

@@ -162,3 +162,29 @@ class HCP1200(PatternDataGrabber):
             "pattern"
         ].replace("{suffix}", suffix)
         super().validate_datagrabber_params()
+
+    def _replace_patterns_glob(self, element: dict, pattern: str) -> str:
+        """Replace ``pattern`` with the ``element`` so it can be globbed.
+
+        The BOLD data of the resting-state tasks is in ``rfMRI_*`` and of the
+        other tasks in ``tfMRI_*``, so the path of an element is built
+        without searching the files, which are in sub-datasets in
+        :class:`.DataladHCP1200`.
+
+        Parameters
+        ----------
+        element : dict
+            The element to be used in the replacement.
+        pattern : str
+            The pattern to be replaced.
+
+        Returns
+        -------
+        str
+            The pattern with the element replaced.
+
+        """
+        if "task" in element:
+            prefix = "rfMRI" if "REST" in element["task"] else "tfMRI"
+            pattern = pattern.replace("?fMRI_", f"{prefix}_")
+        return super()._replace_patterns_glob(element, pattern)