junifer 0.0.8.dev303__tar.gz → 0.0.8.dev311__tar.gz

{junifer-0.0.8.dev303 → junifer-0.0.8.dev311}/.gitmodules

@@ -1,6 +1,3 @@
-[submodule "junifer/external/h5io"]
-	path = junifer/external/h5io
-	url = https://github.com/juaml/h5io
 [submodule "junifer/external/BrainPrint"]
 	path = junifer/external/BrainPrint
 	url = https://github.com/juaml/BrainPrint.git

{junifer-0.0.8.dev303/junifer.egg-info → junifer-0.0.8.dev311}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: junifer
-Version: 0.0.8.dev303
+Version: 0.0.8.dev311
 Summary: JUelich NeuroImaging FEature extractoR
 Author-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
 Maintainer-email: Fede Raimondo <f.raimondo@fz-juelich.de>, Synchon Mandal <s.mandal@fz-juelich.de>
@@ -37,6 +37,7 @@ Requires-Dist: nilearn<0.15,>=0.14.0
 Requires-Dist: sqlalchemy<2.2.0,>=2.0.25
 Requires-Dist: ruamel.yaml<0.20,>=0.17
 Requires-Dist: h5py>=3.10
+Requires-Dist: h5io<0.3,>=0.2.5
 Requires-Dist: tqdm<4.71.0,>=4.66.1
 Requires-Dist: templateflow<26.0.0,>=23.0.0
 Requires-Dist: lapy<2.0.0,>=1.0.0

junifer-0.0.8.dev311/docs/changes/newsfragments/504.bugfix

@@ -0,0 +1 @@
+Fix the data collected by :class:`.HDF5FeatureStorage` being stored as ``float32`` when ``force_float32=False`` and the ``h5py`` deprecation warning when collecting by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/504.misc

@@ -0,0 +1 @@
+Use original ``h5io`` instead of junifer's fork, keeping support for reading the features collected with the fork by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.bugfix

@@ -0,0 +1 @@
+Fix partial element selectors (e.g., only the subject) always being reported as invalid by :func:`.run`, after computing their elements by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.bugfix.1

@@ -0,0 +1 @@
+Fix a relative ``datadir`` of the DataGrabber in the YAML being relative to the current directory (e.g., the job directory when queueing) instead of the YAML file by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.change

@@ -0,0 +1 @@
+Use only the replacements of the data types to grab as element keys in :class:`.PatternDataGrabber` (e.g., only ``subject`` with VBM data), so the data that does not depend on, e.g., the task is not processed once for each task by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.change.1

@@ -0,0 +1 @@
+Find the elements of :class:`.DataladAOMICPIOP1`, :class:`.DataladAOMICPIOP2`, :class:`.HCP1200` and :class:`.DMCC13Benchmark` in the dataset, so only the elements with data are listed, and return absolute paths in :class:`.PatternDataGrabber` by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.change.2

@@ -0,0 +1 @@
+Require the element selectors to have the same number of values, and each data type to be grabbed by only one of the DataGrabbers of :class:`.MultipleDataGrabber` by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.doc

@@ -0,0 +1 @@
+Document the elements of the DataGrabbers, combining datasets with :class:`.MultipleDataGrabber`, complete and partial elements, and restricting the values of the replacements by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.enh

@@ -0,0 +1 @@
+Run complete elements (with a value for each element key, e.g., in each queued job) without searching the dataset for its elements by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.feature

@@ -0,0 +1 @@
+Allow restricting the values of the replacements to grab in :class:`.PatternDataGrabber`, with ``replacements`` as a dictionary or with fields linked with ``_REPLACEMENT_FIELDS`` (e.g., ``tasks`` in the built-in DataGrabbers), and using the ``*`` and ``?`` wildcards in the patterns to find the elements by `Fede Raimondo`_

junifer-0.0.8.dev311/docs/changes/newsfragments/524.feature.1

@@ -0,0 +1 @@
+Allow :class:`.MultipleDataGrabber` to combine DataGrabbers with different element keys, joining their elements on the common keys, and to replace nested data types (e.g., the confounds of ``BOLD``) with a DataGrabber that only grabs them by `Fede Raimondo`_

{junifer-0.0.8.dev303 → junifer-0.0.8.dev311}/docs/extending/datagrabber.rst

@@ -48,6 +48,13 @@ acquired for each task, but once in the entire session. So in this case, the
 ``T1w`` image for the element (``sub001``, ``ses1``, ``rest``) will be the same
 as the ``T1w`` image for the element (``sub001``, ``ses1``, ``stroop``).
 
+When the DataGrabber only grabs some of the data types, the elements only have
+the items that these data types need. For example, if only the ``T1w`` image is
+grabbed, the elements are (``subject``, ``session``), e.g., (``sub001``,
+``ses1``), so that the same ``T1w`` image is not processed once for each task.
+For the :class:`.PatternDataGrabber` (see below), the items that the data types
+need are the replacements in their patterns.
+
 We will now continue this section using as an example, a dataset in BIDS format
 in which 9 subjects (``sub-01`` to ``sub-09``) were scanned each during 3
 sessions (``ses-01``, ``ses-02``, ``ses-03``) and each session included a
@@ -200,6 +207,102 @@ to set the ``datadir``.
          kind: ExampleBIDSDataGrabber
          datadir: /data/project/test/data
 
+The DataGrabber finds the elements by searching the dataset for the files that
+match the pattern of each data type. An element is only available if it has the
+files of all the data types to grab. In our BIDS example, ``ses-03`` only has
+anatomical data, so the elements with ``ses-03`` are only available when only
+the ``T1w`` data type is grabbed.
+
+
+.. _extending_datagrabbers_replacement_values:
+
+Restricting the values of the replacements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, the DataGrabber grabs all the values of the replacements found in
+the dataset. To only grab some of them, ``replacements`` can be a dictionary
+with the values to grab for each replacement (or ``None`` to grab all of them).
+For example, to only grab the first two sessions of our BIDS example with the
+:class:`.PatternDataGrabber` in the YAML file:
+
+.. code-block:: yaml
+
+      datagrabber:
+         kind: PatternDataGrabber
+         datadir: /data/project/test/data
+         types:
+            - T1w
+            - BOLD
+         patterns:
+            T1w:
+               pattern: "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
+               space: native
+            BOLD:
+               pattern: "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz"
+               space: MNI152NLin6Asym
+         replacements:
+            subject: null
+            session:
+               - ses-01
+               - ses-02
+
+The elements with other values are not listed, and grabbing them raises an
+error. The values are ignored if the data types to grab do not use the
+replacement, e.g., the ``session`` values with a data type that is the same for
+all the sessions.
+
+A DataGrabber can also take the values to grab from one of its fields, e.g.,
+``sessions``, by linking the field to the replacement with
+``_REPLACEMENT_FIELDS``. The values of the field can be strings or enumerations
+(a :class:`enum.StrEnum` is recommended, so its members are also the strings),
+and the replacement cannot have values in ``replacements`` too:
+
+.. code-block:: python
+
+    from enum import StrEnum
+    from typing import ClassVar
+
+    from junifer.api.decorators import register_datagrabber
+    from junifer.datagrabber import DataType, PatternDataGrabber
+    from junifer.typing import DataGrabberPatterns
+
+
+    class Sessions(StrEnum):
+        ses_01 = "ses-01"
+        ses_02 = "ses-02"
+        ses_03 = "ses-03"
+
+
+    @register_datagrabber
+    class ExampleBIDSDataGrabber(PatternDataGrabber):
+
+        types: list[DataType] = [DataType.T1w, DataType.BOLD]
+        patterns: DataGrabberPatterns = {
+            "T1w": {
+                "pattern": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
+                "space": "native",
+            },
+            "BOLD": {
+                "pattern": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
+                "space": "MNI152NLin6Asym",
+            },
+        }
+        replacements: list[str] = ["subject", "session"]
+        sessions: list[Sessions] = list(Sessions)
+        # Only grab the sessions in the `sessions` field
+        _REPLACEMENT_FIELDS: ClassVar[dict[str, str]] = {"session": "sessions"}
+
+Then, the sessions to grab can be set in the YAML file:
+
+.. code-block:: yaml
+
+      datagrabber:
+         kind: ExampleBIDSDataGrabber
+         datadir: /data/project/test/data
+         sessions:
+            - ses-01
+            - ses-02
+
 
 Optional: Using datalad
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -296,20 +399,38 @@ This approach can be used directly from the YAML, like so:
      uri: "https://cerebra.fz-juelich.de/junifer/datalad-example-bids-ses.git"
      rootdir: example_bids_ses
 
+.. _extending_datagrabbers_path_expansion:
+
 Advanced: Using Unix-like path expansion directives
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 It is also possible to use some advanced Unix-like path expansion tricks to
 define our patterns.
 
-A very common thing would be to use ``*`` to match any number of
-characters but we cannot use it right after a replacement like:
+A very common thing would be to use ``*`` to match any number of characters
+or ``?`` to match a single character, both to find the elements and to grab
+their data. For example, if the acquisition of the BOLD images depends on the
+task (e.g., ``sub-01_task-rest_acq-mb3_bold.nii.gz`` and
+``sub-01_task-stroop_acq-seq_bold.nii.gz``), the pattern can match any
+acquisition, so the task values are ``rest`` and ``stroop``:
 
 .. code-block:: python
 
-    "derivatives/freesurfer/{subject}*"
+    "{subject}/func/{subject}_task-{task}_acq-*_bold.nii.gz"
+
+Similarly, ``?`` can match the first character of ``rfMRI`` (resting-state)
+and ``tfMRI`` (task) in the HCP dataset:
 
-or if there are multiple files or no files which can be globbed.
+.. code-block:: python
+
+    "{subject}/MNINonLinear/Results/?fMRI_{task}_{phase_encoding}/?fMRI_{task}_{phase_encoding}.nii.gz"
+
+The pattern must match exactly one file for each element, and we cannot use
+``*`` right after a replacement like:
+
+.. code-block:: python
+
+    "derivatives/freesurfer/{subject}*"
 
 We can also use ``[]`` and ``[!]`` to glob certain tricky files like with the
 case of FreeSurfer derivatives. The file structure seen in a typical

junifer-0.0.8.dev311/docs/understanding/datagrabber.rst

@@ -0,0 +1,302 @@
+.. include:: ../links.inc
+
+.. _datagrabber:
+
+Data Grabber
+============
+
+Description
+-----------
+
+The ``DataGrabber`` is an object that can provide an interface to datasets you
+want to work with in ``junifer``. Every concrete implementation of a DataGrabber
+is aware of a particular dataset's structure and thus allows you to fetch
+specific elements of interest from the dataset. It adds the ``path`` key to each
+:ref:`data type <data_types>` in the :ref:`Data object <data_object>`.
+
+DataGrabbers are intended to be used as context managers. When used within a
+context, a DataGrabber takes care of any pre and post steps for interacting with
+the dataset, for example, downloading and cleaning up. As the interface
+is consistent, you always use the same procedure to interact with the DataGrabber.
+
+For example, a concrete implementation of :class:`.DataladDataGrabber` can
+provide ``junifer`` with data from a Datalad dataset. Of course, DataGrabbers are
+not only meant to work with Datalad datasets but any dataset.
+
+If you are interested in using already provided DataGrabbers, please go to
+:doc:`../builtin`. And, if you want to implement your own DataGrabber, you need
+to provide concrete implementations of abstract base classes already provided.
+
+.. _datagrabber_elements:
+
+Elements
+--------
+
+A DataGrabber gives the data of a dataset in small units called *elements*. An
+element is the smallest unit of data that can be processed: it has one file
+for each :ref:`data type <data_types>` (e.g., one ``T1w`` image and one
+``BOLD`` image). Each element is identified by the values of its *keys*, e.g.,
+the element ``("sub-01", "rest")`` has the value ``sub-01`` for the key
+``subject`` and ``rest`` for the key ``task``.
+
+To see how this works, we will use this small dataset with two subjects, a
+``T1w`` image for each subject and a ``BOLD`` image for each subject and task
+(``sub-02`` has no ``movie`` task):
+
+.. code-block:: text
+
+    sub-01/anat/sub-01_T1w.nii.gz
+    sub-01/func/sub-01_task-rest_bold.nii.gz
+    sub-01/func/sub-01_task-movie_bold.nii.gz
+    sub-02/anat/sub-02_T1w.nii.gz
+    sub-02/func/sub-02_task-rest_bold.nii.gz
+
+A :class:`.PatternDataGrabber` describes the files of each data type with a
+*pattern*, where the parts that change between the files (the *replacements*)
+are written between braces:
+
+.. code-block:: yaml
+
+  datagrabber:
+    kind: PatternDataGrabber
+    datadir: /data/example
+    types:
+      - T1w
+      - BOLD
+    patterns:
+      T1w:
+        pattern: "{subject}/anat/{subject}_T1w.nii.gz"
+        space: native
+      BOLD:
+        pattern: "{subject}/func/{subject}_task-{task}_bold.nii.gz"
+        space: native
+    replacements:
+      - subject
+      - task
+
+Which data types are grabbed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The DataGrabber finds the elements by searching the dataset for the files that
+match the patterns. The keys of the elements are the replacements in the
+patterns of the data types to grab (``types``), and an element is only
+available if it has a file for each of these data types:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - ``types``
+     - Keys
+     - Elements
+   * - ``T1w``
+     - ``subject``
+     - ``sub-01``, ``sub-02``
+   * - ``BOLD``
+     - ``subject``, ``task``
+     - ``(sub-01, movie)``, ``(sub-01, rest)``, ``(sub-02, rest)``
+   * - ``T1w``, ``BOLD``
+     - ``subject``, ``task``
+     - ``(sub-01, movie)``, ``(sub-01, rest)``, ``(sub-02, rest)``
+
+With only ``T1w``, the elements do not have the ``task`` key, as the ``T1w``
+image does not depend on the task: each ``T1w`` image is processed once. With
+``BOLD``, the elements need the task, so the ``T1w`` image of ``sub-01`` is part
+of two elements, ``(sub-01, movie)`` and ``(sub-01, rest)``.
+
+Which values are grabbed
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, all the values found in the dataset are grabbed. They can be
+restricted with ``replacements``, giving the values to grab for each
+replacement (or ``null`` to grab all of them). For example, to only grab the
+``rest`` task:
+
+.. code-block:: yaml
+
+    replacements:
+      subject: null
+      task:
+        - rest
+
+The :doc:`built-in DataGrabbers <../builtin>` have parameters for this instead,
+e.g., ``tasks`` in :class:`.DataladAOMICPIOP1`. The values only restrict the
+elements of the data types that use the replacement:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - ``types``
+     - Values to grab
+     - Elements
+   * - ``BOLD``
+     - ``task``: ``rest``
+     - ``(sub-01, rest)``, ``(sub-02, rest)``
+   * - ``BOLD``
+     - ``subject``: ``sub-02``
+     - ``(sub-02, rest)``
+   * - ``T1w``
+     - ``task``: ``rest``
+     - ``sub-01``, ``sub-02`` (the ``T1w`` image does not depend on the task)
+
+The elements with other values are not listed, and asking for them (e.g.,
+``(sub-01, movie)`` when only ``rest`` is grabbed) raises an error. See
+:ref:`extending_datagrabbers_replacement_values` for more details, and
+:ref:`extending_datagrabbers_path_expansion` for patterns with ``*``, e.g., to
+match files whose names change with the task.
+
+Which elements are processed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``list-elements`` command lists the elements of the dataset. By default,
+the ``run`` and ``queue`` commands process all of them, but the ``--element``
+option can select some of them, with the values of the keys separated by
+``,``. With ``BOLD``:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - ``--element``
+     - Elements processed
+   * - ``sub-01,rest``
+     - ``(sub-01, rest)``: a *complete* element (a value for each key, in
+       order), the dataset is not searched
+   * - ``sub-01``
+     - ``(sub-01, movie)``, ``(sub-01, rest)``: a *partial* element selects all
+       the elements with its values
+   * - ``rest``
+     - ``(sub-01, rest)``, ``(sub-02, rest)``
+   * - ``sub-03``
+     - None: an error is raised, as no element matches it
+
+See :ref:`running_elements` for more details.
+
+.. _datagrabber_multiple:
+
+Combining datasets
+------------------
+
+The :class:`.MultipleDataGrabber` combines several DataGrabbers, e.g., to use
+data from different datasets. For example, we can take the ``BOLD`` and
+``T1w`` images from a dataset with the fMRIPrep derivatives, the ``VBM_GM``
+images from a dataset with the CAT12 derivatives, and the confounds of the
+``BOLD`` images from a third dataset:
+
+.. code-block:: yaml
+
+  datagrabber:
+    kind: MultipleDataGrabber
+    datagrabbers:
+      # BOLD and T1w from the fMRIPrep derivatives
+      - kind: DataladAOMICPIOP1
+        types:
+          - BOLD
+          - T1w
+        tasks:
+          - restingstate
+      # VBM from the CAT12 derivatives
+      - kind: PatternDataladDataGrabber
+        uri: https://example.org/aomic-piop1-cat12
+        types:
+          - VBM_GM
+        patterns:
+          VBM_GM:
+            pattern: "{subject}/mri/mwp1{subject}_T1w.nii"
+            space: MNI152NLin2009cAsym
+        replacements:
+          - subject
+      # Only the confounds of the BOLD images
+      - kind: PatternDataladDataGrabber
+        uri: https://example.org/aomic-piop1-confounds
+        types:
+          - BOLD
+        patterns:
+          BOLD:
+            confounds:
+              pattern: "{subject}/func/{subject}_task-{task}_acq-*_desc-confounds.tsv"
+              format: fmriprep
+        replacements:
+          - subject
+          - task
+        partial_pattern_ok: true
+
+The DataGrabbers are combined with these rules:
+
+* **Each data type is grabbed by one DataGrabber**, which is chosen with the
+  ``types`` of each DataGrabber. Here, the first one grabs ``BOLD`` and
+  ``T1w``, and the second one ``VBM_GM``. If the first one also grabbed
+  ``VBM_GM`` (e.g., without setting its ``types``), an error would be raised,
+  asking to remove it from the ``types`` of one of them.
+* **A DataGrabber that only grabs the nested data types of a data type**
+  (without the main file of the data type) replaces these nested data types
+  of the other DataGrabber. Here, the third one only grabs the ``confounds``
+  of ``BOLD``, so they replace the ones of the first DataGrabber, regardless
+  of the order of the DataGrabbers.
+* **The elements are the ones available in all the DataGrabbers.** The
+  DataGrabbers can have different keys: the element keys are the ones of the
+  first DataGrabber, followed by the other keys of the other DataGrabbers, and
+  each DataGrabber grabs an element with its own keys. The keys with the same
+  name must have the same values in all the datasets (e.g., the same subject
+  names).
+
+For the element ``(sub-0001, restingstate)``:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - DataGrabber
+     - Keys
+     - Grabs
+   * - First (fMRIPrep)
+     - ``subject``, ``task``
+     - ``(sub-0001, restingstate)``: the ``BOLD`` image (with its mask) and
+       the ``T1w`` image
+   * - Second (CAT12)
+     - ``subject``
+     - ``sub-0001``: the ``VBM_GM`` image
+   * - Third (confounds)
+     - ``subject``, ``task``
+     - ``(sub-0001, restingstate)``: the confounds of the ``BOLD`` image
+
+Base Classes
+------------
+
+In this section, we showcase different abstract and concrete base classes you
+might want to use to implement your own DataGrabber.
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * - Name
+     - Description
+   * - :class:`.BaseDataGrabber`
+     - | The abstract base class providing you an interface to implement your
+       | own DataGrabber. You should try to avoid using this directly and
+       | instead use :class:`.PatternDataGrabber` or
+       | :class:`.DataladDataGrabber`. To build your own custom *low-level*
+       | DataGrabber, you need to override the ``get_element_keys``,
+       | ``get_elements`` and ``get_item`` methods, and most of the time you
+       | should also override other existing methods like ``__enter__`` and
+       | ``__exit__``.
+   * - :class:`.PatternDataGrabber`
+     - | It implements functionality to help you define the pattern of the
+       | dataset you want to get. For example, you know that T1w images are
+       | found in a directory following the pattern:
+       | ``{subject}/anat/{subject}_T1w.nii.gz`` inside of the dataset. Now you
+       | can provide this to the :class:`.PatternDataGrabber` and it will be
+       | able to get the file.
+   * - :class:`.DataladDataGrabber`
+     - | It implements functionality to deal with Datalad datasets. Specifically,
+       | the ``__enter__`` and ``__exit__`` methods take care of cloning and
+       | removing the Datalad dataset.
+   * - :class:`.PatternDataladDataGrabber`
+     - | It is a combination of :class:`.PatternDataGrabber` and
+       | :class:`.DataladDataGrabber`. This is probably the class you are looking
+       | for when using Datalad datasets.
+   * - :class:`.MultipleDataGrabber`
+     - | It combines several DataGrabbers, e.g., to use data from different
+       | datasets (see :ref:`datagrabber_multiple`).

{junifer-0.0.8.dev303 → junifer-0.0.8.dev311}/docs/using/codeless.rst

@@ -96,6 +96,20 @@ task.
     kind: DataladAOMICPIOP1
     tasks: restingstate
 
+The ``types`` parameter restricts the :ref:`data types <data_types>` to grab,
+which also determines the elements to process. For example, with only the VBM
+data, which does not depend on the task, the elements only have the subject, so
+the VBM data of each subject is processed once (and ``tasks`` is not needed):
+
+.. code-block:: yaml
+
+  datagrabber:
+    kind: DataladAOMICPIOP1
+    types:
+      - VBM_GM
+
+See :ref:`datagrabber_elements` for more details on the elements.
+
 In the ``UCLA CNP VBM Testing dataset`` example, the section will look like this:
 
 .. code-block:: yaml

{junifer-0.0.8.dev303 → junifer-0.0.8.dev311}/docs/using/queueing.rst

@@ -185,5 +185,7 @@ The ``queue`` command accepts the following arguments:
   job submit files will be created but not submitted.
 * ``--overwrite``: Overwrite the job folder if it already exists. If not
   specified, the command will fail if the job folder already exists.
-* ``--element``: Queue only the specified element(s). If not specified, all
-  elements will be queued.
+* ``--element``: Queue only the specified element(s), complete or partial (see
+  :ref:`running_elements`). If not specified, all elements will be queued.
+  Each job runs one complete element, so it does not search the dataset for
+  its elements.

{junifer-0.0.8.dev303 → junifer-0.0.8.dev311}/docs/using/running.rst

@@ -30,7 +30,8 @@ The ``run`` command accepts the following additional arguments:
   run. This parameter can be specified multiple times to run multiple elements.
   If the *element* requires several parameters, they can be specified by
   separating them with ``,``. It also accepts a file (e.g., ``elements.txt``)
-  containing complete or partial element(s).
+  containing complete or partial element(s) (see
+  :ref:`running_elements`).
 
 Example of running two elements:
 --------------------------------
@@ -90,6 +91,40 @@ and then the ``elements.txt`` would be like so:
 
     sub-01
 
+.. _running_elements:
+
+Complete and partial elements
+-----------------------------
+
+An *element* has a value for each of its *keys*, e.g., ``subject`` and
+``session``. The keys depend on the :ref:`data types <data_types>` that the
+DataGrabber grabs: an element only has the keys that the data types need. For
+example, a DataGrabber with resting-state BOLD (one per subject and session) and
+VBM data (one per subject) has the keys ``subject`` and ``session`` when
+grabbing the BOLD data, but only ``subject`` when grabbing only the VBM data, so
+that the VBM data of a subject is not processed once for each session. The
+``list-elements`` command lists the elements with their values, in the order of
+the keys:
+
+.. code-block:: bash
+
+    junifer list-elements config.yaml
+
+The elements given with ``--element`` can be:
+
+* *Complete*, with a value for each key, in the order of the keys, e.g.,
+  ``sub-01,ses-01``. Only these elements are run, without searching the
+  dataset for its elements. This is the case for the jobs created by
+  :ref:`queue <queueing>`, which run one element each.
+* *Partial*, with fewer values, e.g., ``sub-01``. They run all the elements of
+  the dataset that have these values for any key, e.g., all the sessions of
+  ``sub-01``. This requires searching the dataset for its elements.
+
+All the elements given in a command must have the same number of values, so
+they are either all complete or all partial. A partial element that does not
+match any element of the dataset is reported before running any element, while
+a complete element without data fails when its data is grabbed.
+
 
 .. _collect:
 

{junifer-0.0.8.dev303 → junifer-0.0.8.dev311}/junifer/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.0.8.dev303'
-__version_tuple__ = version_tuple = (0, 0, 8, 'dev303')
+__version__ = version = '0.0.8.dev311'
+__version_tuple__ = version_tuple = (0, 0, 8, 'dev311')
 
-__commit_id__ = commit_id = 'g4a74bdf26'
+__commit_id__ = commit_id = 'gf37e90687'

{junifer-0.0.8.dev303 → junifer-0.0.8.dev311}/junifer/api/functions.py

@@ -261,25 +261,12 @@ def run(
     # Fit elements
     with datagrabber_object:
         if elements is not None:
-            # Keep track of valid selectors
-            valid_elements = []
-            for t_element in datagrabber_object.filter(elements):
-                valid_elements.append(t_element)
-                mc.fit(datagrabber_object[t_element])
-            # Compute invalid selectors
-            invalid_elements = set(elements) - set(valid_elements)
-            # Report if invalid selectors are found
-            if invalid_elements:
-                raise_error(
-                    msg=(
-                        "The following element selectors are invalid:\n"
-                        f"{invalid_elements}"
-                    ),
-                    klass=RuntimeError,
-                )
+            # Only get the elements of the dataset for partial selectors
+            to_fit = datagrabber_object.select_elements(elements)
         else:
-            for t_element in datagrabber_object:
-                mc.fit(datagrabber_object[t_element])
+            to_fit = datagrabber_object.get_elements()
+        for t_element in to_fit:
+            mc.fit(datagrabber_object[t_element])
 
 
 def collect(storage: dict, workdir: str | Path | dict | None = None) -> None:
@@ -628,6 +615,23 @@ def parse_yaml(filepath: str | Path) -> dict:  # noqa: C901
                     (filepath.parent / uri_path).resolve()
                 )
 
+    # Compute the absolute path of the data directory of the datagrabber (and
+    # of the datagrabbers of a MultipleDataGrabber) if it is relative; same
+    # motivation as above. Otherwise, it would be relative to the current
+    # working directory, which is the job directory when queueing.
+    if "datagrabber" in contents:
+        datagrabbers = [
+            contents["datagrabber"],
+            *contents["datagrabber"].get("datagrabbers", []),
+        ]
+        for datagrabber in datagrabbers:
+            if "datadir" in datagrabber:
+                datadir = Path(datagrabber["datadir"])
+                if not datadir.is_absolute():
+                    datagrabber["datadir"] = str(
+                        (filepath.parent / datadir).resolve()
+                    )
+
     # Allow relative path if queue env kind is venv; same motivation as above
     if "queue" in contents:
         if "env" in contents["queue"]: