xnatbidscli 2.0.0__tar.gz → 2.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/PKG-INFO +8 -6
  2. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/changelog.md +21 -1
  3. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/download.md +25 -8
  4. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/physioconvert.md +1 -3
  5. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/faq.md +9 -1
  6. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/installation.md +1 -3
  7. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/manual.md +1 -0
  8. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/pyproject.toml +9 -9
  9. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/archive.py +12 -4
  10. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/cli.py +11 -0
  11. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/download.py +233 -47
  12. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/physioconvert.py +3 -4
  13. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/test_plan.md +7 -0
  14. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.claude/CLAUDE.md +0 -0
  15. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.github/workflows/publish.yml +0 -0
  16. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.gitignore +0 -0
  17. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.python-version +0 -0
  18. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.readthedocs.yaml +0 -0
  19. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/AGENTS.md +0 -0
  20. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/CHANGELOG.md +0 -0
  21. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/LICENSE +0 -0
  22. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/README.md +0 -0
  23. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/assets/images/favicon.png +0 -0
  24. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/assets/images/logo.png +0 -0
  25. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/assets/js/faq.js +0 -0
  26. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/bidsmap.md +0 -0
  27. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/cubids.md +0 -0
  28. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/login.md +0 -0
  29. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/mriconfig.md +0 -0
  30. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/mriconvert.md +0 -0
  31. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/query.md +0 -0
  32. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/design.md +0 -0
  33. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/excel.md +0 -0
  34. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/index.md +0 -0
  35. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/quickstart.md +0 -0
  36. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/requirements.txt +0 -0
  37. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/assets/mriconvert_qc.json +0 -0
  38. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/assets/physioconvert_qc.json +0 -0
  39. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/__init__.py +0 -0
  40. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/bidsmap.py +0 -0
  41. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/cubids.py +0 -0
  42. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/login.py +0 -0
  43. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/mriconfig.py +0 -0
  44. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/mriconvert.py +0 -0
  45. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/query.py +0 -0
  46. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/sysinfo.py +0 -0
  47. {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/zensical.toml +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: xnatbidscli
3
- Version: 2.0.0
3
+ Version: 2.1.0
4
4
  Summary: CLI for logging into XNAT, downloading neuroimaging experiments, and converting them to BIDS.
5
5
  Project-URL: Homepage, https://github.com/nimh-dsst/xnat-bids-cli
6
6
  Project-URL: Documentation, https://xnatbidscli.readthedocs.io/
@@ -17,19 +17,21 @@ Classifier: Operating System :: OS Independent
17
17
  Classifier: Programming Language :: Python :: 3
18
18
  Classifier: Programming Language :: Python :: 3 :: Only
19
19
  Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Programming Language :: Python :: 3.14
20
23
  Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
21
24
  Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
22
25
  Requires-Python: >=3.11
23
- Requires-Dist: bioread>=3; python_version < '3.12'
26
+ Requires-Dist: bioread>=3
24
27
  Requires-Dist: cubids>=1.2.1
25
28
  Requires-Dist: dcm2bids>=3
26
29
  Requires-Dist: dcm2niix>=1.0.20260416
27
- Requires-Dist: nibabel<5.4,>=5.3; python_version < '3.12'
28
- Requires-Dist: nibabel>=5.4.2; python_version >= '3.12'
29
- Requires-Dist: numpy<2
30
+ Requires-Dist: nibabel>=5.4.2
30
31
  Requires-Dist: pandas<2.4,>=2
31
- Requires-Dist: phys2bids>=2.10; python_version < '3.12'
32
+ Requires-Dist: phys2bids>=2.11
32
33
  Requires-Dist: pydicom>=2.4
34
+ Requires-Dist: pymatreader>=1
33
35
  Requires-Dist: pyxnat>=1.6
34
36
  Requires-Dist: requests>=2
35
37
  Description-Content-Type: text/markdown
@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [2.1.0] - 2026-10-28
11
+
12
+ `physioconvert` now works on every supported Python version, using the new `phys2bids` 2.11 release. `download` no longer downloads over an experiment that is already on disk, and has a new `--overwrite` flag.
13
+
14
+ ### Added
15
+
16
+ - `download --overwrite` deletes an experiment's existing directory and downloads it again. With `-a/--archive`, it also replaces an existing archive after a `COMPLETE` download.
17
+ - `download --csv` skips with a `WARNING` every row of an experiment listed more than once (`DUPLICATE`, or `RENAME_CONFLICT` when its `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` values differ), and every experiment that would download into the same directory as another (`COLLISION`).
18
+ - `pymatreader` is now installed with `xnatbidscli`, so `physioconvert` can read MATLAB `.mat` files without an extra install.
19
+
20
+ ### Changed
21
+
22
+ - `download` skips an experiment whose directory already exists and is not empty, with a `WARNING` and a new `SKIPPED` status, instead of downloading over it. `-a/--archive` and `-d/--delete` are not run for it, with a louder `WARNING`. Skipped experiments do not change the exit code.
23
+ - `physioconvert` works on every supported Python version (3.11 and newer), not just 3.11. `xnatbidscli` now requires `phys2bids` 2.11 or newer, which no longer caps `numpy` below 1.24. `phys2bids` and `bioread` install on every Python version, and the `numpy<2` and `nibabel<5.4` pins are removed.
24
+
25
+ ### Fixed
26
+
27
+ - The `READER_MISSING` hint for `.mat` files now names `pymatreader`, the package `phys2bids` actually uses, instead of `scipy`.
28
+
10
29
  ## [2.0.0] - 2026-09-28
11
30
 
12
31
  The project is renamed from `xnatcli` to `xnatbidscli` and is published on PyPI. Downloads are faster and more reliable, and you can now rename subjects and experiments between `query` and `download`.
@@ -58,6 +77,7 @@ First release.
58
77
  - `xnatcli` subcommands: `login`, `query`, `download`, `mriconfig`, `mriconvert`, `physioconvert`, `bidsmap` and `cubids`.
59
78
  - A documentation site built with Zensical and hosted on Read the Docs.
60
79
 
61
- [Unreleased]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v2.0.0...HEAD
80
+ [Unreleased]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v2.1.0...HEAD
81
+ [2.1.0]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v2.0.0...v2.1.0
62
82
  [2.0.0]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v1.0.0...v2.0.0
63
83
  [1.0.0]: https://github.com/nimh-dsst/xnat-bids-cli/releases/tag/v1.0.0
@@ -4,12 +4,14 @@ Downloads every file belonging to one XNAT experiment (single-experiment mode, `
4
4
 
5
5
  1. Loads credentials from `~/.xnatbidscli/credentials.cfg`; if the file is missing or incomplete, exits with a message telling you to run `xnatbidscli login`.
6
6
  2. Connects to the stored server via PyXNAT.
7
- 3. For each experiment, walks `project → subject → experiment`, then issues zip requests against XNAT's REST API: one bulk request for all scans, and one request per session-level resource (XNAT has no bulk "all resources" export endpoint, unlike scans). With `--accession`, an extra server-wide lookup happens first, in this order: subject ID, experiment ID, study UID (XNAT's stored session `UID`), then subject and experiment labels together. IDs and UIDs are unique across the server. Labels are only unique within their parent, so a label that matches more than one subject or experiment exits with an error listing each match's XNAT ID, project and subject; rerun with one of those IDs. A UID only matches sessions where XNAT stored it.
8
- 4. Each zip is extracted directly into `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT/`, following XNAT's own scan/resource folder naming (not a custom path scheme), then discarded. If a resource resolves to a single file, XNAT sometimes streams that file directly instead of wrapping it in a zip; this is detected and the file is saved as-is rather than failing the experiment. If some resources download successfully and others fail, the successful ones are still kept and the experiment is reported as `FAILURE` with each failing resource named in the error.
7
+ 3. In `--csv` mode, every row is screened before anything downloads, and repeated or colliding experiments are skipped (see [Repeated experiments in the CSV](#repeated-experiments-in-the-csv)).
8
+ 4. For each experiment, if `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` (using any rename) already exists and is not empty, it counts as already downloaded. A `WARNING` is printed and the experiment is reported `SKIPPED` without contacting XNAT. `-a`/`-d` are not run for it either, and a second, louder `WARNING` says so. Pass `--overwrite` to delete the directory and download it again instead. The directory is deleted only after the experiment is found on XNAT. An empty directory does not count as downloaded. Because `-d` removes the directory, re-running after `-a -d` downloads those experiments again.
9
+ 5. For each experiment, walks `project → subject → experiment`, then issues zip requests against XNAT's REST API: one bulk request for all scans, and one request per session-level resource (XNAT has no bulk "all resources" export endpoint, unlike scans). With `--accession`, an extra server-wide lookup happens first, in this order: subject ID, experiment ID, study UID (XNAT's stored session `UID`), then subject and experiment labels together. IDs and UIDs are unique across the server. Labels are only unique within their parent, so a label that matches more than one subject or experiment exits with an error listing each match's XNAT ID, project and subject; rerun with one of those IDs. A UID only matches sessions where XNAT stored it.
10
+ 6. Each zip is extracted directly into `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT/`, following XNAT's own scan/resource folder naming (not a custom path scheme), then discarded. If a resource resolves to a single file, XNAT sometimes streams that file directly instead of wrapping it in a zip; this is detected and the file is saved as-is rather than failing the experiment. If some resources download successfully and others fail, the successful ones are still kept and the experiment is reported as `FAILURE` with each failing resource named in the error.
9
11
 
10
12
  `PROJECT` is the canonical XNAT project ID; `SUBJECT` and `EXPERIMENT` are the user-facing labels emitted by `xnatbidscli query`, unless overridden by that row's `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` values in `--csv` mode, or by `--rename-subject`/`--rename-experiment` in `-1`/`--accession` mode (see [Manual Interventions](../manual.md)) — XNAT is still queried using the original labels either way. In `--accession` mode, the resolved XNAT IDs (not labels) are used both to query XNAT and, absent a rename, as the on-disk `SUBJECT`/`EXPERIMENT` directory names.
11
13
 
12
- 5. With `-a/--archive`, after each experiment is downloaded, its `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory (using any rename override) is tar+gzipped to `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz`. An existing archive at that path is left untouched and reported as `SKIPPED`. With `-d/--delete` (requires `-a/--archive`), the `EXPERIMENT` directory is removed once its archive is `COMPLETE` or `SKIPPED`; the `SUBJECT` and then `PROJECT` parent directories are also removed if they become empty as a result.
14
+ 7. With `-a/--archive`, after each experiment is downloaded, its `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory (using any rename override) is tar+gzipped to `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz`. An existing archive at that path is left untouched and reported as `SKIPPED`. With `-d/--delete` (requires `-a/--archive`), the `EXPERIMENT` directory is removed once its archive is `COMPLETE` or `SKIPPED`; the `SUBJECT` and then `PROJECT` parent directories are also removed if they become empty as a result.
13
15
 
14
16
  ```bash
15
17
  # Single experiment
@@ -45,6 +47,7 @@ xnatbidscli download --csv PATH/TO/QUERY.csv -o OUTPUT_DIR -a -d
45
47
  | `-l`, `--log` | *Optional.* Write a per-experiment log CSV to `OUTPUT_DIR/log/download_<YYYYMMDD_HHMMSS>_log.csv` (local time, captured at run start). |
46
48
  | `-a`, `--archive` | *Optional.* After downloading each experiment, tar+gzip its `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory into `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz`. Existing archives are skipped with a warning. |
47
49
  | `-d`, `--delete` | *Optional.* Requires `-a/--archive`. After a successful archive, delete the `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory. Empty `SUBJECT` and `PROJECT` parent directories are also pruned. |
50
+ | `--overwrite` | *Optional.* Delete a non-empty `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory and download it again, instead of skipping it as `SKIPPED`. The directory is deleted only after the experiment is found on XNAT. With `-a/--archive`, an existing archive is also replaced, but only after a `COMPLETE` download; otherwise it is kept, with a `WARNING`. |
48
51
 
49
52
  ## Per-experiment STATUS (and exit code)
50
53
 
@@ -56,17 +59,31 @@ In `--csv` mode, and in `--accession` mode when it resolves to a subject, the ru
56
59
  | `FAILURE` | The experiment exists, but a zip request raised an error. If the underlying HTTP connection was dropped mid-download (a transient network/server timeout), the reported error names this cause explicitly rather than a generic message. |
57
60
  | `NONEXISTENT` | The experiment lookup did not find anything on the server. |
58
61
  | `EMPTY` | The experiment exists but has no scans and no session-level resources. |
62
+ | `SKIPPED` | Its experiment directory already existed and was not empty, so nothing was downloaded, archived or deleted. Use `--overwrite` to download it again. |
63
+ | `DUPLICATE` | `--csv` only. The experiment is listed more than once with the same renames; every instance is skipped. |
64
+ | `RENAME_CONFLICT` | `--csv` only. The experiment is listed more than once with different renames; every instance is skipped. |
65
+ | `COLLISION` | `--csv` only. Different experiments would download into the same directory; all of them are skipped. |
59
66
 
60
- Exit code is `0` if every processed experiment is `COMPLETE` or `EMPTY`, and `1` otherwise; archiving/deletion outcome (see below) does not affect it.
67
+ Exit code is `1` if any processed experiment is `FAILURE` or `NONEXISTENT`, and `0` otherwise, so skipped experiments do not fail a run. The archiving/deletion outcome (see below) does not affect it.
68
+
69
+ ## Repeated experiments in the CSV
70
+
71
+ An experiment is identified by its `PROJECT`, `SUBJECT_LABEL` and `EXPERIMENT_LABEL`. Before anything downloads, `--csv` mode prints one `WARNING` per problem and skips every affected row:
72
+
73
+ - **`DUPLICATE`:** the experiment appears on more than one row with the same `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` values (after the `sub-`/`ses-` prefix is added, so `01` and `sub-01` are the same). Delete the extra rows.
74
+ - **`RENAME_CONFLICT`:** the experiment appears on more than one row with different rename values, including blank vs. filled. Keep one row.
75
+ - **`COLLISION`:** different experiments would download into the same `PROJECT/SUBJECT/EXPERIMENT` directory, compared case-insensitively. Give each a unique rename.
76
+
77
+ Fix the CSV and re-run; experiments that already downloaded are `SKIPPED`.
61
78
 
62
79
  ## Archiving and deletion (`-a`/`--archive`, `-d`/`--delete`)
63
80
 
64
- With `-a/--archive`, each experiment's directory is tar+gzipped regardless of its download STATUS, and one of these is printed:
81
+ With `-a/--archive`, each downloaded experiment's directory is tar+gzipped regardless of its download STATUS, and one of these is printed. Experiments skipped before downloading (`SKIPPED`, `DUPLICATE`, `RENAME_CONFLICT`, `COLLISION`) are neither archived nor deleted, and the run summary repeats a `WARNING` with the number of `SKIPPED` ones.
65
82
 
66
83
  | Archive STATUS | Meaning |
67
84
  | --- | --- |
68
- | `COMPLETE` | The tarball was written successfully. |
69
- | `SKIPPED` | An archive already exists at that path; it is left untouched. |
85
+ | `COMPLETE` | The tarball was written successfully. With `--overwrite`, this includes replacing an existing archive after a `COMPLETE` download. |
86
+ | `SKIPPED` | An archive already exists at that path and was not replaced; it is left untouched. |
70
87
  | `FAILURE` | An error occurred while writing the tarball. |
71
88
  | `NONEXISTENT` | `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` does not exist (e.g., the download itself failed). |
72
89
 
@@ -94,4 +111,4 @@ When `-l/--log` is supplied, a CSV is written at `OUTPUT_DIR/log/download_<YYYYM
94
111
  DATESTAMP,USER,PROJECT,SUBJECT,EXPERIMENT,STATUS
95
112
  ```
96
113
 
97
- `DATESTAMP` is the per-experiment download attempt begin time, formatted to match Python's `logging` module default `asctime` (`YYYY-MM-DD HH:MM:SS,mmm`, local time). `USER` is the OS-level username running the command (via Python's `getpass.getuser()`). `SUBJECT`/`EXPERIMENT` reflect any rename override (`SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` in `--csv` mode, `--rename-subject`/`--rename-experiment` in `-1`/`--accession` mode — see [Manual Interventions](../manual.md)), i.e. the on-disk directory names actually written under `OUTPUT_DIR` (for `--accession` without a rename, this is the resolved XNAT ID, not a label), not necessarily the original XNAT labels. One row is appended per processed experiment; rows are written under a lock so concurrent workers do not interleave.
114
+ `DATESTAMP` is the per-experiment download attempt begin time, formatted to match Python's `logging` module default `asctime` (`YYYY-MM-DD HH:MM:SS,mmm`, local time). `USER` is the OS-level username running the command (via Python's `getpass.getuser()`). `SUBJECT`/`EXPERIMENT` reflect any rename override (`SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` in `--csv` mode, `--rename-subject`/`--rename-experiment` in `-1`/`--accession` mode — see [Manual Interventions](../manual.md)), i.e. the on-disk directory names actually written under `OUTPUT_DIR` (for `--accession` without a rename, this is the resolved XNAT ID, not a label), not necessarily the original XNAT labels. One row is appended per processed experiment, including skipped ones (one row per CSV row for `DUPLICATE`, `RENAME_CONFLICT` and `COLLISION`); rows are written under a lock so concurrent workers do not interleave.
@@ -33,7 +33,7 @@ With `-n/--nphysio` > 1, the `phys2bids` conversions run in parallel across sepa
33
33
  | `CONVERTED` | `phys2bids` read the raw file and its `_physio.tsv.gz`/`.json` were written next to the paired `.nii.gz`. |
34
34
  | `SKIPPED` | The expected single-frequency `_physio.tsv.gz` already existed at its destination and was a valid, non-corrupt GZIP file; conversion was not attempted. |
35
35
  | `NOT_PHYSIO` | The referenced file matched a supported extension but could not be loaded as physiological data. |
36
- | `READER_MISSING` | The optional reader package `phys2bids` needs for this format is not installed (e.g. `bioread` for `.acq`, `scipy` for `.mat`, `sonpy` for `.smr`). This is an environment problem, not a data problem — install the package and re-run. |
36
+ | `READER_MISSING` | The optional reader package `phys2bids` needs for this format is not installed (e.g. `bioread` for `.acq`, `pymatreader` for `.mat`, `sonpy` for `.smr`). This is an environment problem, not a data problem — install the package and re-run. |
37
37
  | `CONVERT_ERROR` | `phys2bids` raised while converting, produced no `.tsv.gz` output, or its destination was already occupied by a different association. |
38
38
  | `SOURCE_MISSING` | `PhysioParent` was unset/not a directory, the named basename was not found under it, or the `mriconvert_qc.tsv` row is missing `participant_id`/`datatype`. |
39
39
  | `COLLISION` | This `physio` basename is referenced by more than one `mriconvert_qc.tsv` row; none were converted until resolved. |
@@ -46,5 +46,3 @@ Exit code is `1` if any association is `CONVERT_ERROR` or `READER_MISSING`, and
46
46
  | `-p`, `--project` | **Required.** Project directory name under `OUTPUT_DIR` identifying the BIDS dataset produced by `xnatbidscli mriconvert`. |
47
47
  | `-n`, `--nphysio` | *Optional.* Number of physio conversions to run in parallel, one `phys2bids` conversion per process (default `1`). |
48
48
  | `-l`, `--log` | *Optional.* Write a per-association log CSV to `OUTPUT_DIR/log/physioconvert_<YYYYMMDD_HHMMSS>_log.csv` (header `DATESTAMP,USER,STATUS,MRI_FILENAME,PHYSIO_SOURCE,DESTINATION_PATH`). One row per processed association, except a conversion that produced several outputs emits one row per output; associations with no output get a single blank-`DESTINATION_PATH` row. Also mirrors everything printed to stdout/stderr into a companion text log at `OUTPUT_DIR/log/physioconvert_<YYYYMMDD_HHMMSS>_log.txt`, the Python equivalent of piping through `tee`. Off by default. |
49
-
50
- > **Note:** `physioconvert` needs Python 3.11. `phys2bids` 2.10.0, its latest release, caps `numpy` at `<1.24`, and `numpy` 1.23 has no builds for Python 3.12 or newer. So `phys2bids` (and `bioread`) install with `xnatbidscli` only on Python 3.11, where `nibabel` is held below 5.4. On Python 3.12 or newer, every other subcommand works and `physioconvert` exits with a message. This limit goes away once `phys2bids` releases its uncapped `master` branch.
@@ -25,7 +25,7 @@ Click a question to expand its answer, or use the buttons below to open or close
25
25
 
26
26
  ??? question "Why does physioconvert say phys2bids is required?"
27
27
 
28
- `phys2bids` installs with `xnatbidscli` only on Python 3.11. Its latest release (2.10.0) caps `numpy` below 1.24, and that `numpy` has no builds for Python 3.12 or newer. On 3.12 or newer, every other subcommand works, but `physioconvert` exits with this message. To convert physio, install `xnatbidscli` under Python 3.11. This limit goes away once `phys2bids` makes a new release.
28
+ `phys2bids` could not be imported in your environment. Since `xnatbidscli` 2.1.0, `phys2bids` installs with `xnatbidscli` on every supported Python version. Older releases installed it only on Python 3.11. Upgrade with `pip install --upgrade xnatbidscli`.
29
29
 
30
30
  **References:**
31
31
 
@@ -69,6 +69,14 @@ Click a question to expand its answer, or use the buttons below to open or close
69
69
 
70
70
  - [`xnatbidscli download`](cli/download.md)
71
71
 
72
+ ??? question "What happens if I re-run download on experiments I already downloaded?"
73
+
74
+ An experiment whose `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory exists and is not empty is skipped with a `WARNING` and reported `SKIPPED`. It is not downloaded, archived or deleted. To download it again, pass `--overwrite`, which deletes the directory first and, with `-a`, replaces its archive once the download is `COMPLETE`. Because `-a -d` removes the directory, a re-run after it downloads again. In `--csv` mode, an experiment listed on more than one row, or two experiments renamed into the same directory, are skipped with a `WARNING` too (`DUPLICATE`, `RENAME_CONFLICT`, `COLLISION`). None of these skips change the exit code.
75
+
76
+ **References:**
77
+
78
+ - [`xnatbidscli download` — repeated experiments](cli/download.md#repeated-experiments-in-the-csv)
79
+
72
80
  ??? question "How do I archive downloaded data?"
73
81
 
74
82
  Pass `-a`/`--archive` to `download` (or `mriconvert`) to tar+gzip each experiment's directory into `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz` once it's processed; add `-d`/`--delete` to remove the original directory after the archive is confirmed written.
@@ -8,9 +8,7 @@ pip install xnatbidscli
8
8
 
9
9
  This puts the `xnatbidscli` command on your `PATH`.
10
10
 
11
- > **Note:** [`physioconvert`](cli/physioconvert.md) needs Python 3.11 for now, because the latest `phys2bids` release does not support newer `numpy`. On Python 3.12 or newer, every other subcommand works.
12
-
13
- Runtime dependencies (installed automatically): `pyxnat`, `dcm2bids`, `dcm2niix` (the [`dcm2niix`](https://pypi.org/project/dcm2niix/) PyPI package vendors the binary onto your `PATH`), `pydicom`, `cubids`, `nibabel` (used to read NIfTI shapes for the `mriconvert` `scans.tsv`), `pandas` (used by `bidsmap`), and `phys2bids` (used by `physioconvert` to read physiological recordings and write BIDS physio files; Python 3.11 only, see the note under [`xnatbidscli physioconvert`](cli/physioconvert.md)). `bioread` is also pulled in on Python 3.11 for `phys2bids` to read BIOPAC `.acq` files (phys2bids imports it lazily but does not depend on it directly).
11
+ Runtime dependencies (installed automatically): `pyxnat`, `dcm2bids`, `dcm2niix` (the [`dcm2niix`](https://pypi.org/project/dcm2niix/) PyPI package vendors the binary onto your `PATH`), `pydicom`, `cubids`, `nibabel` (used to read NIfTI shapes for the `mriconvert` `scans.tsv`), `pandas` (used by `bidsmap`), and `phys2bids` (used by `physioconvert` to read physiological recordings and write BIDS physio files). `bioread` and `pymatreader` are also pulled in for `phys2bids` to read BIOPAC `.acq` and MATLAB `.mat` files (phys2bids imports them lazily but does not depend on them directly).
14
12
 
15
13
  ## Latest unreleased features
16
14
 
@@ -44,6 +44,7 @@ After an optional `sub-`/`ses-` prefix is stripped, the remaining value must be
44
44
 
45
45
  - In `--csv` mode, `download` validates every row of the input CSV before downloading anything; if any `SUBJECT_BIDS_RENAME` or `EXPERIMENT_BIDS_RENAME` value fails this check, the entire run exits with an error and nothing is downloaded. Fix the offending cell(s) in the CSV and re-run `download`.
46
46
  - In `-1`/`--accession` mode, an invalid `--rename-subject`/`--rename-experiment` value exits with an error before downloading.
47
+ - In `--csv` mode, an experiment listed on more than one row, with the same or different renames, or two experiments renamed into the same directory, are skipped with a `WARNING` rather than downloaded. See [Repeated experiments in the CSV](cli/download.md#repeated-experiments-in-the-csv).
47
48
 
48
49
  `--rename-subject`/`--rename-experiment` only apply to `-1`/`--accession` mode; supplying either alongside `--csv` is an error. `--rename-experiment` is additionally rejected when `--accession` resolves to a subject rather than an experiment.
49
50
 
@@ -19,24 +19,24 @@ classifiers = [
19
19
  "Programming Language :: Python :: 3",
20
20
  "Programming Language :: Python :: 3 :: Only",
21
21
  "Programming Language :: Python :: 3.11",
22
+ "Programming Language :: Python :: 3.12",
23
+ "Programming Language :: Python :: 3.13",
24
+ "Programming Language :: Python :: 3.14",
22
25
  "Topic :: Scientific/Engineering :: Bio-Informatics",
23
26
  "Topic :: Scientific/Engineering :: Medical Science Apps.",
24
27
  ]
25
- # phys2bids 2.10.0 (latest release) caps numpy at <1.24, and numpy 1.23 has no
26
- # wheels past Python 3.11, so phys2bids (and its bioread reader) install only on
27
- # Python 3.11, where nibabel is held below 5.4 (5.4 needs numpy>=1.25). Drop
28
- # these markers once phys2bids releases its uncapped master.
28
+ # bioread and pymatreader are the readers phys2bids imports lazily for .acq and
29
+ # .mat files; it does not depend on them directly.
29
30
  dependencies = [
30
- "bioread>=3; python_version < '3.12'",
31
+ "bioread>=3",
31
32
  "cubids>=1.2.1",
32
33
  "dcm2bids>=3",
33
34
  "dcm2niix>=1.0.20260416",
34
- "nibabel>=5.3,<5.4; python_version < '3.12'",
35
- "nibabel>=5.4.2; python_version >= '3.12'",
36
- "numpy<2",
35
+ "nibabel>=5.4.2",
37
36
  "pandas>=2,<2.4",
38
- "phys2bids>=2.10; python_version < '3.12'",
37
+ "phys2bids>=2.11",
39
38
  "pydicom>=2.4",
39
+ "pymatreader>=1",
40
40
  "pyxnat>=1.6",
41
41
  "requests>=2",
42
42
  ]
@@ -21,20 +21,26 @@ def archive_path(
21
21
 
22
22
 
23
23
  def archive_experiment(
24
- data_root: Path, project: str, subject: str, experiment: str
24
+ data_root: Path,
25
+ project: str,
26
+ subject: str,
27
+ experiment: str,
28
+ overwrite: bool = False,
25
29
  ) -> tuple[str, str | None]:
26
30
  """Tar+gzip data_root/PROJECT/SUBJECT/EXPERIMENT into data_root/archive/.
27
31
 
28
32
  Returns (status, detail). Existing archives are left untouched and reported
29
- as SKIPPED; the tarball is written to a .tmp sibling and renamed into place
30
- only on success so an interrupted run never leaves a partial file behind.
33
+ as SKIPPED unless `overwrite` is set; the tarball is written to a .tmp
34
+ sibling and renamed into place only on success so an interrupted run never
35
+ leaves a partial file behind (or loses the archive it was replacing).
31
36
  """
32
37
  src = data_root / project / subject / experiment
33
38
  if not src.is_dir():
34
39
  return STATUS_NONEXISTENT, f"source directory not found: {src}"
35
40
 
36
41
  dest = archive_path(data_root, project, subject, experiment)
37
- if dest.exists():
42
+ replacing = dest.exists()
43
+ if replacing and not overwrite:
38
44
  return STATUS_SKIPPED, f"archive already exists: {dest}"
39
45
 
40
46
  dest.parent.mkdir(parents=True, exist_ok=True)
@@ -50,6 +56,8 @@ def archive_experiment(
50
56
  except Exception:
51
57
  pass
52
58
  return STATUS_FAILURE, f"error creating archive: {e}"
59
+ if replacing:
60
+ return STATUS_COMPLETE, f"replaced existing archive: {dest}"
53
61
  return STATUS_COMPLETE, None
54
62
 
55
63
 
@@ -197,6 +197,17 @@ def build_parser() -> argparse.ArgumentParser:
197
197
  "OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT directory. The SUBJECT and "
198
198
  "PROJECT parent directories are also removed if they become empty.",
199
199
  )
200
+ download_parser.add_argument(
201
+ "--overwrite",
202
+ action="store_true",
203
+ help="Delete a non-empty OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT "
204
+ "directory and download it again. Without this flag, such an "
205
+ "experiment is skipped with a warning (STATUS SKIPPED), along with "
206
+ "-a/--archive and -d/--delete for it. The directory is deleted only "
207
+ "after the experiment is found on XNAT. With -a/--archive, an "
208
+ "existing archive is also replaced, but only after a COMPLETE "
209
+ "download.",
210
+ )
200
211
  download_parser.set_defaults(func=download_cmd)
201
212
 
202
213
  mriconfig_parser = subparsers.add_parser(
@@ -20,6 +20,7 @@ from pyxnat.core import uriutil
20
20
  from .archive import (
21
21
  OK_STATUSES as ARCHIVE_OK_STATUSES,
22
22
  archive_experiment,
23
+ archive_path,
23
24
  delete_experiment_dir,
24
25
  )
25
26
  from .login import load_credentials
@@ -29,8 +30,32 @@ STATUS_COMPLETE = "COMPLETE"
29
30
  STATUS_FAILURE = "FAILURE"
30
31
  STATUS_NONEXISTENT = "NONEXISTENT"
31
32
  STATUS_EMPTY = "EMPTY"
33
+ STATUS_SKIPPED = "SKIPPED"
34
+ STATUS_DUPLICATE = "DUPLICATE"
35
+ STATUS_RENAME_CONFLICT = "RENAME_CONFLICT"
36
+ STATUS_COLLISION = "COLLISION"
37
+
38
+ _OK_STATUSES = {
39
+ STATUS_COMPLETE,
40
+ STATUS_EMPTY,
41
+ STATUS_SKIPPED,
42
+ STATUS_DUPLICATE,
43
+ STATUS_RENAME_CONFLICT,
44
+ STATUS_COLLISION,
45
+ }
46
+ _SUMMARY_STATUSES = (
47
+ STATUS_COMPLETE,
48
+ STATUS_FAILURE,
49
+ STATUS_NONEXISTENT,
50
+ STATUS_EMPTY,
51
+ STATUS_SKIPPED,
52
+ STATUS_DUPLICATE,
53
+ STATUS_RENAME_CONFLICT,
54
+ STATUS_COLLISION,
55
+ )
32
56
 
33
- _OK_STATUSES = {STATUS_COMPLETE, STATUS_EMPTY}
57
+ # (PROJECT, SUBJECT_LABEL, EXPERIMENT_LABEL, SUBJECT_BIDS_RENAME, EXPERIMENT_BIDS_RENAME)
58
+ CsvRow = tuple[str, str, str, str | None, str | None]
34
59
 
35
60
  _thread_iface = threading.local()
36
61
 
@@ -418,6 +443,40 @@ def _extract_zip_flattened_or_rescue(
418
443
  zip_path.replace(dest_dir / filename)
419
444
 
420
445
 
446
+ def _has_contents(path: Path) -> bool:
447
+ """Return True if `path` is a directory with at least one entry in it."""
448
+ try:
449
+ return path.is_dir() and any(path.iterdir())
450
+ except OSError:
451
+ return False
452
+
453
+
454
+ def _skip_existing(
455
+ label: str, experiment_root: Path, do_archive: bool, do_delete: bool
456
+ ) -> bool:
457
+ """Warn and return True if `experiment_root` was already downloaded.
458
+
459
+ A non-empty experiment directory counts as already downloaded. This is a
460
+ local check only, made before contacting XNAT. Archiving/deletion are
461
+ skipped along with the download, so the warning is loud when they were
462
+ requested.
463
+ """
464
+ if not _has_contents(experiment_root):
465
+ return False
466
+ _safe_print(
467
+ f"WARNING: {label}: {experiment_root} already exists and is not "
468
+ "empty; skipping download. Pass --overwrite to delete it and "
469
+ "download again."
470
+ )
471
+ if do_archive:
472
+ steps = "ARCHIVE (-a) AND DELETE (-d)" if do_delete else "ARCHIVE (-a)"
473
+ _safe_print(
474
+ f"WARNING: {steps} NOT RUN for {label}: its download was skipped, "
475
+ f"so {experiment_root} and any existing archive were left untouched."
476
+ )
477
+ return True
478
+
479
+
421
480
  def _process_experiment(
422
481
  interface: Interface,
423
482
  project: str,
@@ -428,6 +487,7 @@ def _process_experiment(
428
487
  progress: _ExperimentProgress | None = None,
429
488
  local_subject: str | None = None,
430
489
  local_experiment: str | None = None,
490
+ overwrite: bool = False,
431
491
  ) -> str:
432
492
  """Download one experiment as whole-experiment zip archives.
433
493
 
@@ -444,6 +504,10 @@ def _process_experiment(
444
504
  `EXPERIMENT_BIDS_RENAME`) name the on-disk directory when they differ
445
505
  from `subject`/`experiment`, which are always used to look up the
446
506
  experiment on the XNAT server itself.
507
+
508
+ With `overwrite`, a non-empty existing experiment directory is deleted
509
+ before downloading, but only once the experiment is confirmed to exist
510
+ on XNAT, so a lookup failure never costs the local copy.
447
511
  """
448
512
  label = f"{project}/{subject}/{experiment}"
449
513
  try:
@@ -461,6 +525,13 @@ def _process_experiment(
461
525
  experiment_root = (
462
526
  output_dir / project / (local_subject or subject) / (local_experiment or experiment)
463
527
  )
528
+ if overwrite and _has_contents(experiment_root):
529
+ report(f"WARNING: {label}: deleting existing {experiment_root} before re-downloading (--overwrite).")
530
+ try:
531
+ shutil.rmtree(experiment_root)
532
+ except OSError as e:
533
+ report(f" Error deleting {experiment_root}: {e}")
534
+ return STATUS_FAILURE
464
535
  experiment_root.mkdir(parents=True, exist_ok=True)
465
536
 
466
537
  got_scans = False
@@ -507,12 +578,10 @@ def _format_bids_rename(value: str, prefix: str) -> str | None:
507
578
  return value if value.startswith(prefix) else prefix + value
508
579
 
509
580
 
510
- def _read_csv_rows(
511
- path: Path,
512
- ) -> list[tuple[str, str, str, str | None, str | None]]:
581
+ def _read_csv_rows(path: Path) -> list[CsvRow]:
513
582
  if not path.exists():
514
583
  sys.exit(f"Error: input CSV not found: {path}")
515
- rows: list[tuple[str, str, str, str | None, str | None]] = []
584
+ rows: list[CsvRow] = []
516
585
  with path.open(newline="") as f:
517
586
  reader = csv.DictReader(f)
518
587
  required = {"PROJECT", "SUBJECT_LABEL", "EXPERIMENT_LABEL"}
@@ -557,6 +626,76 @@ def _read_csv_rows(
557
626
  return rows
558
627
 
559
628
 
629
+ def _screen_csv_rows(
630
+ rows: list[CsvRow],
631
+ ) -> tuple[list[CsvRow], list[tuple[CsvRow, str]]]:
632
+ """Split CSV rows into ones to download and ones skipped with a WARNING.
633
+
634
+ An experiment is identified by (PROJECT, SUBJECT_LABEL, EXPERIMENT_LABEL).
635
+ Every row of an experiment listed more than once is skipped: DUPLICATE
636
+ if all its rows have the same (normalized) renames, RENAME_CONFLICT if
637
+ not. Every remaining row whose on-disk directory is shared with a
638
+ different experiment is skipped as COLLISION.
639
+
640
+ Returns
641
+ -------
642
+ tuple[list[CsvRow], list[tuple[CsvRow, str]]]
643
+ Rows to download, and skipped rows paired with their STATUS.
644
+ """
645
+ by_experiment: dict[tuple[str, str, str], list[CsvRow]] = {}
646
+ for row in rows:
647
+ by_experiment.setdefault(row[:3], []).append(row)
648
+
649
+ flagged: dict[tuple[str, str, str], str] = {}
650
+ for key, group in by_experiment.items():
651
+ if len(group) < 2:
652
+ continue
653
+ label = "/".join(key)
654
+ if len({row[3:] for row in group}) == 1:
655
+ flagged[key] = STATUS_DUPLICATE
656
+ print(
657
+ f"WARNING: {label} is listed {len(group)} times in the input "
658
+ "CSV; skipping every instance. Remove the extra row(s) to "
659
+ "download it."
660
+ )
661
+ else:
662
+ flagged[key] = STATUS_RENAME_CONFLICT
663
+ renames = "; ".join(
664
+ f"SUBJECT_BIDS_RENAME={sr or '(blank)'}, "
665
+ f"EXPERIMENT_BIDS_RENAME={er or '(blank)'}"
666
+ for _, _, _, sr, er in group
667
+ )
668
+ print(
669
+ f"WARNING: {label} is listed {len(group)} times in the input "
670
+ "CSV with different SUBJECT_BIDS_RENAME/EXPERIMENT_BIDS_RENAME "
671
+ f"values ({renames}); skipping every instance. Keep only one "
672
+ "row to download it."
673
+ )
674
+
675
+ # Compared case-insensitively, since Windows and macOS filesystems would
676
+ # merge directories that differ only in case.
677
+ by_local_dir: dict[tuple[str, ...], dict[tuple[str, str, str], str]] = {}
678
+ for p, s, e, sr, er in rows:
679
+ local = f"{p}/{sr or s}/{er or e}"
680
+ key = tuple(part.casefold() for part in (p, sr or s, er or e))
681
+ by_local_dir.setdefault(key, {})[(p, s, e)] = local
682
+ for experiments in by_local_dir.values():
683
+ if len(experiments) < 2:
684
+ continue
685
+ labels = ", ".join("/".join(k) for k in experiments)
686
+ print(
687
+ f"WARNING: {labels} would all download into "
688
+ f"{next(iter(experiments.values()))}; skipping all of them. Give "
689
+ "each a unique SUBJECT_BIDS_RENAME/EXPERIMENT_BIDS_RENAME."
690
+ )
691
+ for key in experiments:
692
+ flagged.setdefault(key, STATUS_COLLISION)
693
+
694
+ to_download = [row for row in rows if row[:3] not in flagged]
695
+ skipped = [(row, flagged[row[:3]]) for row in rows if row[:3] in flagged]
696
+ return to_download, skipped
697
+
698
+
560
699
  def _archive_and_maybe_delete(
561
700
  output_dir: Path,
562
701
  project: str,
@@ -565,12 +704,30 @@ def _archive_and_maybe_delete(
565
704
  do_archive: bool,
566
705
  do_delete: bool,
567
706
  report: Callable[[str], None],
707
+ overwrite: bool = False,
708
+ download_status: str = STATUS_COMPLETE,
568
709
  ) -> None:
710
+ """Archive (and with `do_delete`, then delete) one experiment directory.
711
+
712
+ With `overwrite`, an existing archive is replaced, but only after a
713
+ COMPLETE download, so a failed or partial re-download never replaces a
714
+ good archive.
715
+ """
569
716
  if not do_archive:
570
717
  return
571
718
  label = f"{project}/{subject}/{experiment}"
719
+ replace = overwrite and download_status == STATUS_COMPLETE
720
+ if (
721
+ overwrite
722
+ and not replace
723
+ and archive_path(output_dir, project, subject, experiment).exists()
724
+ ):
725
+ report(
726
+ f"WARNING: not replacing the existing archive for {label} "
727
+ f"(--overwrite), since its download was {download_status}."
728
+ )
572
729
  a_status, a_detail = archive_experiment(
573
- output_dir, project, subject, experiment
730
+ output_dir, project, subject, experiment, overwrite=replace
574
731
  )
575
732
  line = f" archive {label}: {a_status}"
576
733
  if a_detail:
@@ -593,15 +750,23 @@ def _run_single(
593
750
  do_delete: bool,
594
751
  local_subject: str | None = None,
595
752
  local_experiment: str | None = None,
753
+ overwrite: bool = False,
596
754
  ) -> str:
597
755
  local_s = local_subject or subject
598
756
  local_e = local_experiment or experiment
757
+ label = f"{project}/{subject}/{experiment}"
758
+ if not overwrite and _skip_existing(
759
+ label, output_dir / project / local_s / local_e, do_archive, do_delete
760
+ ):
761
+ log_writer.write(_logging_now(), project, local_s, local_e, STATUS_SKIPPED)
762
+ return STATUS_SKIPPED
599
763
  iface = Interface(server=server, user=user, password=password)
600
764
  try:
601
765
  start = _logging_now()
602
766
  status = _process_experiment(
603
767
  iface, project, subject, experiment, output_dir, _safe_print,
604
768
  local_subject=local_s, local_experiment=local_e,
769
+ overwrite=overwrite,
605
770
  )
606
771
  finally:
607
772
  try:
@@ -617,6 +782,8 @@ def _run_single(
617
782
  do_archive,
618
783
  do_delete,
619
784
  _safe_print,
785
+ overwrite,
786
+ status,
620
787
  )
621
788
  return status
622
789
 
@@ -625,27 +792,28 @@ def _run_csv(
625
792
  server: str,
626
793
  user: str,
627
794
  password: str,
628
- rows: list[tuple[str, str, str, str | None, str | None]],
795
+ rows: list[CsvRow],
629
796
  output_dir: Path,
630
797
  n_parallel_experiments: int,
631
798
  log_writer: _LogWriter,
632
799
  do_archive: bool,
633
800
  do_delete: bool,
801
+ overwrite: bool = False,
634
802
  ) -> dict[str, int]:
635
- counts = {
636
- STATUS_COMPLETE: 0,
637
- STATUS_FAILURE: 0,
638
- STATUS_NONEXISTENT: 0,
639
- STATUS_EMPTY: 0,
640
- }
641
-
642
- def _worker(row: tuple[str, str, str, str | None, str | None]) -> str:
803
+ counts = dict.fromkeys(_SUMMARY_STATUSES, 0)
804
+
805
+ def _worker(row: CsvRow) -> str:
643
806
  p, s, e, subject_rename, experiment_rename = row
644
807
  local_s = subject_rename or s
645
808
  local_e = experiment_rename or e
646
- iface = _get_thread_interface(server, user, password)
647
809
  label = f"{p}/{s}/{e}"
648
810
  experiment_root = output_dir / p / local_s / local_e
811
+ if not overwrite and _skip_existing(
812
+ label, experiment_root, do_archive, do_delete
813
+ ):
814
+ log_writer.write(_logging_now(), p, local_s, local_e, STATUS_SKIPPED)
815
+ return STATUS_SKIPPED
816
+ iface = _get_thread_interface(server, user, password)
649
817
  progress = _ExperimentProgress()
650
818
  stop_event = threading.Event()
651
819
  monitor = threading.Thread(
@@ -658,14 +826,15 @@ def _run_csv(
658
826
  start = _logging_now()
659
827
  status = _process_experiment(
660
828
  iface, p, s, e, output_dir, _safe_print, progress,
661
- local_s, local_e,
829
+ local_s, local_e, overwrite,
662
830
  )
663
831
  finally:
664
832
  stop_event.set()
665
833
  monitor.join()
666
834
  log_writer.write(start, p, local_s, local_e, status)
667
835
  _archive_and_maybe_delete(
668
- output_dir, p, local_s, local_e, do_archive, do_delete, _safe_print
836
+ output_dir, p, local_s, local_e, do_archive, do_delete, _safe_print,
837
+ overwrite, status,
669
838
  )
670
839
  return status
671
840
 
@@ -714,6 +883,33 @@ def _run_csv(
714
883
  return counts
715
884
 
716
885
 
886
+ def _print_summary(
887
+ heading: str,
888
+ counts: dict[str, int],
889
+ log_path: Path | None,
890
+ do_archive: bool,
891
+ do_delete: bool,
892
+ ) -> int:
893
+ """Print a batch run's per-STATUS counts and return its exit code.
894
+
895
+ Repeats the not-archived warning for SKIPPED experiments at the end,
896
+ since the per-experiment warnings may have scrolled away.
897
+ """
898
+ print(f"\n{heading}")
899
+ for status in _SUMMARY_STATUSES:
900
+ print(f" {status}: {counts[status]}")
901
+ if do_archive and counts[STATUS_SKIPPED]:
902
+ steps = "ARCHIVED OR DELETED" if do_delete else "ARCHIVED"
903
+ print(
904
+ f"WARNING: {counts[STATUS_SKIPPED]} SKIPPED experiment(s) were NOT "
905
+ f"{steps}, since their existing directories were left untouched."
906
+ )
907
+ if log_path is not None:
908
+ print(f"Log written to {log_path}")
909
+ bad = counts[STATUS_FAILURE] + counts[STATUS_NONEXISTENT]
910
+ return 0 if bad == 0 else 1
911
+
912
+
717
913
  def _resolve_accession(
718
914
  interface: Interface, accession: str
719
915
  ) -> tuple[str, str, str, str | None]:
@@ -810,6 +1006,7 @@ def _download_accession(
810
1006
  local_subject: str | None,
811
1007
  local_experiment: str | None,
812
1008
  log_path: Path | None,
1009
+ overwrite: bool = False,
813
1010
  ) -> int:
814
1011
  """Handle `download --accession`: resolve it, then delegate like -1/--csv.
815
1012
 
@@ -860,26 +1057,18 @@ def _download_accession(
860
1057
  ]
861
1058
  counts = _run_csv(
862
1059
  server, user, password, rows, output_dir, n_parallel_experiments,
863
- log_writer, do_archive, do_delete,
1060
+ log_writer, do_archive, do_delete, overwrite,
1061
+ )
1062
+ return _print_summary(
1063
+ f"Processed {sum(counts.values())} experiment(s) for subject "
1064
+ f"accession {accession}:",
1065
+ counts, log_path, do_archive, do_delete,
864
1066
  )
865
- total = sum(counts.values())
866
- print(f"\nProcessed {total} experiment(s) for subject accession {accession}:")
867
- for status in (
868
- STATUS_COMPLETE,
869
- STATUS_FAILURE,
870
- STATUS_NONEXISTENT,
871
- STATUS_EMPTY,
872
- ):
873
- print(f" {status}: {counts[status]}")
874
- if log_path is not None:
875
- print(f"Log written to {log_path}")
876
- bad = counts[STATUS_FAILURE] + counts[STATUS_NONEXISTENT]
877
- return 0 if bad == 0 else 1
878
1067
 
879
1068
  status = _run_single(
880
1069
  server, user, password, project, subject_id, experiment_id,
881
1070
  output_dir, log_writer, do_archive, do_delete,
882
- local_subject, local_experiment,
1071
+ local_subject, local_experiment, overwrite,
883
1072
  )
884
1073
  print(
885
1074
  f"Status for accession {accession} "
@@ -958,6 +1147,7 @@ def download_cmd(args: argparse.Namespace) -> int:
958
1147
  args.delete,
959
1148
  local_subject,
960
1149
  local_experiment,
1150
+ args.overwrite,
961
1151
  )
962
1152
  print(
963
1153
  f"Status for {project}/{subject}/{experiment}: {status}"
@@ -980,9 +1170,10 @@ def download_cmd(args: argparse.Namespace) -> int:
980
1170
  local_subject,
981
1171
  local_experiment,
982
1172
  log_path,
1173
+ args.overwrite,
983
1174
  )
984
1175
 
985
- rows = _read_csv_rows(Path(args.input))
1176
+ rows, screened = _screen_csv_rows(_read_csv_rows(Path(args.input)))
986
1177
  counts = _run_csv(
987
1178
  server,
988
1179
  user,
@@ -993,17 +1184,12 @@ def download_cmd(args: argparse.Namespace) -> int:
993
1184
  log_writer,
994
1185
  args.archive,
995
1186
  args.delete,
1187
+ args.overwrite,
1188
+ )
1189
+ for (p, s, e, sr, er), status in screened:
1190
+ log_writer.write(_logging_now(), p, sr or s, er or e, status)
1191
+ counts[status] += 1
1192
+ return _print_summary(
1193
+ f"Processed {sum(counts.values())} experiment(s):",
1194
+ counts, log_path, args.archive, args.delete,
996
1195
  )
997
- total = sum(counts.values())
998
- print(f"\nProcessed {total} experiment(s):")
999
- for status in (
1000
- STATUS_COMPLETE,
1001
- STATUS_FAILURE,
1002
- STATUS_NONEXISTENT,
1003
- STATUS_EMPTY,
1004
- ):
1005
- print(f" {status}: {counts[status]}")
1006
- if log_path is not None:
1007
- print(f"Log written to {log_path}")
1008
- bad = counts[STATUS_FAILURE] + counts[STATUS_NONEXISTENT]
1009
- return 0 if bad == 0 else 1
@@ -54,7 +54,7 @@ STATUS_SKIPPED = "SKIPPED"
54
54
  # is an environment problem, not a sign the file is not physiological data.
55
55
  _READER_PACKAGES = {
56
56
  ".acq": "bioread",
57
- ".mat": "scipy",
57
+ ".mat": "pymatreader",
58
58
  ".smr": "sonpy",
59
59
  }
60
60
 
@@ -607,9 +607,8 @@ def physioconvert_cmd(args: argparse.Namespace) -> int:
607
607
  from phys2bids.phys2bids import phys2bids as _p2b # noqa: F401
608
608
  except ImportError:
609
609
  sys.exit(
610
- "Error: phys2bids is required for physioconvert. It installs with "
611
- "xnatbidscli on Python 3.11 only until phys2bids releases support for "
612
- "newer numpy; reinstall xnatbidscli under Python 3.11."
610
+ "Error: phys2bids is required for physioconvert. "
611
+ "Reinstall xnatbidscli with 'pip install --upgrade xnatbidscli'."
613
612
  )
614
613
 
615
614
  # physioconvert only reads mriconvert_qc.tsv -- it never writes it back.
@@ -75,6 +75,13 @@ This is a manual test plan for exercising every `xnatbidscli` subcommand and its
75
75
  - [ ] `-1` single-experiment mode ignores `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` entirely (no CSV involved)
76
76
  - [ ] `-a/--archive` produces `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz` after each experiment downloads
77
77
  - [ ] Re-running with `-a/--archive` against an experiment whose archive already exists skips archiving with a warning (does not overwrite)
78
+ - [ ] Re-running against an experiment whose directory exists and is not empty prints a `WARNING`, reports `SKIPPED`, does not contact XNAT for it, and exits `0`; an empty directory still downloads
79
+ - [ ] With `-a` (and `-d`), a `SKIPPED` experiment prints a louder `WARNING` that archive/delete were not run, leaves its directory and archive untouched, and the summary repeats the count
80
+ - [ ] `--overwrite` deletes a non-empty experiment directory and downloads it fresh; a `NONEXISTENT` experiment's local directory is not deleted
81
+ - [ ] `--overwrite -a` replaces an existing archive after a `COMPLETE` download (archive `COMPLETE — replaced existing archive`); after a `FAILURE`/`EMPTY` download it keeps the old archive with a `WARNING`
82
+ - [ ] A CSV listing the same experiment twice with the same renames (`01` vs `sub-01` counts as the same) prints one `WARNING` and reports both rows `DUPLICATE`, downloading neither
83
+ - [ ] A CSV listing the same experiment twice with different renames (including blank vs. filled) prints one `WARNING` and reports both rows `RENAME_CONFLICT`, downloading neither
84
+ - [ ] Two different experiments renamed into the same directory (also when differing only in case) print one `WARNING` and are both reported `COLLISION`, downloading neither
78
85
  - [ ] `-d/--delete` without `-a/--archive` fails immediately with "requires -a/--archive"
79
86
  - [ ] `-a -d` together delete `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` only after a successful archive
80
87
  - [ ] `-a -d` also removes the `SUBJECT` and then `PROJECT` parent directories if they become empty
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes