xnatbidscli 2.0.0__tar.gz → 2.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/PKG-INFO +8 -6
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/changelog.md +21 -1
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/download.md +25 -8
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/physioconvert.md +1 -3
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/faq.md +9 -1
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/installation.md +1 -3
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/manual.md +1 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/pyproject.toml +9 -9
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/archive.py +12 -4
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/cli.py +11 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/download.py +233 -47
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/physioconvert.py +3 -4
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/test_plan.md +7 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.claude/CLAUDE.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.github/workflows/publish.yml +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.gitignore +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.python-version +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/.readthedocs.yaml +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/AGENTS.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/CHANGELOG.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/LICENSE +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/README.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/assets/images/favicon.png +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/assets/images/logo.png +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/assets/js/faq.js +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/bidsmap.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/cubids.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/login.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/mriconfig.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/mriconvert.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/cli/query.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/design.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/excel.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/index.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/quickstart.md +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/docs/requirements.txt +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/assets/mriconvert_qc.json +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/assets/physioconvert_qc.json +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/__init__.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/bidsmap.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/cubids.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/login.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/mriconfig.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/mriconvert.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/query.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/src/xnatbidscli/sysinfo.py +0 -0
- {xnatbidscli-2.0.0 → xnatbidscli-2.1.0}/zensical.toml +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: xnatbidscli
|
|
3
|
-
Version: 2.
|
|
3
|
+
Version: 2.1.0
|
|
4
4
|
Summary: CLI for logging into XNAT, downloading neuroimaging experiments, and converting them to BIDS.
|
|
5
5
|
Project-URL: Homepage, https://github.com/nimh-dsst/xnat-bids-cli
|
|
6
6
|
Project-URL: Documentation, https://xnatbidscli.readthedocs.io/
|
|
@@ -17,19 +17,21 @@ Classifier: Operating System :: OS Independent
|
|
|
17
17
|
Classifier: Programming Language :: Python :: 3
|
|
18
18
|
Classifier: Programming Language :: Python :: 3 :: Only
|
|
19
19
|
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
20
23
|
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
|
|
21
24
|
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
|
|
22
25
|
Requires-Python: >=3.11
|
|
23
|
-
Requires-Dist: bioread>=3
|
|
26
|
+
Requires-Dist: bioread>=3
|
|
24
27
|
Requires-Dist: cubids>=1.2.1
|
|
25
28
|
Requires-Dist: dcm2bids>=3
|
|
26
29
|
Requires-Dist: dcm2niix>=1.0.20260416
|
|
27
|
-
Requires-Dist: nibabel
|
|
28
|
-
Requires-Dist: nibabel>=5.4.2; python_version >= '3.12'
|
|
29
|
-
Requires-Dist: numpy<2
|
|
30
|
+
Requires-Dist: nibabel>=5.4.2
|
|
30
31
|
Requires-Dist: pandas<2.4,>=2
|
|
31
|
-
Requires-Dist: phys2bids>=2.
|
|
32
|
+
Requires-Dist: phys2bids>=2.11
|
|
32
33
|
Requires-Dist: pydicom>=2.4
|
|
34
|
+
Requires-Dist: pymatreader>=1
|
|
33
35
|
Requires-Dist: pyxnat>=1.6
|
|
34
36
|
Requires-Dist: requests>=2
|
|
35
37
|
Description-Content-Type: text/markdown
|
|
@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [2.1.0] - 2026-10-28
|
|
11
|
+
|
|
12
|
+
`physioconvert` now works on every supported Python version, using the new `phys2bids` 2.11 release. `download` no longer downloads over an experiment that is already on disk, and has a new `--overwrite` flag.
|
|
13
|
+
|
|
14
|
+
### Added
|
|
15
|
+
|
|
16
|
+
- `download --overwrite` deletes an experiment's existing directory and downloads it again. With `-a/--archive`, it also replaces an existing archive after a `COMPLETE` download.
|
|
17
|
+
- `download --csv` skips with a `WARNING` every row of an experiment listed more than once (`DUPLICATE`, or `RENAME_CONFLICT` when its `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` values differ), and every experiment that would download into the same directory as another (`COLLISION`).
|
|
18
|
+
- `pymatreader` is now installed with `xnatbidscli`, so `physioconvert` can read MATLAB `.mat` files without an extra install.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
- `download` skips an experiment whose directory already exists and is not empty, with a `WARNING` and a new `SKIPPED` status, instead of downloading over it. `-a/--archive` and `-d/--delete` are not run for it, with a louder `WARNING`. Skipped experiments do not change the exit code.
|
|
23
|
+
- `physioconvert` works on every supported Python version (3.11 and newer), not just 3.11. `xnatbidscli` now requires `phys2bids` 2.11 or newer, which no longer caps `numpy` below 1.24. `phys2bids` and `bioread` install on every Python version, and the `numpy<2` and `nibabel<5.4` pins are removed.
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
|
|
27
|
+
- The `READER_MISSING` hint for `.mat` files now names `pymatreader`, the package `phys2bids` actually uses, instead of `scipy`.
|
|
28
|
+
|
|
10
29
|
## [2.0.0] - 2026-09-28
|
|
11
30
|
|
|
12
31
|
The project is renamed from `xnatcli` to `xnatbidscli` and is published on PyPI. Downloads are faster and more reliable, and you can now rename subjects and experiments between `query` and `download`.
|
|
@@ -58,6 +77,7 @@ First release.
|
|
|
58
77
|
- `xnatcli` subcommands: `login`, `query`, `download`, `mriconfig`, `mriconvert`, `physioconvert`, `bidsmap` and `cubids`.
|
|
59
78
|
- A documentation site built with Zensical and hosted on Read the Docs.
|
|
60
79
|
|
|
61
|
-
[Unreleased]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v2.
|
|
80
|
+
[Unreleased]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v2.1.0...HEAD
|
|
81
|
+
[2.1.0]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v2.0.0...v2.1.0
|
|
62
82
|
[2.0.0]: https://github.com/nimh-dsst/xnat-bids-cli/compare/v1.0.0...v2.0.0
|
|
63
83
|
[1.0.0]: https://github.com/nimh-dsst/xnat-bids-cli/releases/tag/v1.0.0
|
|
@@ -4,12 +4,14 @@ Downloads every file belonging to one XNAT experiment (single-experiment mode, `
|
|
|
4
4
|
|
|
5
5
|
1. Loads credentials from `~/.xnatbidscli/credentials.cfg`; if the file is missing or incomplete, exits with a message telling you to run `xnatbidscli login`.
|
|
6
6
|
2. Connects to the stored server via PyXNAT.
|
|
7
|
-
3.
|
|
8
|
-
4.
|
|
7
|
+
3. In `--csv` mode, every row is screened before anything downloads, and repeated or colliding experiments are skipped (see [Repeated experiments in the CSV](#repeated-experiments-in-the-csv)).
|
|
8
|
+
4. For each experiment, if `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` (using any rename) already exists and is not empty, it counts as already downloaded. A `WARNING` is printed and the experiment is reported `SKIPPED` without contacting XNAT. `-a`/`-d` are not run for it either, and a second, louder `WARNING` says so. Pass `--overwrite` to delete the directory and download it again instead. The directory is deleted only after the experiment is found on XNAT. An empty directory does not count as downloaded. Because `-d` removes the directory, re-running after `-a -d` downloads those experiments again.
|
|
9
|
+
5. For each experiment, walks `project → subject → experiment`, then issues zip requests against XNAT's REST API: one bulk request for all scans, and one request per session-level resource (XNAT has no bulk "all resources" export endpoint, unlike scans). With `--accession`, an extra server-wide lookup happens first, in this order: subject ID, experiment ID, study UID (XNAT's stored session `UID`), then subject and experiment labels together. IDs and UIDs are unique across the server. Labels are only unique within their parent, so a label that matches more than one subject or experiment exits with an error listing each match's XNAT ID, project and subject; rerun with one of those IDs. A UID only matches sessions where XNAT stored it.
|
|
10
|
+
6. Each zip is extracted directly into `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT/`, following XNAT's own scan/resource folder naming (not a custom path scheme), then discarded. If a resource resolves to a single file, XNAT sometimes streams that file directly instead of wrapping it in a zip; this is detected and the file is saved as-is rather than failing the experiment. If some resources download successfully and others fail, the successful ones are still kept and the experiment is reported as `FAILURE` with each failing resource named in the error.
|
|
9
11
|
|
|
10
12
|
`PROJECT` is the canonical XNAT project ID; `SUBJECT` and `EXPERIMENT` are the user-facing labels emitted by `xnatbidscli query`, unless overridden by that row's `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` values in `--csv` mode, or by `--rename-subject`/`--rename-experiment` in `-1`/`--accession` mode (see [Manual Interventions](../manual.md)) — XNAT is still queried using the original labels either way. In `--accession` mode, the resolved XNAT IDs (not labels) are used both to query XNAT and, absent a rename, as the on-disk `SUBJECT`/`EXPERIMENT` directory names.
|
|
11
13
|
|
|
12
|
-
|
|
14
|
+
7. With `-a/--archive`, after each experiment is downloaded, its `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory (using any rename override) is tar+gzipped to `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz`. An existing archive at that path is left untouched and reported as `SKIPPED`. With `-d/--delete` (requires `-a/--archive`), the `EXPERIMENT` directory is removed once its archive is `COMPLETE` or `SKIPPED`; the `SUBJECT` and then `PROJECT` parent directories are also removed if they become empty as a result.
|
|
13
15
|
|
|
14
16
|
```bash
|
|
15
17
|
# Single experiment
|
|
@@ -45,6 +47,7 @@ xnatbidscli download --csv PATH/TO/QUERY.csv -o OUTPUT_DIR -a -d
|
|
|
45
47
|
| `-l`, `--log` | *Optional.* Write a per-experiment log CSV to `OUTPUT_DIR/log/download_<YYYYMMDD_HHMMSS>_log.csv` (local time, captured at run start). |
|
|
46
48
|
| `-a`, `--archive` | *Optional.* After downloading each experiment, tar+gzip its `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory into `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz`. Existing archives are skipped with a warning. |
|
|
47
49
|
| `-d`, `--delete` | *Optional.* Requires `-a/--archive`. After a successful archive, delete the `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory. Empty `SUBJECT` and `PROJECT` parent directories are also pruned. |
|
|
50
|
+
| `--overwrite` | *Optional.* Delete a non-empty `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory and download it again, instead of skipping it as `SKIPPED`. The directory is deleted only after the experiment is found on XNAT. With `-a/--archive`, an existing archive is also replaced, but only after a `COMPLETE` download; otherwise it is kept, with a `WARNING`. |
|
|
48
51
|
|
|
49
52
|
## Per-experiment STATUS (and exit code)
|
|
50
53
|
|
|
@@ -56,17 +59,31 @@ In `--csv` mode, and in `--accession` mode when it resolves to a subject, the ru
|
|
|
56
59
|
| `FAILURE` | The experiment exists, but a zip request raised an error. If the underlying HTTP connection was dropped mid-download (a transient network/server timeout), the reported error names this cause explicitly rather than a generic message. |
|
|
57
60
|
| `NONEXISTENT` | The experiment lookup did not find anything on the server. |
|
|
58
61
|
| `EMPTY` | The experiment exists but has no scans and no session-level resources. |
|
|
62
|
+
| `SKIPPED` | Its experiment directory already existed and was not empty, so nothing was downloaded, archived or deleted. Use `--overwrite` to download it again. |
|
|
63
|
+
| `DUPLICATE` | `--csv` only. The experiment is listed more than once with the same renames; every instance is skipped. |
|
|
64
|
+
| `RENAME_CONFLICT` | `--csv` only. The experiment is listed more than once with different renames; every instance is skipped. |
|
|
65
|
+
| `COLLISION` | `--csv` only. Different experiments would download into the same directory; all of them are skipped. |
|
|
59
66
|
|
|
60
|
-
Exit code is `
|
|
67
|
+
Exit code is `1` if any processed experiment is `FAILURE` or `NONEXISTENT`, and `0` otherwise, so skipped experiments do not fail a run. The archiving/deletion outcome (see below) does not affect it.
|
|
68
|
+
|
|
69
|
+
## Repeated experiments in the CSV
|
|
70
|
+
|
|
71
|
+
An experiment is identified by its `PROJECT`, `SUBJECT_LABEL` and `EXPERIMENT_LABEL`. Before anything downloads, `--csv` mode prints one `WARNING` per problem and skips every affected row:
|
|
72
|
+
|
|
73
|
+
- **`DUPLICATE`:** the experiment appears on more than one row with the same `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` values (after the `sub-`/`ses-` prefix is added, so `01` and `sub-01` are the same). Delete the extra rows.
|
|
74
|
+
- **`RENAME_CONFLICT`:** the experiment appears on more than one row with different rename values, including blank vs. filled. Keep one row.
|
|
75
|
+
- **`COLLISION`:** different experiments would download into the same `PROJECT/SUBJECT/EXPERIMENT` directory, compared case-insensitively. Give each a unique rename.
|
|
76
|
+
|
|
77
|
+
Fix the CSV and re-run; experiments that already downloaded are `SKIPPED`.
|
|
61
78
|
|
|
62
79
|
## Archiving and deletion (`-a`/`--archive`, `-d`/`--delete`)
|
|
63
80
|
|
|
64
|
-
With `-a/--archive`, each experiment's directory is tar+gzipped regardless of its download STATUS, and one of these is printed
|
|
81
|
+
With `-a/--archive`, each downloaded experiment's directory is tar+gzipped regardless of its download STATUS, and one of these is printed. Experiments skipped before downloading (`SKIPPED`, `DUPLICATE`, `RENAME_CONFLICT`, `COLLISION`) are neither archived nor deleted, and the run summary repeats a `WARNING` with the number of `SKIPPED` ones.
|
|
65
82
|
|
|
66
83
|
| Archive STATUS | Meaning |
|
|
67
84
|
| --- | --- |
|
|
68
|
-
| `COMPLETE` | The tarball was written successfully. |
|
|
69
|
-
| `SKIPPED` | An archive already exists at that path; it is left untouched. |
|
|
85
|
+
| `COMPLETE` | The tarball was written successfully. With `--overwrite`, this includes replacing an existing archive after a `COMPLETE` download. |
|
|
86
|
+
| `SKIPPED` | An archive already exists at that path and was not replaced; it is left untouched. |
|
|
70
87
|
| `FAILURE` | An error occurred while writing the tarball. |
|
|
71
88
|
| `NONEXISTENT` | `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` does not exist (e.g., the download itself failed). |
|
|
72
89
|
|
|
@@ -94,4 +111,4 @@ When `-l/--log` is supplied, a CSV is written at `OUTPUT_DIR/log/download_<YYYYM
|
|
|
94
111
|
DATESTAMP,USER,PROJECT,SUBJECT,EXPERIMENT,STATUS
|
|
95
112
|
```
|
|
96
113
|
|
|
97
|
-
`DATESTAMP` is the per-experiment download attempt begin time, formatted to match Python's `logging` module default `asctime` (`YYYY-MM-DD HH:MM:SS,mmm`, local time). `USER` is the OS-level username running the command (via Python's `getpass.getuser()`). `SUBJECT`/`EXPERIMENT` reflect any rename override (`SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` in `--csv` mode, `--rename-subject`/`--rename-experiment` in `-1`/`--accession` mode — see [Manual Interventions](../manual.md)), i.e. the on-disk directory names actually written under `OUTPUT_DIR` (for `--accession` without a rename, this is the resolved XNAT ID, not a label), not necessarily the original XNAT labels. One row is appended per processed experiment; rows are written under a lock so concurrent workers do not interleave.
|
|
114
|
+
`DATESTAMP` is the per-experiment download attempt begin time, formatted to match Python's `logging` module default `asctime` (`YYYY-MM-DD HH:MM:SS,mmm`, local time). `USER` is the OS-level username running the command (via Python's `getpass.getuser()`). `SUBJECT`/`EXPERIMENT` reflect any rename override (`SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` in `--csv` mode, `--rename-subject`/`--rename-experiment` in `-1`/`--accession` mode — see [Manual Interventions](../manual.md)), i.e. the on-disk directory names actually written under `OUTPUT_DIR` (for `--accession` without a rename, this is the resolved XNAT ID, not a label), not necessarily the original XNAT labels. One row is appended per processed experiment, including skipped ones (one row per CSV row for `DUPLICATE`, `RENAME_CONFLICT` and `COLLISION`); rows are written under a lock so concurrent workers do not interleave.
|
|
@@ -33,7 +33,7 @@ With `-n/--nphysio` > 1, the `phys2bids` conversions run in parallel across sepa
|
|
|
33
33
|
| `CONVERTED` | `phys2bids` read the raw file and its `_physio.tsv.gz`/`.json` were written next to the paired `.nii.gz`. |
|
|
34
34
|
| `SKIPPED` | The expected single-frequency `_physio.tsv.gz` already existed at its destination and was a valid, non-corrupt GZIP file; conversion was not attempted. |
|
|
35
35
|
| `NOT_PHYSIO` | The referenced file matched a supported extension but could not be loaded as physiological data. |
|
|
36
|
-
| `READER_MISSING` | The optional reader package `phys2bids` needs for this format is not installed (e.g. `bioread` for `.acq`, `
|
|
36
|
+
| `READER_MISSING` | The optional reader package `phys2bids` needs for this format is not installed (e.g. `bioread` for `.acq`, `pymatreader` for `.mat`, `sonpy` for `.smr`). This is an environment problem, not a data problem — install the package and re-run. |
|
|
37
37
|
| `CONVERT_ERROR` | `phys2bids` raised while converting, produced no `.tsv.gz` output, or its destination was already occupied by a different association. |
|
|
38
38
|
| `SOURCE_MISSING` | `PhysioParent` was unset/not a directory, the named basename was not found under it, or the `mriconvert_qc.tsv` row is missing `participant_id`/`datatype`. |
|
|
39
39
|
| `COLLISION` | This `physio` basename is referenced by more than one `mriconvert_qc.tsv` row; none were converted until resolved. |
|
|
@@ -46,5 +46,3 @@ Exit code is `1` if any association is `CONVERT_ERROR` or `READER_MISSING`, and
|
|
|
46
46
|
| `-p`, `--project` | **Required.** Project directory name under `OUTPUT_DIR` identifying the BIDS dataset produced by `xnatbidscli mriconvert`. |
|
|
47
47
|
| `-n`, `--nphysio` | *Optional.* Number of physio conversions to run in parallel, one `phys2bids` conversion per process (default `1`). |
|
|
48
48
|
| `-l`, `--log` | *Optional.* Write a per-association log CSV to `OUTPUT_DIR/log/physioconvert_<YYYYMMDD_HHMMSS>_log.csv` (header `DATESTAMP,USER,STATUS,MRI_FILENAME,PHYSIO_SOURCE,DESTINATION_PATH`). One row per processed association, except a conversion that produced several outputs emits one row per output; associations with no output get a single blank-`DESTINATION_PATH` row. Also mirrors everything printed to stdout/stderr into a companion text log at `OUTPUT_DIR/log/physioconvert_<YYYYMMDD_HHMMSS>_log.txt`, the Python equivalent of piping through `tee`. Off by default. |
|
|
49
|
-
|
|
50
|
-
> **Note:** `physioconvert` needs Python 3.11. `phys2bids` 2.10.0, its latest release, caps `numpy` at `<1.24`, and `numpy` 1.23 has no builds for Python 3.12 or newer. So `phys2bids` (and `bioread`) install with `xnatbidscli` only on Python 3.11, where `nibabel` is held below 5.4. On Python 3.12 or newer, every other subcommand works and `physioconvert` exits with a message. This limit goes away once `phys2bids` releases its uncapped `master` branch.
|
|
@@ -25,7 +25,7 @@ Click a question to expand its answer, or use the buttons below to open or close
|
|
|
25
25
|
|
|
26
26
|
??? question "Why does physioconvert say phys2bids is required?"
|
|
27
27
|
|
|
28
|
-
`phys2bids`
|
|
28
|
+
`phys2bids` could not be imported in your environment. Since `xnatbidscli` 2.1.0, `phys2bids` installs with `xnatbidscli` on every supported Python version. Older releases installed it only on Python 3.11. Upgrade with `pip install --upgrade xnatbidscli`.
|
|
29
29
|
|
|
30
30
|
**References:**
|
|
31
31
|
|
|
@@ -69,6 +69,14 @@ Click a question to expand its answer, or use the buttons below to open or close
|
|
|
69
69
|
|
|
70
70
|
- [`xnatbidscli download`](cli/download.md)
|
|
71
71
|
|
|
72
|
+
??? question "What happens if I re-run download on experiments I already downloaded?"
|
|
73
|
+
|
|
74
|
+
An experiment whose `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` directory exists and is not empty is skipped with a `WARNING` and reported `SKIPPED`. It is not downloaded, archived or deleted. To download it again, pass `--overwrite`, which deletes the directory first and, with `-a`, replaces its archive once the download is `COMPLETE`. Because `-a -d` removes the directory, a re-run after it downloads again. In `--csv` mode, an experiment listed on more than one row, or two experiments renamed into the same directory, are skipped with a `WARNING` too (`DUPLICATE`, `RENAME_CONFLICT`, `COLLISION`). None of these skips change the exit code.
|
|
75
|
+
|
|
76
|
+
**References:**
|
|
77
|
+
|
|
78
|
+
- [`xnatbidscli download` — repeated experiments](cli/download.md#repeated-experiments-in-the-csv)
|
|
79
|
+
|
|
72
80
|
??? question "How do I archive downloaded data?"
|
|
73
81
|
|
|
74
82
|
Pass `-a`/`--archive` to `download` (or `mriconvert`) to tar+gzip each experiment's directory into `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz` once it's processed; add `-d`/`--delete` to remove the original directory after the archive is confirmed written.
|
|
@@ -8,9 +8,7 @@ pip install xnatbidscli
|
|
|
8
8
|
|
|
9
9
|
This puts the `xnatbidscli` command on your `PATH`.
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
Runtime dependencies (installed automatically): `pyxnat`, `dcm2bids`, `dcm2niix` (the [`dcm2niix`](https://pypi.org/project/dcm2niix/) PyPI package vendors the binary onto your `PATH`), `pydicom`, `cubids`, `nibabel` (used to read NIfTI shapes for the `mriconvert` `scans.tsv`), `pandas` (used by `bidsmap`), and `phys2bids` (used by `physioconvert` to read physiological recordings and write BIDS physio files; Python 3.11 only, see the note under [`xnatbidscli physioconvert`](cli/physioconvert.md)). `bioread` is also pulled in on Python 3.11 for `phys2bids` to read BIOPAC `.acq` files (phys2bids imports it lazily but does not depend on it directly).
|
|
11
|
+
Runtime dependencies (installed automatically): `pyxnat`, `dcm2bids`, `dcm2niix` (the [`dcm2niix`](https://pypi.org/project/dcm2niix/) PyPI package vendors the binary onto your `PATH`), `pydicom`, `cubids`, `nibabel` (used to read NIfTI shapes for the `mriconvert` `scans.tsv`), `pandas` (used by `bidsmap`), and `phys2bids` (used by `physioconvert` to read physiological recordings and write BIDS physio files). `bioread` and `pymatreader` are also pulled in for `phys2bids` to read BIOPAC `.acq` and MATLAB `.mat` files (phys2bids imports them lazily but does not depend on them directly).
|
|
14
12
|
|
|
15
13
|
## Latest unreleased features
|
|
16
14
|
|
|
@@ -44,6 +44,7 @@ After an optional `sub-`/`ses-` prefix is stripped, the remaining value must be
|
|
|
44
44
|
|
|
45
45
|
- In `--csv` mode, `download` validates every row of the input CSV before downloading anything; if any `SUBJECT_BIDS_RENAME` or `EXPERIMENT_BIDS_RENAME` value fails this check, the entire run exits with an error and nothing is downloaded. Fix the offending cell(s) in the CSV and re-run `download`.
|
|
46
46
|
- In `-1`/`--accession` mode, an invalid `--rename-subject`/`--rename-experiment` value exits with an error before downloading.
|
|
47
|
+
- In `--csv` mode, an experiment listed on more than one row, with the same or different renames, or two experiments renamed into the same directory, are skipped with a `WARNING` rather than downloaded. See [Repeated experiments in the CSV](cli/download.md#repeated-experiments-in-the-csv).
|
|
47
48
|
|
|
48
49
|
`--rename-subject`/`--rename-experiment` only apply to `-1`/`--accession` mode; supplying either alongside `--csv` is an error. `--rename-experiment` is additionally rejected when `--accession` resolves to a subject rather than an experiment.
|
|
49
50
|
|
|
@@ -19,24 +19,24 @@ classifiers = [
|
|
|
19
19
|
"Programming Language :: Python :: 3",
|
|
20
20
|
"Programming Language :: Python :: 3 :: Only",
|
|
21
21
|
"Programming Language :: Python :: 3.11",
|
|
22
|
+
"Programming Language :: Python :: 3.12",
|
|
23
|
+
"Programming Language :: Python :: 3.13",
|
|
24
|
+
"Programming Language :: Python :: 3.14",
|
|
22
25
|
"Topic :: Scientific/Engineering :: Bio-Informatics",
|
|
23
26
|
"Topic :: Scientific/Engineering :: Medical Science Apps.",
|
|
24
27
|
]
|
|
25
|
-
#
|
|
26
|
-
#
|
|
27
|
-
# Python 3.11, where nibabel is held below 5.4 (5.4 needs numpy>=1.25). Drop
|
|
28
|
-
# these markers once phys2bids releases its uncapped master.
|
|
28
|
+
# bioread and pymatreader are the readers phys2bids imports lazily for .acq and
|
|
29
|
+
# .mat files; it does not depend on them directly.
|
|
29
30
|
dependencies = [
|
|
30
|
-
"bioread>=3
|
|
31
|
+
"bioread>=3",
|
|
31
32
|
"cubids>=1.2.1",
|
|
32
33
|
"dcm2bids>=3",
|
|
33
34
|
"dcm2niix>=1.0.20260416",
|
|
34
|
-
"nibabel>=5.
|
|
35
|
-
"nibabel>=5.4.2; python_version >= '3.12'",
|
|
36
|
-
"numpy<2",
|
|
35
|
+
"nibabel>=5.4.2",
|
|
37
36
|
"pandas>=2,<2.4",
|
|
38
|
-
"phys2bids>=2.
|
|
37
|
+
"phys2bids>=2.11",
|
|
39
38
|
"pydicom>=2.4",
|
|
39
|
+
"pymatreader>=1",
|
|
40
40
|
"pyxnat>=1.6",
|
|
41
41
|
"requests>=2",
|
|
42
42
|
]
|
|
@@ -21,20 +21,26 @@ def archive_path(
|
|
|
21
21
|
|
|
22
22
|
|
|
23
23
|
def archive_experiment(
|
|
24
|
-
data_root: Path,
|
|
24
|
+
data_root: Path,
|
|
25
|
+
project: str,
|
|
26
|
+
subject: str,
|
|
27
|
+
experiment: str,
|
|
28
|
+
overwrite: bool = False,
|
|
25
29
|
) -> tuple[str, str | None]:
|
|
26
30
|
"""Tar+gzip data_root/PROJECT/SUBJECT/EXPERIMENT into data_root/archive/.
|
|
27
31
|
|
|
28
32
|
Returns (status, detail). Existing archives are left untouched and reported
|
|
29
|
-
as SKIPPED; the tarball is written to a .tmp
|
|
30
|
-
only on success so an interrupted run never
|
|
33
|
+
as SKIPPED unless `overwrite` is set; the tarball is written to a .tmp
|
|
34
|
+
sibling and renamed into place only on success so an interrupted run never
|
|
35
|
+
leaves a partial file behind (or loses the archive it was replacing).
|
|
31
36
|
"""
|
|
32
37
|
src = data_root / project / subject / experiment
|
|
33
38
|
if not src.is_dir():
|
|
34
39
|
return STATUS_NONEXISTENT, f"source directory not found: {src}"
|
|
35
40
|
|
|
36
41
|
dest = archive_path(data_root, project, subject, experiment)
|
|
37
|
-
|
|
42
|
+
replacing = dest.exists()
|
|
43
|
+
if replacing and not overwrite:
|
|
38
44
|
return STATUS_SKIPPED, f"archive already exists: {dest}"
|
|
39
45
|
|
|
40
46
|
dest.parent.mkdir(parents=True, exist_ok=True)
|
|
@@ -50,6 +56,8 @@ def archive_experiment(
|
|
|
50
56
|
except Exception:
|
|
51
57
|
pass
|
|
52
58
|
return STATUS_FAILURE, f"error creating archive: {e}"
|
|
59
|
+
if replacing:
|
|
60
|
+
return STATUS_COMPLETE, f"replaced existing archive: {dest}"
|
|
53
61
|
return STATUS_COMPLETE, None
|
|
54
62
|
|
|
55
63
|
|
|
@@ -197,6 +197,17 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
197
197
|
"OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT directory. The SUBJECT and "
|
|
198
198
|
"PROJECT parent directories are also removed if they become empty.",
|
|
199
199
|
)
|
|
200
|
+
download_parser.add_argument(
|
|
201
|
+
"--overwrite",
|
|
202
|
+
action="store_true",
|
|
203
|
+
help="Delete a non-empty OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT "
|
|
204
|
+
"directory and download it again. Without this flag, such an "
|
|
205
|
+
"experiment is skipped with a warning (STATUS SKIPPED), along with "
|
|
206
|
+
"-a/--archive and -d/--delete for it. The directory is deleted only "
|
|
207
|
+
"after the experiment is found on XNAT. With -a/--archive, an "
|
|
208
|
+
"existing archive is also replaced, but only after a COMPLETE "
|
|
209
|
+
"download.",
|
|
210
|
+
)
|
|
200
211
|
download_parser.set_defaults(func=download_cmd)
|
|
201
212
|
|
|
202
213
|
mriconfig_parser = subparsers.add_parser(
|
|
@@ -20,6 +20,7 @@ from pyxnat.core import uriutil
|
|
|
20
20
|
from .archive import (
|
|
21
21
|
OK_STATUSES as ARCHIVE_OK_STATUSES,
|
|
22
22
|
archive_experiment,
|
|
23
|
+
archive_path,
|
|
23
24
|
delete_experiment_dir,
|
|
24
25
|
)
|
|
25
26
|
from .login import load_credentials
|
|
@@ -29,8 +30,32 @@ STATUS_COMPLETE = "COMPLETE"
|
|
|
29
30
|
STATUS_FAILURE = "FAILURE"
|
|
30
31
|
STATUS_NONEXISTENT = "NONEXISTENT"
|
|
31
32
|
STATUS_EMPTY = "EMPTY"
|
|
33
|
+
STATUS_SKIPPED = "SKIPPED"
|
|
34
|
+
STATUS_DUPLICATE = "DUPLICATE"
|
|
35
|
+
STATUS_RENAME_CONFLICT = "RENAME_CONFLICT"
|
|
36
|
+
STATUS_COLLISION = "COLLISION"
|
|
37
|
+
|
|
38
|
+
_OK_STATUSES = {
|
|
39
|
+
STATUS_COMPLETE,
|
|
40
|
+
STATUS_EMPTY,
|
|
41
|
+
STATUS_SKIPPED,
|
|
42
|
+
STATUS_DUPLICATE,
|
|
43
|
+
STATUS_RENAME_CONFLICT,
|
|
44
|
+
STATUS_COLLISION,
|
|
45
|
+
}
|
|
46
|
+
_SUMMARY_STATUSES = (
|
|
47
|
+
STATUS_COMPLETE,
|
|
48
|
+
STATUS_FAILURE,
|
|
49
|
+
STATUS_NONEXISTENT,
|
|
50
|
+
STATUS_EMPTY,
|
|
51
|
+
STATUS_SKIPPED,
|
|
52
|
+
STATUS_DUPLICATE,
|
|
53
|
+
STATUS_RENAME_CONFLICT,
|
|
54
|
+
STATUS_COLLISION,
|
|
55
|
+
)
|
|
32
56
|
|
|
33
|
-
|
|
57
|
+
# (PROJECT, SUBJECT_LABEL, EXPERIMENT_LABEL, SUBJECT_BIDS_RENAME, EXPERIMENT_BIDS_RENAME)
|
|
58
|
+
CsvRow = tuple[str, str, str, str | None, str | None]
|
|
34
59
|
|
|
35
60
|
_thread_iface = threading.local()
|
|
36
61
|
|
|
@@ -418,6 +443,40 @@ def _extract_zip_flattened_or_rescue(
|
|
|
418
443
|
zip_path.replace(dest_dir / filename)
|
|
419
444
|
|
|
420
445
|
|
|
446
|
+
def _has_contents(path: Path) -> bool:
|
|
447
|
+
"""Return True if `path` is a directory with at least one entry in it."""
|
|
448
|
+
try:
|
|
449
|
+
return path.is_dir() and any(path.iterdir())
|
|
450
|
+
except OSError:
|
|
451
|
+
return False
|
|
452
|
+
|
|
453
|
+
|
|
454
|
+
def _skip_existing(
|
|
455
|
+
label: str, experiment_root: Path, do_archive: bool, do_delete: bool
|
|
456
|
+
) -> bool:
|
|
457
|
+
"""Warn and return True if `experiment_root` was already downloaded.
|
|
458
|
+
|
|
459
|
+
A non-empty experiment directory counts as already downloaded. This is a
|
|
460
|
+
local check only, made before contacting XNAT. Archiving/deletion are
|
|
461
|
+
skipped along with the download, so the warning is loud when they were
|
|
462
|
+
requested.
|
|
463
|
+
"""
|
|
464
|
+
if not _has_contents(experiment_root):
|
|
465
|
+
return False
|
|
466
|
+
_safe_print(
|
|
467
|
+
f"WARNING: {label}: {experiment_root} already exists and is not "
|
|
468
|
+
"empty; skipping download. Pass --overwrite to delete it and "
|
|
469
|
+
"download again."
|
|
470
|
+
)
|
|
471
|
+
if do_archive:
|
|
472
|
+
steps = "ARCHIVE (-a) AND DELETE (-d)" if do_delete else "ARCHIVE (-a)"
|
|
473
|
+
_safe_print(
|
|
474
|
+
f"WARNING: {steps} NOT RUN for {label}: its download was skipped, "
|
|
475
|
+
f"so {experiment_root} and any existing archive were left untouched."
|
|
476
|
+
)
|
|
477
|
+
return True
|
|
478
|
+
|
|
479
|
+
|
|
421
480
|
def _process_experiment(
|
|
422
481
|
interface: Interface,
|
|
423
482
|
project: str,
|
|
@@ -428,6 +487,7 @@ def _process_experiment(
|
|
|
428
487
|
progress: _ExperimentProgress | None = None,
|
|
429
488
|
local_subject: str | None = None,
|
|
430
489
|
local_experiment: str | None = None,
|
|
490
|
+
overwrite: bool = False,
|
|
431
491
|
) -> str:
|
|
432
492
|
"""Download one experiment as whole-experiment zip archives.
|
|
433
493
|
|
|
@@ -444,6 +504,10 @@ def _process_experiment(
|
|
|
444
504
|
`EXPERIMENT_BIDS_RENAME`) name the on-disk directory when they differ
|
|
445
505
|
from `subject`/`experiment`, which are always used to look up the
|
|
446
506
|
experiment on the XNAT server itself.
|
|
507
|
+
|
|
508
|
+
With `overwrite`, a non-empty existing experiment directory is deleted
|
|
509
|
+
before downloading, but only once the experiment is confirmed to exist
|
|
510
|
+
on XNAT, so a lookup failure never costs the local copy.
|
|
447
511
|
"""
|
|
448
512
|
label = f"{project}/{subject}/{experiment}"
|
|
449
513
|
try:
|
|
@@ -461,6 +525,13 @@ def _process_experiment(
|
|
|
461
525
|
experiment_root = (
|
|
462
526
|
output_dir / project / (local_subject or subject) / (local_experiment or experiment)
|
|
463
527
|
)
|
|
528
|
+
if overwrite and _has_contents(experiment_root):
|
|
529
|
+
report(f"WARNING: {label}: deleting existing {experiment_root} before re-downloading (--overwrite).")
|
|
530
|
+
try:
|
|
531
|
+
shutil.rmtree(experiment_root)
|
|
532
|
+
except OSError as e:
|
|
533
|
+
report(f" Error deleting {experiment_root}: {e}")
|
|
534
|
+
return STATUS_FAILURE
|
|
464
535
|
experiment_root.mkdir(parents=True, exist_ok=True)
|
|
465
536
|
|
|
466
537
|
got_scans = False
|
|
@@ -507,12 +578,10 @@ def _format_bids_rename(value: str, prefix: str) -> str | None:
|
|
|
507
578
|
return value if value.startswith(prefix) else prefix + value
|
|
508
579
|
|
|
509
580
|
|
|
510
|
-
def _read_csv_rows(
|
|
511
|
-
path: Path,
|
|
512
|
-
) -> list[tuple[str, str, str, str | None, str | None]]:
|
|
581
|
+
def _read_csv_rows(path: Path) -> list[CsvRow]:
|
|
513
582
|
if not path.exists():
|
|
514
583
|
sys.exit(f"Error: input CSV not found: {path}")
|
|
515
|
-
rows: list[
|
|
584
|
+
rows: list[CsvRow] = []
|
|
516
585
|
with path.open(newline="") as f:
|
|
517
586
|
reader = csv.DictReader(f)
|
|
518
587
|
required = {"PROJECT", "SUBJECT_LABEL", "EXPERIMENT_LABEL"}
|
|
@@ -557,6 +626,76 @@ def _read_csv_rows(
|
|
|
557
626
|
return rows
|
|
558
627
|
|
|
559
628
|
|
|
629
|
+
def _screen_csv_rows(
|
|
630
|
+
rows: list[CsvRow],
|
|
631
|
+
) -> tuple[list[CsvRow], list[tuple[CsvRow, str]]]:
|
|
632
|
+
"""Split CSV rows into ones to download and ones skipped with a WARNING.
|
|
633
|
+
|
|
634
|
+
An experiment is identified by (PROJECT, SUBJECT_LABEL, EXPERIMENT_LABEL).
|
|
635
|
+
Every row of an experiment listed more than once is skipped: DUPLICATE
|
|
636
|
+
if all its rows have the same (normalized) renames, RENAME_CONFLICT if
|
|
637
|
+
not. Every remaining row whose on-disk directory is shared with a
|
|
638
|
+
different experiment is skipped as COLLISION.
|
|
639
|
+
|
|
640
|
+
Returns
|
|
641
|
+
-------
|
|
642
|
+
tuple[list[CsvRow], list[tuple[CsvRow, str]]]
|
|
643
|
+
Rows to download, and skipped rows paired with their STATUS.
|
|
644
|
+
"""
|
|
645
|
+
by_experiment: dict[tuple[str, str, str], list[CsvRow]] = {}
|
|
646
|
+
for row in rows:
|
|
647
|
+
by_experiment.setdefault(row[:3], []).append(row)
|
|
648
|
+
|
|
649
|
+
flagged: dict[tuple[str, str, str], str] = {}
|
|
650
|
+
for key, group in by_experiment.items():
|
|
651
|
+
if len(group) < 2:
|
|
652
|
+
continue
|
|
653
|
+
label = "/".join(key)
|
|
654
|
+
if len({row[3:] for row in group}) == 1:
|
|
655
|
+
flagged[key] = STATUS_DUPLICATE
|
|
656
|
+
print(
|
|
657
|
+
f"WARNING: {label} is listed {len(group)} times in the input "
|
|
658
|
+
"CSV; skipping every instance. Remove the extra row(s) to "
|
|
659
|
+
"download it."
|
|
660
|
+
)
|
|
661
|
+
else:
|
|
662
|
+
flagged[key] = STATUS_RENAME_CONFLICT
|
|
663
|
+
renames = "; ".join(
|
|
664
|
+
f"SUBJECT_BIDS_RENAME={sr or '(blank)'}, "
|
|
665
|
+
f"EXPERIMENT_BIDS_RENAME={er or '(blank)'}"
|
|
666
|
+
for _, _, _, sr, er in group
|
|
667
|
+
)
|
|
668
|
+
print(
|
|
669
|
+
f"WARNING: {label} is listed {len(group)} times in the input "
|
|
670
|
+
"CSV with different SUBJECT_BIDS_RENAME/EXPERIMENT_BIDS_RENAME "
|
|
671
|
+
f"values ({renames}); skipping every instance. Keep only one "
|
|
672
|
+
"row to download it."
|
|
673
|
+
)
|
|
674
|
+
|
|
675
|
+
# Compared case-insensitively, since Windows and macOS filesystems would
|
|
676
|
+
# merge directories that differ only in case.
|
|
677
|
+
by_local_dir: dict[tuple[str, ...], dict[tuple[str, str, str], str]] = {}
|
|
678
|
+
for p, s, e, sr, er in rows:
|
|
679
|
+
local = f"{p}/{sr or s}/{er or e}"
|
|
680
|
+
key = tuple(part.casefold() for part in (p, sr or s, er or e))
|
|
681
|
+
by_local_dir.setdefault(key, {})[(p, s, e)] = local
|
|
682
|
+
for experiments in by_local_dir.values():
|
|
683
|
+
if len(experiments) < 2:
|
|
684
|
+
continue
|
|
685
|
+
labels = ", ".join("/".join(k) for k in experiments)
|
|
686
|
+
print(
|
|
687
|
+
f"WARNING: {labels} would all download into "
|
|
688
|
+
f"{next(iter(experiments.values()))}; skipping all of them. Give "
|
|
689
|
+
"each a unique SUBJECT_BIDS_RENAME/EXPERIMENT_BIDS_RENAME."
|
|
690
|
+
)
|
|
691
|
+
for key in experiments:
|
|
692
|
+
flagged.setdefault(key, STATUS_COLLISION)
|
|
693
|
+
|
|
694
|
+
to_download = [row for row in rows if row[:3] not in flagged]
|
|
695
|
+
skipped = [(row, flagged[row[:3]]) for row in rows if row[:3] in flagged]
|
|
696
|
+
return to_download, skipped
|
|
697
|
+
|
|
698
|
+
|
|
560
699
|
def _archive_and_maybe_delete(
|
|
561
700
|
output_dir: Path,
|
|
562
701
|
project: str,
|
|
@@ -565,12 +704,30 @@ def _archive_and_maybe_delete(
|
|
|
565
704
|
do_archive: bool,
|
|
566
705
|
do_delete: bool,
|
|
567
706
|
report: Callable[[str], None],
|
|
707
|
+
overwrite: bool = False,
|
|
708
|
+
download_status: str = STATUS_COMPLETE,
|
|
568
709
|
) -> None:
|
|
710
|
+
"""Archive (and with `do_delete`, then delete) one experiment directory.
|
|
711
|
+
|
|
712
|
+
With `overwrite`, an existing archive is replaced, but only after a
|
|
713
|
+
COMPLETE download, so a failed or partial re-download never replaces a
|
|
714
|
+
good archive.
|
|
715
|
+
"""
|
|
569
716
|
if not do_archive:
|
|
570
717
|
return
|
|
571
718
|
label = f"{project}/{subject}/{experiment}"
|
|
719
|
+
replace = overwrite and download_status == STATUS_COMPLETE
|
|
720
|
+
if (
|
|
721
|
+
overwrite
|
|
722
|
+
and not replace
|
|
723
|
+
and archive_path(output_dir, project, subject, experiment).exists()
|
|
724
|
+
):
|
|
725
|
+
report(
|
|
726
|
+
f"WARNING: not replacing the existing archive for {label} "
|
|
727
|
+
f"(--overwrite), since its download was {download_status}."
|
|
728
|
+
)
|
|
572
729
|
a_status, a_detail = archive_experiment(
|
|
573
|
-
output_dir, project, subject, experiment
|
|
730
|
+
output_dir, project, subject, experiment, overwrite=replace
|
|
574
731
|
)
|
|
575
732
|
line = f" archive {label}: {a_status}"
|
|
576
733
|
if a_detail:
|
|
@@ -593,15 +750,23 @@ def _run_single(
|
|
|
593
750
|
do_delete: bool,
|
|
594
751
|
local_subject: str | None = None,
|
|
595
752
|
local_experiment: str | None = None,
|
|
753
|
+
overwrite: bool = False,
|
|
596
754
|
) -> str:
|
|
597
755
|
local_s = local_subject or subject
|
|
598
756
|
local_e = local_experiment or experiment
|
|
757
|
+
label = f"{project}/{subject}/{experiment}"
|
|
758
|
+
if not overwrite and _skip_existing(
|
|
759
|
+
label, output_dir / project / local_s / local_e, do_archive, do_delete
|
|
760
|
+
):
|
|
761
|
+
log_writer.write(_logging_now(), project, local_s, local_e, STATUS_SKIPPED)
|
|
762
|
+
return STATUS_SKIPPED
|
|
599
763
|
iface = Interface(server=server, user=user, password=password)
|
|
600
764
|
try:
|
|
601
765
|
start = _logging_now()
|
|
602
766
|
status = _process_experiment(
|
|
603
767
|
iface, project, subject, experiment, output_dir, _safe_print,
|
|
604
768
|
local_subject=local_s, local_experiment=local_e,
|
|
769
|
+
overwrite=overwrite,
|
|
605
770
|
)
|
|
606
771
|
finally:
|
|
607
772
|
try:
|
|
@@ -617,6 +782,8 @@ def _run_single(
|
|
|
617
782
|
do_archive,
|
|
618
783
|
do_delete,
|
|
619
784
|
_safe_print,
|
|
785
|
+
overwrite,
|
|
786
|
+
status,
|
|
620
787
|
)
|
|
621
788
|
return status
|
|
622
789
|
|
|
@@ -625,27 +792,28 @@ def _run_csv(
|
|
|
625
792
|
server: str,
|
|
626
793
|
user: str,
|
|
627
794
|
password: str,
|
|
628
|
-
rows: list[
|
|
795
|
+
rows: list[CsvRow],
|
|
629
796
|
output_dir: Path,
|
|
630
797
|
n_parallel_experiments: int,
|
|
631
798
|
log_writer: _LogWriter,
|
|
632
799
|
do_archive: bool,
|
|
633
800
|
do_delete: bool,
|
|
801
|
+
overwrite: bool = False,
|
|
634
802
|
) -> dict[str, int]:
|
|
635
|
-
counts =
|
|
636
|
-
|
|
637
|
-
|
|
638
|
-
STATUS_NONEXISTENT: 0,
|
|
639
|
-
STATUS_EMPTY: 0,
|
|
640
|
-
}
|
|
641
|
-
|
|
642
|
-
def _worker(row: tuple[str, str, str, str | None, str | None]) -> str:
|
|
803
|
+
counts = dict.fromkeys(_SUMMARY_STATUSES, 0)
|
|
804
|
+
|
|
805
|
+
def _worker(row: CsvRow) -> str:
|
|
643
806
|
p, s, e, subject_rename, experiment_rename = row
|
|
644
807
|
local_s = subject_rename or s
|
|
645
808
|
local_e = experiment_rename or e
|
|
646
|
-
iface = _get_thread_interface(server, user, password)
|
|
647
809
|
label = f"{p}/{s}/{e}"
|
|
648
810
|
experiment_root = output_dir / p / local_s / local_e
|
|
811
|
+
if not overwrite and _skip_existing(
|
|
812
|
+
label, experiment_root, do_archive, do_delete
|
|
813
|
+
):
|
|
814
|
+
log_writer.write(_logging_now(), p, local_s, local_e, STATUS_SKIPPED)
|
|
815
|
+
return STATUS_SKIPPED
|
|
816
|
+
iface = _get_thread_interface(server, user, password)
|
|
649
817
|
progress = _ExperimentProgress()
|
|
650
818
|
stop_event = threading.Event()
|
|
651
819
|
monitor = threading.Thread(
|
|
@@ -658,14 +826,15 @@ def _run_csv(
|
|
|
658
826
|
start = _logging_now()
|
|
659
827
|
status = _process_experiment(
|
|
660
828
|
iface, p, s, e, output_dir, _safe_print, progress,
|
|
661
|
-
local_s, local_e,
|
|
829
|
+
local_s, local_e, overwrite,
|
|
662
830
|
)
|
|
663
831
|
finally:
|
|
664
832
|
stop_event.set()
|
|
665
833
|
monitor.join()
|
|
666
834
|
log_writer.write(start, p, local_s, local_e, status)
|
|
667
835
|
_archive_and_maybe_delete(
|
|
668
|
-
output_dir, p, local_s, local_e, do_archive, do_delete, _safe_print
|
|
836
|
+
output_dir, p, local_s, local_e, do_archive, do_delete, _safe_print,
|
|
837
|
+
overwrite, status,
|
|
669
838
|
)
|
|
670
839
|
return status
|
|
671
840
|
|
|
@@ -714,6 +883,33 @@ def _run_csv(
|
|
|
714
883
|
return counts
|
|
715
884
|
|
|
716
885
|
|
|
886
|
+
def _print_summary(
|
|
887
|
+
heading: str,
|
|
888
|
+
counts: dict[str, int],
|
|
889
|
+
log_path: Path | None,
|
|
890
|
+
do_archive: bool,
|
|
891
|
+
do_delete: bool,
|
|
892
|
+
) -> int:
|
|
893
|
+
"""Print a batch run's per-STATUS counts and return its exit code.
|
|
894
|
+
|
|
895
|
+
Repeats the not-archived warning for SKIPPED experiments at the end,
|
|
896
|
+
since the per-experiment warnings may have scrolled away.
|
|
897
|
+
"""
|
|
898
|
+
print(f"\n{heading}")
|
|
899
|
+
for status in _SUMMARY_STATUSES:
|
|
900
|
+
print(f" {status}: {counts[status]}")
|
|
901
|
+
if do_archive and counts[STATUS_SKIPPED]:
|
|
902
|
+
steps = "ARCHIVED OR DELETED" if do_delete else "ARCHIVED"
|
|
903
|
+
print(
|
|
904
|
+
f"WARNING: {counts[STATUS_SKIPPED]} SKIPPED experiment(s) were NOT "
|
|
905
|
+
f"{steps}, since their existing directories were left untouched."
|
|
906
|
+
)
|
|
907
|
+
if log_path is not None:
|
|
908
|
+
print(f"Log written to {log_path}")
|
|
909
|
+
bad = counts[STATUS_FAILURE] + counts[STATUS_NONEXISTENT]
|
|
910
|
+
return 0 if bad == 0 else 1
|
|
911
|
+
|
|
912
|
+
|
|
717
913
|
def _resolve_accession(
|
|
718
914
|
interface: Interface, accession: str
|
|
719
915
|
) -> tuple[str, str, str, str | None]:
|
|
@@ -810,6 +1006,7 @@ def _download_accession(
|
|
|
810
1006
|
local_subject: str | None,
|
|
811
1007
|
local_experiment: str | None,
|
|
812
1008
|
log_path: Path | None,
|
|
1009
|
+
overwrite: bool = False,
|
|
813
1010
|
) -> int:
|
|
814
1011
|
"""Handle `download --accession`: resolve it, then delegate like -1/--csv.
|
|
815
1012
|
|
|
@@ -860,26 +1057,18 @@ def _download_accession(
|
|
|
860
1057
|
]
|
|
861
1058
|
counts = _run_csv(
|
|
862
1059
|
server, user, password, rows, output_dir, n_parallel_experiments,
|
|
863
|
-
log_writer, do_archive, do_delete,
|
|
1060
|
+
log_writer, do_archive, do_delete, overwrite,
|
|
1061
|
+
)
|
|
1062
|
+
return _print_summary(
|
|
1063
|
+
f"Processed {sum(counts.values())} experiment(s) for subject "
|
|
1064
|
+
f"accession {accession}:",
|
|
1065
|
+
counts, log_path, do_archive, do_delete,
|
|
864
1066
|
)
|
|
865
|
-
total = sum(counts.values())
|
|
866
|
-
print(f"\nProcessed {total} experiment(s) for subject accession {accession}:")
|
|
867
|
-
for status in (
|
|
868
|
-
STATUS_COMPLETE,
|
|
869
|
-
STATUS_FAILURE,
|
|
870
|
-
STATUS_NONEXISTENT,
|
|
871
|
-
STATUS_EMPTY,
|
|
872
|
-
):
|
|
873
|
-
print(f" {status}: {counts[status]}")
|
|
874
|
-
if log_path is not None:
|
|
875
|
-
print(f"Log written to {log_path}")
|
|
876
|
-
bad = counts[STATUS_FAILURE] + counts[STATUS_NONEXISTENT]
|
|
877
|
-
return 0 if bad == 0 else 1
|
|
878
1067
|
|
|
879
1068
|
status = _run_single(
|
|
880
1069
|
server, user, password, project, subject_id, experiment_id,
|
|
881
1070
|
output_dir, log_writer, do_archive, do_delete,
|
|
882
|
-
local_subject, local_experiment,
|
|
1071
|
+
local_subject, local_experiment, overwrite,
|
|
883
1072
|
)
|
|
884
1073
|
print(
|
|
885
1074
|
f"Status for accession {accession} "
|
|
@@ -958,6 +1147,7 @@ def download_cmd(args: argparse.Namespace) -> int:
|
|
|
958
1147
|
args.delete,
|
|
959
1148
|
local_subject,
|
|
960
1149
|
local_experiment,
|
|
1150
|
+
args.overwrite,
|
|
961
1151
|
)
|
|
962
1152
|
print(
|
|
963
1153
|
f"Status for {project}/{subject}/{experiment}: {status}"
|
|
@@ -980,9 +1170,10 @@ def download_cmd(args: argparse.Namespace) -> int:
|
|
|
980
1170
|
local_subject,
|
|
981
1171
|
local_experiment,
|
|
982
1172
|
log_path,
|
|
1173
|
+
args.overwrite,
|
|
983
1174
|
)
|
|
984
1175
|
|
|
985
|
-
rows = _read_csv_rows(Path(args.input))
|
|
1176
|
+
rows, screened = _screen_csv_rows(_read_csv_rows(Path(args.input)))
|
|
986
1177
|
counts = _run_csv(
|
|
987
1178
|
server,
|
|
988
1179
|
user,
|
|
@@ -993,17 +1184,12 @@ def download_cmd(args: argparse.Namespace) -> int:
|
|
|
993
1184
|
log_writer,
|
|
994
1185
|
args.archive,
|
|
995
1186
|
args.delete,
|
|
1187
|
+
args.overwrite,
|
|
1188
|
+
)
|
|
1189
|
+
for (p, s, e, sr, er), status in screened:
|
|
1190
|
+
log_writer.write(_logging_now(), p, sr or s, er or e, status)
|
|
1191
|
+
counts[status] += 1
|
|
1192
|
+
return _print_summary(
|
|
1193
|
+
f"Processed {sum(counts.values())} experiment(s):",
|
|
1194
|
+
counts, log_path, args.archive, args.delete,
|
|
996
1195
|
)
|
|
997
|
-
total = sum(counts.values())
|
|
998
|
-
print(f"\nProcessed {total} experiment(s):")
|
|
999
|
-
for status in (
|
|
1000
|
-
STATUS_COMPLETE,
|
|
1001
|
-
STATUS_FAILURE,
|
|
1002
|
-
STATUS_NONEXISTENT,
|
|
1003
|
-
STATUS_EMPTY,
|
|
1004
|
-
):
|
|
1005
|
-
print(f" {status}: {counts[status]}")
|
|
1006
|
-
if log_path is not None:
|
|
1007
|
-
print(f"Log written to {log_path}")
|
|
1008
|
-
bad = counts[STATUS_FAILURE] + counts[STATUS_NONEXISTENT]
|
|
1009
|
-
return 0 if bad == 0 else 1
|
|
@@ -54,7 +54,7 @@ STATUS_SKIPPED = "SKIPPED"
|
|
|
54
54
|
# is an environment problem, not a sign the file is not physiological data.
|
|
55
55
|
_READER_PACKAGES = {
|
|
56
56
|
".acq": "bioread",
|
|
57
|
-
".mat": "
|
|
57
|
+
".mat": "pymatreader",
|
|
58
58
|
".smr": "sonpy",
|
|
59
59
|
}
|
|
60
60
|
|
|
@@ -607,9 +607,8 @@ def physioconvert_cmd(args: argparse.Namespace) -> int:
|
|
|
607
607
|
from phys2bids.phys2bids import phys2bids as _p2b # noqa: F401
|
|
608
608
|
except ImportError:
|
|
609
609
|
sys.exit(
|
|
610
|
-
"Error: phys2bids is required for physioconvert.
|
|
611
|
-
"xnatbidscli
|
|
612
|
-
"newer numpy; reinstall xnatbidscli under Python 3.11."
|
|
610
|
+
"Error: phys2bids is required for physioconvert. "
|
|
611
|
+
"Reinstall xnatbidscli with 'pip install --upgrade xnatbidscli'."
|
|
613
612
|
)
|
|
614
613
|
|
|
615
614
|
# physioconvert only reads mriconvert_qc.tsv -- it never writes it back.
|
|
@@ -75,6 +75,13 @@ This is a manual test plan for exercising every `xnatbidscli` subcommand and its
|
|
|
75
75
|
- [ ] `-1` single-experiment mode ignores `SUBJECT_BIDS_RENAME`/`EXPERIMENT_BIDS_RENAME` entirely (no CSV involved)
|
|
76
76
|
- [ ] `-a/--archive` produces `OUTPUT_DIR/archive/PROJECT-<P>_SUBJECT-<S>_EXPERIMENT-<E>.tar.gz` after each experiment downloads
|
|
77
77
|
- [ ] Re-running with `-a/--archive` against an experiment whose archive already exists skips archiving with a warning (does not overwrite)
|
|
78
|
+
- [ ] Re-running against an experiment whose directory exists and is not empty prints a `WARNING`, reports `SKIPPED`, does not contact XNAT for it, and exits `0`; an empty directory still downloads
|
|
79
|
+
- [ ] With `-a` (and `-d`), a `SKIPPED` experiment prints a louder `WARNING` that archive/delete were not run, leaves its directory and archive untouched, and the summary repeats the count
|
|
80
|
+
- [ ] `--overwrite` deletes a non-empty experiment directory and downloads it fresh; a `NONEXISTENT` experiment's local directory is not deleted
|
|
81
|
+
- [ ] `--overwrite -a` replaces an existing archive after a `COMPLETE` download (archive `COMPLETE — replaced existing archive`); after a `FAILURE`/`EMPTY` download it keeps the old archive with a `WARNING`
|
|
82
|
+
- [ ] A CSV listing the same experiment twice with the same renames (`01` vs `sub-01` counts as the same) prints one `WARNING` and reports both rows `DUPLICATE`, downloading neither
|
|
83
|
+
- [ ] A CSV listing the same experiment twice with different renames (including blank vs. filled) prints one `WARNING` and reports both rows `RENAME_CONFLICT`, downloading neither
|
|
84
|
+
- [ ] Two different experiments renamed into the same directory (also when differing only in case) print one `WARNING` and are both reported `COLLISION`, downloading neither
|
|
78
85
|
- [ ] `-d/--delete` without `-a/--archive` fails immediately with "requires -a/--archive"
|
|
79
86
|
- [ ] `-a -d` together delete `OUTPUT_DIR/PROJECT/SUBJECT/EXPERIMENT` only after a successful archive
|
|
80
87
|
- [ ] `-a -d` also removes the `SUBJECT` and then `PROJECT` parent directories if they become empty
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|