metis-test-runner 0.2.0a0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- metis_test_runner-0.2.0a0/.gitignore +46 -0
- metis_test_runner-0.2.0a0/CHANGELOG.md +53 -0
- metis_test_runner-0.2.0a0/LICENSE +28 -0
- metis_test_runner-0.2.0a0/PKG-INFO +379 -0
- metis_test_runner-0.2.0a0/README.md +347 -0
- metis_test_runner-0.2.0a0/pyproject.toml +62 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/__init__.py +3 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/__main__.py +4 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/archive.py +745 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/examples/LMS_RAD_06.yaml +4016 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/examples/small_test.csv +13 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/examples/small_test.yaml +44 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/examples/small_test_img_n.yaml +64 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/gui.py +2252 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/indexes.py +6 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/paths.py +59 -0
- metis_test_runner-0.2.0a0/src/metis_test_runner/run_metis.py +1391 -0
- metis_test_runner-0.2.0a0/tests/conftest.py +29 -0
- metis_test_runner-0.2.0a0/tests/test_archive.py +1068 -0
- metis_test_runner-0.2.0a0/tests/test_gui.py +899 -0
- metis_test_runner-0.2.0a0/tests/test_run_metis.py +1442 -0
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# External repositories (cloned separately via metis-meta-package bootstrap)
|
|
2
|
+
METIS_Pipeline/
|
|
3
|
+
METIS_Simulations/
|
|
4
|
+
METIS_DRLD/
|
|
5
|
+
metis-meta-package/
|
|
6
|
+
MetisWISE/
|
|
7
|
+
inst_pkgs/
|
|
8
|
+
|
|
9
|
+
# Test/scratch YAML files (root-level only; examples/ are tracked)
|
|
10
|
+
/*.yaml
|
|
11
|
+
|
|
12
|
+
# Local test data and pipeline outputs
|
|
13
|
+
data/
|
|
14
|
+
output/
|
|
15
|
+
logs/
|
|
16
|
+
uv-cache/
|
|
17
|
+
EDPS_data/
|
|
18
|
+
|
|
19
|
+
# Claude Code internal files
|
|
20
|
+
.claude/
|
|
21
|
+
|
|
22
|
+
# Pipeline environment (generated by Install tab; contains local paths)
|
|
23
|
+
.env
|
|
24
|
+
|
|
25
|
+
# Python
|
|
26
|
+
__pycache__/
|
|
27
|
+
*.py[cod]
|
|
28
|
+
*.egg-info/
|
|
29
|
+
.venv/
|
|
30
|
+
dist/
|
|
31
|
+
build/
|
|
32
|
+
*.egg
|
|
33
|
+
uv.lock
|
|
34
|
+
*.log
|
|
35
|
+
|
|
36
|
+
# FITS files (large binary data)
|
|
37
|
+
*.fits
|
|
38
|
+
*.fits.gz
|
|
39
|
+
|
|
40
|
+
# OS / editor
|
|
41
|
+
.DS_Store
|
|
42
|
+
*.swp
|
|
43
|
+
*.swo
|
|
44
|
+
*~
|
|
45
|
+
.idea/
|
|
46
|
+
.vscode/
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.2.0
|
|
4
|
+
|
|
5
|
+
First PyPI release as `metis-test-runner`.
|
|
6
|
+
|
|
7
|
+
### Changed
|
|
8
|
+
- Restructured the project as a proper Python package under
|
|
9
|
+
`src/metis_test_runner/`. Modules are now imported as
|
|
10
|
+
`from metis_test_runner import gui, run_metis, archive`.
|
|
11
|
+
- Switched the build backend from `uv` to `hatchling`. The `[tool.uv]` and
|
|
12
|
+
`[dependency-groups]` sections are gone. Pipeline dependencies that live on
|
|
13
|
+
ESO mirrors (`pycpl`, `edps`, `pyesorex`, `adari_core`) are still installed
|
|
14
|
+
at runtime by the GUI's Install tab — they are not in `[project.dependencies]`
|
|
15
|
+
because they are not on PyPI.
|
|
16
|
+
- The GUI's Install tab and the Archive tab's MetisWISE installer now shell
|
|
17
|
+
out to `pip` (via `sys.executable -m pip`) with `--extra-index-url` for the
|
|
18
|
+
ESO and pycpl mirrors. `uv` is no longer required to use MTR.
|
|
19
|
+
- Runtime data (the `METIS_Pipeline` and `METIS_Simulations` clones, `.env`,
|
|
20
|
+
instrument packages, DRLD) now lives in a user data directory resolved via
|
|
21
|
+
`platformdirs` — `~/.local/share/metis-test-runner/` on Linux. Override the
|
|
22
|
+
location with the `METIS_DATA_DIR` environment variable. The per-asset env
|
|
23
|
+
vars (`METIS_META_PKG`, `METIS_SIMULATIONS_DIR`, `METIS_INST_PKGS`) still
|
|
24
|
+
work for back-compat.
|
|
25
|
+
- The CLI is now invoked as `mtr-cli` (or `python -m metis_test_runner.run_metis`)
|
|
26
|
+
instead of `python src/run_metis.py`.
|
|
27
|
+
|
|
28
|
+
### Added
|
|
29
|
+
- `mtr` and `mtr-cli` console-script entry points. After installation, both
|
|
30
|
+
are on `PATH` and can be invoked from any directory.
|
|
31
|
+
- `metis_test_runner.paths` — a single seam for every filesystem location MTR
|
|
32
|
+
reads or writes outside the package itself.
|
|
33
|
+
- `metis_test_runner.indexes` — ESO mirror URL constants.
|
|
34
|
+
- BSD-3-Clause `LICENSE` file.
|
|
35
|
+
- Trusted-publishing GitHub workflow (`.github/workflows/publish.yml`)
|
|
36
|
+
publishes to PyPI on `v*.*.*` tag pushes.
|
|
37
|
+
|
|
38
|
+
### Removed
|
|
39
|
+
- `launch.sh` — superseded by the `mtr` console script. Developers now use
|
|
40
|
+
`pip install -e .[dev]` to set up a working copy.
|
|
41
|
+
|
|
42
|
+
### Notes / known limitations
|
|
43
|
+
- The `--runner metapkg` mode of `mtr-cli` still shells out to `uv run --project
|
|
44
|
+
<metis-meta-package>`. This is the only remaining `uv` dependency and is
|
|
45
|
+
guarded with a clear error message if `uv` is not installed. It will be
|
|
46
|
+
removed once `metis-meta-package` is published to PyPI.
|
|
47
|
+
- `gui.REPO_ROOT` is kept as a back-compat alias pointing at `paths.data_dir()`
|
|
48
|
+
so existing test monkeypatches continue to work. To be removed in a future
|
|
49
|
+
release.
|
|
50
|
+
|
|
51
|
+
## 0.1.0
|
|
52
|
+
|
|
53
|
+
Internal pre-PyPI release. Distributed via `git clone` + `./launch.sh`.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025, AstarVienna
|
|
4
|
+
|
|
5
|
+
Redistribution and use in source and binary forms, with or without
|
|
6
|
+
modification, are permitted provided that the following conditions are met:
|
|
7
|
+
|
|
8
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
9
|
+
list of conditions and the following disclaimer.
|
|
10
|
+
|
|
11
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
12
|
+
this list of conditions and the following disclaimer in the documentation
|
|
13
|
+
and/or other materials provided with the distribution.
|
|
14
|
+
|
|
15
|
+
3. Neither the name of the copyright holder nor the names of its
|
|
16
|
+
contributors may be used to endorse or promote products derived from
|
|
17
|
+
this software without specific prior written permission.
|
|
18
|
+
|
|
19
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
20
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
21
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
22
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
23
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
24
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
25
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
26
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
27
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
28
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@@ -0,0 +1,379 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: metis-test-runner
|
|
3
|
+
Version: 0.2.0a0
|
|
4
|
+
Summary: GUI and CLI wrapper for end-to-end ESO METIS pipeline testing
|
|
5
|
+
Project-URL: Homepage, https://github.com/AstarVienna/MTR
|
|
6
|
+
Project-URL: Issues, https://github.com/AstarVienna/MTR/issues
|
|
7
|
+
Project-URL: Changelog, https://github.com/AstarVienna/MTR/blob/main/CHANGELOG.md
|
|
8
|
+
Author: AstarVienna
|
|
9
|
+
License: BSD-3-Clause
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: ELT,ESO,METIS,astronomy,pipeline,scopesim
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Environment :: X11 Applications :: Qt
|
|
14
|
+
Classifier: Intended Audience :: Science/Research
|
|
15
|
+
Classifier: License :: OSI Approved :: BSD License
|
|
16
|
+
Classifier: Operating System :: MacOS
|
|
17
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Astronomy
|
|
21
|
+
Requires-Python: <3.14,>=3.12
|
|
22
|
+
Requires-Dist: platformdirs>=4.0
|
|
23
|
+
Requires-Dist: pyqt6-qt6==6.6.0
|
|
24
|
+
Requires-Dist: pyqt6==6.6.0
|
|
25
|
+
Requires-Dist: python-dotenv>=1.0
|
|
26
|
+
Requires-Dist: pyyaml
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: build; extra == 'dev'
|
|
29
|
+
Requires-Dist: pytest; extra == 'dev'
|
|
30
|
+
Requires-Dist: twine; extra == 'dev'
|
|
31
|
+
Description-Content-Type: text/markdown
|
|
32
|
+
|
|
33
|
+
# METIS Test Runner
|
|
34
|
+
|
|
35
|
+
<p align="center">
|
|
36
|
+
<a href="https://github.com/eiseleb47/MTR/actions/workflows/unit_tests.yaml"><img src="https://img.shields.io/github/actions/workflow/status/eiseleb47/MTR/unit_tests.yaml?branch=main&label=unit%20tests&style=for-the-badge&labelColor=1e1e2e&color=a6e3a1&logo=github&logoColor=cdd6f4" alt="Unit Tests"></a>
|
|
37
|
+
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3-89b4fa?style=for-the-badge&labelColor=1e1e2e&logo=python&logoColor=cdd6f4" alt="Python 3"></a>
|
|
38
|
+
<a href="https://github.com/eiseleb47/MTR/commits/main"><img src="https://img.shields.io/github/last-commit/eiseleb47/MTR?style=for-the-badge&labelColor=1e1e2e&color=cba6f7&logo=git&logoColor=cdd6f4" alt="Last Commit"></a>
|
|
39
|
+
<a href="https://github.com/eiseleb47/MTR"><img src="https://img.shields.io/badge/platform-linux-fab387?style=for-the-badge&labelColor=1e1e2e&logo=linux&logoColor=cdd6f4" alt="Platform"></a>
|
|
40
|
+
</p>
|
|
41
|
+
|
|
42
|
+
A graphical front-end for end-to-end testing of the [METIS instrument pipeline](https://github.com/AstarVienna/METIS_Pipeline). It generates synthetic FITS observations via [ScopeSim](https://scopesim.readthedocs.io/) and then runs the matching [EDPS](https://www.eso.org/sci/software/edps/) reduction workflow — all from a single, self-contained GUI. A command-line interface (`mtr-cli`) is also shipped as a fallback for scripted or headless use.
|
|
43
|
+
|
|
44
|
+
## Install
|
|
45
|
+
|
|
46
|
+
MTR is distributed on PyPI. The recommended installer is **pipx**, which creates an isolated venv per application and never touches your system Python.
|
|
47
|
+
|
|
48
|
+
**Step 1 — install pipx** (one-time, skip if you already have it):
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
# Debian / Ubuntu
|
|
52
|
+
sudo apt install pipx && pipx ensurepath
|
|
53
|
+
|
|
54
|
+
# Fedora
|
|
55
|
+
sudo dnf install pipx && pipx ensurepath
|
|
56
|
+
|
|
57
|
+
# macOS (Homebrew)
|
|
58
|
+
brew install pipx && pipx ensurepath
|
|
59
|
+
|
|
60
|
+
# Any platform with Python ≥ 3.12 (fallback)
|
|
61
|
+
python -m pip install --user pipx && pipx ensurepath
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
**Step 2 — install MTR**:
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
pipx install metis-test-runner
|
|
68
|
+
mtr # launches the GUI from anywhere
|
|
69
|
+
mtr-cli examples/LMS_RAD_06.yaml # CLI equivalent
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
If you'd rather not install pipx, use Python's built-in venv module:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
python -m venv ~/.venvs/mtr
|
|
76
|
+
~/.venvs/mtr/bin/pip install metis-test-runner
|
|
77
|
+
~/.venvs/mtr/bin/mtr # or: source ~/.venvs/mtr/bin/activate && mtr
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
On first launch, open the **Install** tab and click *Install / Update* to fetch the METIS_Pipeline, METIS_Simulations, and ESO pipeline dependencies into `~/.local/share/metis-test-runner/` (override with `METIS_DATA_DIR=/path`). All pipeline dependencies install into the same isolated venv that MTR itself lives in — they never leak into your system Python.
|
|
81
|
+
|
|
82
|
+
## System Dependencies
|
|
83
|
+
|
|
84
|
+
The GUI requires a handful of system libraries for Qt6 / OpenGL rendering. Python, Git, and basic CLI tools (`curl`, `tar`, …) are assumed to already be present.
|
|
85
|
+
|
|
86
|
+
**Debian / Ubuntu**
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
sudo apt install \
|
|
90
|
+
libgl1 libegl1 libglib2.0-0 libxkbcommon0 \
|
|
91
|
+
libfontconfig1 libfreetype6 libdbus-1-3 \
|
|
92
|
+
libxkbcommon-x11-0 libxcb-cursor0 libxcb-keysyms1 libxcb-icccm4 libxcb-shape0
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
**Fedora**
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
sudo dnf install \
|
|
99
|
+
mesa-libGL mesa-libEGL glib2 libxkbcommon \
|
|
100
|
+
fontconfig freetype dbus-libs \
|
|
101
|
+
libxkbcommon-x11 xcb-util-cursor xcb-util-keysyms xcb-util-wm
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**macOS (Homebrew)**
|
|
105
|
+
|
|
106
|
+
On macOS, Qt uses the native Cocoa backend so X11/xcb packages are not needed:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
brew install freetype fontconfig glib
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
> **Tip:** if the GUI crashes at startup with *"could not find or load the Qt platform plugin"*, set `QT_DEBUG_PLUGINS=1` to get detailed diagnostics:
|
|
113
|
+
> ```bash
|
|
114
|
+
> QT_DEBUG_PLUGINS=1 mtr
|
|
115
|
+
> ```
|
|
116
|
+
> The output will list exactly which shared library failed to load.
|
|
117
|
+
|
|
118
|
+
## The GUI
|
|
119
|
+
|
|
120
|
+
The GUI is the recommended way to drive the test runner. It exposes every CLI flag through labelled controls, remembers your settings between sessions, and streams colour-coded live output from the pipeline.
|
|
121
|
+
|
|
122
|
+
Launch it with:
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
mtr
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
A **Light / Dark theme** button lives in the toolbar and toggles on the fly.
|
|
129
|
+
|
|
130
|
+
### Install tab
|
|
131
|
+
|
|
132
|
+
The Install tab performs the full pipeline bootstrap non-interactively. Use it if you do **not** already have the pipeline installed. Clicking **Install / Update** will:
|
|
133
|
+
|
|
134
|
+
1. Clone (or update, if already present) `METIS_Pipeline` and `METIS_Simulations` into the user data directory (`~/.local/share/metis-test-runner/` by default)
|
|
135
|
+
2. `pip install` all ESO pipeline Python dependencies — `pycpl`, `edps`, `pyesorex`, `adari_core`, `scopesim`, `scopesim_templates` — into the same isolated venv that hosts MTR (via `--extra-index-url` against the ESO mirrors). Out-of-band installs such as MetisWISE from the Archive tab are preserved.
|
|
136
|
+
3. Write `.env` into the user data directory (environment variables for PYTHONPATH, plugin directories, etc.)
|
|
137
|
+
4. Initialise and configure EDPS on port 4444
|
|
138
|
+
|
|
139
|
+
Re-running is safe — existing repositories are updated in place rather than re-cloned.
|
|
140
|
+
|
|
141
|
+
**Skip this tab** if you already have the pipeline installed via one of these paths — jump straight to the Run tab instead:
|
|
142
|
+
|
|
143
|
+
- **metis-meta-package** — choose runner `metapkg` and set *Meta-package dir* to your `metis-meta-package` folder
|
|
144
|
+
- **Bare-metal / ESO docs install** — choose runner `native`
|
|
145
|
+
- **Pipeline container** (Docker / Podman) — choose runner `docker` or `podman` and supply the container name
|
|
146
|
+
|
|
147
|
+
### Archive tab
|
|
148
|
+
|
|
149
|
+
The Archive tab connects to the remote METIS AIT archive via the
|
|
150
|
+
[MetisWISE](https://github.com/AstarVienna/MetisWISE) client. It has two pages:
|
|
151
|
+
|
|
152
|
+
1. **Install & Configure** — paste the OmegaCEN credentials from the
|
|
153
|
+
[METIS wiki](https://metis.strw.leidenuniv.nl/wiki/doku.php?id=ait:archive)
|
|
154
|
+
and click **Install MetisWISE** to pip-install the package into the same
|
|
155
|
+
isolated venv that hosts MTR. Then fill in the five `[global]` fields (`database_user`,
|
|
156
|
+
`database_password`, `project`, `database_tablespacename`, `database_name`)
|
|
157
|
+
and click **Save & Test Connection**. The values are written to
|
|
158
|
+
`~/.awe/Environment.cfg`; `data_server`, port and protocol are inherited
|
|
159
|
+
from the MetisWISE-packaged default (`metis-ds.hpc.rug.nl:8013`, https).
|
|
160
|
+
|
|
161
|
+
2. **Query & Download** — filter by raw classification tag or master
|
|
162
|
+
`PRO.CATG`, click **Search**, select files and download them to a local
|
|
163
|
+
directory.
|
|
164
|
+
|
|
165
|
+
To auto-pull missing master calibrations during a pipeline run, add
|
|
166
|
+
`--auto-fetch-calibrations` to the Run tab's options (or the CLI).
|
|
167
|
+
|
|
168
|
+
### Run tab
|
|
169
|
+
|
|
170
|
+
The Run tab wraps `mtr-cli` in a file-picker UI. All CLI options are exposed as form controls; runner-specific fields (container name, meta-package path) show and hide based on the selected runner.
|
|
171
|
+
|
|
172
|
+
Workflow:
|
|
173
|
+
|
|
174
|
+
1. **Add input files** via the file browser — YAML observation blocks (`*.yaml`, `*.yml`) and AIT-format CSV test sequences (`*.csv`) may be added freely and mixed in a single run. A live tally (e.g. `2 YAML · 1 CSV`) appears below the list.
|
|
175
|
+
2. **Tune options** — output directory, CPU cores, auto-calibration, runner mode, pipeline mode (simulate + run, simulate only, pipeline only), simulations directory, instrument packages directory
|
|
176
|
+
3. **Pick a workflow** (only for CSV-only runs that include the pipeline) — workflow auto-detection reads YAML content, so when the list contains only CSV files and you intend to run the pipeline, the *Workflow* dropdown appears and must be set; otherwise it stays hidden
|
|
177
|
+
4. **Click Run** — the Run button becomes Stop, and pipeline output streams into the log view with ANSI colouring stripped and stderr highlighted
|
|
178
|
+
5. **Inspect output** — the pane below the option form shows exactly where simulation frames and pipeline products will be written, updating live as you edit the output path
|
|
179
|
+
|
|
180
|
+
Settings are persisted via `QSettings` and restored on next launch, so you can re-run the last configuration with two clicks.
|
|
181
|
+
|
|
182
|
+
## Prerequisites (runner modes)
|
|
183
|
+
|
|
184
|
+
Regardless of whether you drive the runner from the GUI or the CLI, the underlying pipeline tools have to live *somewhere*. Three layouts are supported — pick the one that matches your install:
|
|
185
|
+
|
|
186
|
+
**Option A — consolidated install** (runner `metapkg`, default)
|
|
187
|
+
|
|
188
|
+
The Install tab takes care of this automatically. It pip-installs all pipeline
|
|
189
|
+
dependencies (`pycpl`, `edps`, `pyesorex`, `adari_core`, `scopesim`,
|
|
190
|
+
`scopesim_templates`) into the same isolated venv that hosts MTR, clones
|
|
191
|
+
`METIS_Pipeline` and `METIS_Simulations` into the user data directory
|
|
192
|
+
(`~/.local/share/metis-test-runner/` by default), and writes a `.env` file
|
|
193
|
+
there. The runner reads that `.env` automatically.
|
|
194
|
+
|
|
195
|
+
Use the GUI's *Meta-package dir* field (or `--meta-pkg` / `--simulations-dir`)
|
|
196
|
+
to point at an external installation if needed.
|
|
197
|
+
|
|
198
|
+
**Option B — Docker or Podman container** (runner `docker` / `podman`)
|
|
199
|
+
|
|
200
|
+
Build and start the pipeline container from [METIS_Pipeline/toolbox/](https://github.com/AstarVienna/METIS_Pipeline/tree/main/toolbox):
|
|
201
|
+
|
|
202
|
+
```bash
|
|
203
|
+
cd METIS_Pipeline/toolbox
|
|
204
|
+
docker build -t metispipeline .
|
|
205
|
+
docker run -d --name metis-pipeline --net=host \
|
|
206
|
+
--mount type=bind,source=/path/to/output,target=/output \
|
|
207
|
+
metispipeline
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Then in the GUI, set runner to `docker` (or `podman`) and enter the container name. The output directory must be bind-mounted into the container so EDPS can write products back to the host.
|
|
211
|
+
|
|
212
|
+
**Option C — bare-metal or inside a container** (runner `native`)
|
|
213
|
+
|
|
214
|
+
If the pipeline tools (`edps`, `python`, ScopeSim) are already on your PATH — either because you are running *inside* a container, or have installed everything directly — no extra setup is needed. Select runner `native`.
|
|
215
|
+
|
|
216
|
+
ScopeSim instrument packages (Armazones, ELT, METIS) will be downloaded into `./inst_pkgs/` in your current working directory on first use. Set the GUI's *Instrument packages* field (or `--inst-pkgs PATH`) to download or reuse packages from a fixed location instead.
|
|
217
|
+
|
|
218
|
+
> **Tip:** always launch the GUI (or invoke `mtr-cli`) from the same directory — otherwise ScopeSim will download a fresh copy of the instrument packages into every new directory, cluttering your filesystem.
|
|
219
|
+
|
|
220
|
+
## Input Formats
|
|
221
|
+
|
|
222
|
+
Two input formats are supported and may be mixed in a single run:
|
|
223
|
+
|
|
224
|
+
- **YAML observation blocks** (`*.yaml`, `*.yml`) — the primary, human-authored format. Workflow auto-detection works on YAML content.
|
|
225
|
+
- **AIT-format CSV test sequences** (`*.csv`) — the AIT performance-test sheet exported as CSV. Parsed by `metis_simulations.csvParser` at simulation time. Because MTR does **not** inspect CSV content itself, CSV-only runs that execute the pipeline require an explicit workflow choice (the GUI's *Workflow* dropdown, or `--workflow NAME` on the CLI).
|
|
226
|
+
|
|
227
|
+
See [`examples/small_test.csv`](examples/small_test.csv) for a minimal CSV; the rest of this section covers YAML.
|
|
228
|
+
|
|
229
|
+
### YAML Format
|
|
230
|
+
|
|
231
|
+
Each top-level key in the YAML is one *observation block*. The workflow (`lm_img`, `n_img`, `ifu`, `lm_lss`, `n_lss`, …) and the deepest pipeline target task are inferred automatically from the YAML content — primarily from `properties.tech`, falling back to `mode`.
|
|
232
|
+
|
|
233
|
+
Required fields per block:
|
|
234
|
+
|
|
235
|
+
```yaml
|
|
236
|
+
BLOCK_NAME:
|
|
237
|
+
do.catg: <EDPS classification tag> # e.g. DETLIN_IFU_RAW, IFU_SCI_RAW
|
|
238
|
+
mode: <scopesim mode> # e.g. wcu_lms, lms
|
|
239
|
+
source:
|
|
240
|
+
name: <scopesim source name> # e.g. empty_sky, star
|
|
241
|
+
kwargs: {}
|
|
242
|
+
properties:
|
|
243
|
+
dit: <float> # detector integration time (s)
|
|
244
|
+
ndit: <int> # number of integrations
|
|
245
|
+
catg: <CALIB|SCIENCE>
|
|
246
|
+
tech: <LMS|IMAGE,LM|LSS,LM|…>
|
|
247
|
+
type: <DETLIN|DARK|FLAT|…>
|
|
248
|
+
tplname: <ESO template name>
|
|
249
|
+
nObs: <int> # number of exposures to simulate
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
See `examples/LMS_RAD_06.yaml` for a complete IFU example covering the full calibration + science chain.
|
|
253
|
+
|
|
254
|
+
## Supported Workflows
|
|
255
|
+
|
|
256
|
+
| EDPS Workflow | `tech` values |
|
|
257
|
+
|---|---|
|
|
258
|
+
| `metis_lm_img_wkf` | `IMAGE,LM` |
|
|
259
|
+
| `metis_n_img_wkf` | `IMAGE,N` |
|
|
260
|
+
| `metis_ifu_wkf` | `LMS`, `IFU`, `RAVC,IFU` |
|
|
261
|
+
| `metis_lm_lss_wkf` | `LSS,LM` |
|
|
262
|
+
| `metis_n_lss_wkf` | `LSS,N` |
|
|
263
|
+
| `metis_lm_ravc_wkf` | `RAVC,LM` |
|
|
264
|
+
| `metis_lm_app_wkf` | `APP,LM` |
|
|
265
|
+
| `metis_pupil_imaging_wkf` | `PUP,LM`, `PUP,N` |
|
|
266
|
+
|
|
267
|
+
## Output Layout
|
|
268
|
+
|
|
269
|
+
Output is written under the chosen output directory (default: `./output/<timestamp>/`):
|
|
270
|
+
|
|
271
|
+
- `<output-dir>/sim/` — synthetic FITS frames from ScopeSim
|
|
272
|
+
- `<output-dir>/pipeline/` — reduced data products from EDPS
|
|
273
|
+
|
|
274
|
+
The GUI displays the resolved paths live under the *Output directory* field so you can see exactly where products will land before you hit Run.
|
|
275
|
+
|
|
276
|
+
## Command-Line Fallback
|
|
277
|
+
|
|
278
|
+
`mtr-cli` is the headless interface that the GUI drives under the hood. It is useful for scripting, CI jobs, and SSH sessions without a display. It accepts the same options as the GUI.
|
|
279
|
+
|
|
280
|
+
```bash
|
|
281
|
+
mtr-cli [OPTIONS] input1.yaml [input2.csv ...]
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
YAML and CSV inputs may be mixed in any combination.
|
|
285
|
+
|
|
286
|
+
### Options
|
|
287
|
+
|
|
288
|
+
| Flag | Default | Description |
|
|
289
|
+
|---|---|---|
|
|
290
|
+
| `-o / --output` | `./output/<timestamp>` | Root directory for all outputs (env: `METIS_OUTPUT_DIR`) |
|
|
291
|
+
| `--runner {metapkg,native,docker,podman}` | `metapkg` | Execution mode (see below; env: `METIS_RUNNER`) |
|
|
292
|
+
| `--container NAME` | — | Container name/ID for `docker` / `podman` runners (env: `METIS_CONTAINER`) |
|
|
293
|
+
| `--calib [N]` | `1` | Auto-generate N calibration frames (dark/flat) per unique config, inferred from input content. Pass `--calib 0` to disable. |
|
|
294
|
+
| `--workflow NAME` | auto-detect | Force EDPS workflow name (e.g. `metis.metis_lm_img_wkf`). Required when all inputs are `.csv` and the pipeline will run, because workflow auto-detection is YAML-only. Overrides auto-detection when YAML is also present. |
|
|
295
|
+
| `--cores N` | `4` | CPU cores used for parallel simulations |
|
|
296
|
+
| `--no-sim` | off | Skip simulation; run pipeline on existing FITS data (source defaults to `<output>/sim/` — override with `--pipeline-input`) |
|
|
297
|
+
| `--pipeline-input DIR` | `<output>/sim/` | Directory containing FITS files to feed the pipeline (only with `--no-sim`; env: `METIS_PIPELINE_INPUT`) |
|
|
298
|
+
| `--no-pipeline` | off | Run simulation only; skip EDPS pipeline |
|
|
299
|
+
| `--meta-pkg PATH` | `.` (repo root) | Path to the pipeline environment directory (`metapkg` runner only). Falls back to `./pipeline/` then `./metis-meta-package/`. (env: `METIS_META_PKG`) |
|
|
300
|
+
| `--simulations-dir PATH` | `./METIS_Simulations` (host) or `/home/metis/METIS_Simulations` (container) | Path to ScopeSim scripts (env: `METIS_SIMULATIONS_DIR`) |
|
|
301
|
+
| `--inst-pkgs PATH` | see below | Path to ScopeSim instrument packages (Armazones, ELT, METIS). Defaults to `./inst_pkgs` for `metapkg`/`native`, and container-resolved `./inst_pkgs` for `docker`/`podman` (env: `METIS_INST_PKGS`) |
|
|
302
|
+
| `--auto-fetch-calibrations` | off | Before running the pipeline, query the remote METIS archive (via MetisWISE) for any master calibrations the input set is missing and download them into the pipeline input directory. Requires MetisWISE to be installed and `~/.awe/Environment.cfg` to hold valid credentials — see the Archive tab. |
|
|
303
|
+
| `--prefer-masters` | off | Set EDPS `association_preference` to `master_per_quality_level` for this run, preferring master calibrations over reduced raw data. |
|
|
304
|
+
|
|
305
|
+
### Runner modes
|
|
306
|
+
|
|
307
|
+
| Mode | When to use |
|
|
308
|
+
|---|---|
|
|
309
|
+
| `metapkg` (default) | You used the GUI's Install tab (or an external `metis-meta-package` install). Pipeline tools are pip-installed alongside MTR in the same isolated venv. For external `metis-meta-package` checkouts, `uv` must be on PATH (`pipx install uv`). |
|
|
310
|
+
| `native` | Tools (`edps`, `python`, ScopeSim) are installed directly on PATH — e.g. you are running **inside** a Docker/Podman container, or have a bare-metal install. |
|
|
311
|
+
| `docker` / `podman` | Tools live inside a container and you are running the script **outside** it. The runner wraps every command with `docker exec` / `podman exec`. |
|
|
312
|
+
|
|
313
|
+
> **Note for `docker` / `podman` runners:** the output directory (`-o`) must be bind-mounted into the container so EDPS can write pipeline products to it. The `--simulations-dir` flag should point to the path of `METIS_Simulations/Simulations` *inside* the container (default: `/home/metis/METIS_Simulations`).
|
|
314
|
+
|
|
315
|
+
### Examples
|
|
316
|
+
|
|
317
|
+
```bash
|
|
318
|
+
# Full run with metis-meta-package (default)
|
|
319
|
+
mtr-cli examples/LMS_RAD_06.yaml
|
|
320
|
+
|
|
321
|
+
# Inside a container or bare-metal install (tools on PATH)
|
|
322
|
+
mtr-cli --runner native examples/LMS_RAD_06.yaml
|
|
323
|
+
|
|
324
|
+
# Exec into a running Docker container from the host
|
|
325
|
+
mtr-cli --runner docker --container metis-pipeline examples/LMS_RAD_06.yaml
|
|
326
|
+
|
|
327
|
+
# Exec into a running Podman container; set runner via env var
|
|
328
|
+
METIS_RUNNER=podman METIS_CONTAINER=metis-pipeline mtr-cli examples/LMS_RAD_06.yaml
|
|
329
|
+
|
|
330
|
+
# Multiple YAML files, custom output dir, with auto-calibration frames
|
|
331
|
+
mtr-cli -o /tmp/myrun --calib obs1.yaml obs2.yaml
|
|
332
|
+
|
|
333
|
+
# Crank up parallelism for big simulation batches
|
|
334
|
+
mtr-cli --cores 12 examples/LMS_RAD_06.yaml
|
|
335
|
+
|
|
336
|
+
# Only simulate, inspect the FITS files manually
|
|
337
|
+
mtr-cli --no-pipeline examples/LMS_RAD_06.yaml
|
|
338
|
+
|
|
339
|
+
# Only run the pipeline on previously simulated data
|
|
340
|
+
mtr-cli --no-sim -o /tmp/myrun examples/LMS_RAD_06.yaml
|
|
341
|
+
|
|
342
|
+
# Pipeline-only with FITS files from a custom location
|
|
343
|
+
mtr-cli --no-sim --pipeline-input /data/sim_fits -o /tmp/myrun
|
|
344
|
+
|
|
345
|
+
# CSV-only input, simulate only (workflow auto-detection not needed)
|
|
346
|
+
mtr-cli --no-pipeline examples/small_test.csv
|
|
347
|
+
|
|
348
|
+
# CSV-only input, full simulate + pipeline run (workflow must be explicit)
|
|
349
|
+
mtr-cli --workflow metis.metis_lm_img_wkf examples/small_test.csv
|
|
350
|
+
|
|
351
|
+
# Mixed YAML + CSV in a single run
|
|
352
|
+
mtr-cli examples/small_test.yaml examples/small_test.csv
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
## Repository Layout
|
|
356
|
+
|
|
357
|
+
```
|
|
358
|
+
MTR/
|
|
359
|
+
├── src/metis_test_runner/
|
|
360
|
+
│ ├── gui.py # Graphical front-end (PyQt6) — primary entry point
|
|
361
|
+
│ ├── run_metis.py # Headless CLI (used directly or wrapped by the GUI)
|
|
362
|
+
│ ├── archive.py # MetisWISE archive integration
|
|
363
|
+
│ ├── paths.py # User data directory resolution (platformdirs)
|
|
364
|
+
│ ├── indexes.py # ESO mirror pip-index URL constants
|
|
365
|
+
│ └── examples/ # Bundled YAML / CSV example inputs
|
|
366
|
+
├── container/
|
|
367
|
+
│ ├── Dockerfile # Ubuntu 24.04 GUI container (Qt6 / Wayland)
|
|
368
|
+
│ └── compose.yml # Podman / Docker Compose for the GUI service
|
|
369
|
+
├── tests/ # Unit tests (pytest)
|
|
370
|
+
└── pyproject.toml # Project metadata (hatchling build backend)
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
## Related Repositories
|
|
374
|
+
|
|
375
|
+
This runner is designed to work alongside the following repos, which are installed via the GUI's Install tab:
|
|
376
|
+
|
|
377
|
+
- **[METIS_Pipeline](https://github.com/AstarVienna/METIS_Pipeline)** — the core Python/C pipeline, EDPS workflows, and PyEsoRex recipes
|
|
378
|
+
- **[METIS_Simulations](https://github.com/AstarVienna/METIS_Simulations)** — ScopeSim scripts that generate synthetic FITS observations for each observing mode
|
|
379
|
+
- **[metis-meta-package](https://github.com/eiseleb47/metis-meta-package)** — legacy standalone meta-installer (still supported as a fallback; the Install tab now installs pipeline dependencies directly into the main virtual environment)
|