mavdac 0.1.1__tar.gz

mavdac-0.1.1/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.3
+Name: mavdac
+Version: 0.1.1
+Classifier: Programming Language :: Rust
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Summary: MAVIS Differential Astrometric Calibrator
+License: MIT
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Source Code, https://github.com/jcranney/mavdac
+
+# mavdac
+MAVIS Differential Astrometric Calibration. This image processing pipeline is designed to ingest images acquired during astrometric calibration of (e.g.) [MAVIS](https://mavis-ao.org), and output the estimated distortion field derived from these images. The bulk of the pipeline is written in Rust, but the most useful functions and the CLI tool are wrapped in Python.
+
+# Install
+Eventually this will be pip installable, but for now, the easiest way to get going
+is through `maturin` (which is `pip install`able). Clone this repo then build it:
+```bash
+git clone https://github.com/jcranney/mavdac
+cd mavdac
+maturin develop --release
+```
+If successful, you should now be able to run `mavdac` as per [Usage](#usage)
+
+# Quickstart
+The simplest usage of this software is by calling
+```bash
+mavdac "./some/pattern*.fits"
+```
+which tries to parse all images on disk matching the glob pattern argument, and prints the distortion basis-function coefficients to stdout. These are not very useful on their own, so a user can also provide a set of coordinates to evaluate the distortion functions:
+```bash
+mavdac "./some/pattern*.fits" ./coordinates.txt
+```
+In this case, `mavdac` prints the sampled distortion function values to stdout, which the user can (e.g.) cat to some output file:
+```bash
+mavdac "./some/pattern*.fits" ./coordinates.txt > output.txt
+```
+
+# Usage
+
+There are a few ways to use `mavdac`, e.g.,:
+ - Through the Python CLI,
+ - As a Python module (part of another python-based application),
+ - As a Rust crate (part of another rust-based application),
+
+Here, I assume you are a standard user, hoping to perform astrometric calibration of an optical system using the MAVIS Differential Astrometric Calibration technique. In that case, the workflow is as follows:
+
+ 1) **Collect images** of calibration source into `./some/dir/img_0*.fits`,
+ 2) **Define coordinates** in pixel space to evaluate the measured distortions, e.g., `./some/dir/coords`:
+ ```
+ 0,0,
+ 0,8,
+ 0,16,
+ 0,32,
+ ...
+ 4096,4088,
+ 4096,4096,
+ ```
+ 3) **Write a `grid.yaml` file**, specifying the geometry of the pinhole mask/calibration source.
+ 4) **Run `mavdac`** pipeline, specifying image filename pattern and coordinates filename path, piping `stdout` to some `./results` file.
+
+Below we provide the requirements for each of the above tasks.
+### 1) Collecting images
+ - Each image must be saved in FITS format, as any of: `{u8, f32, f64, i32, u32}` type. Note that internally, all images are converted to `f64` upon loading.
+ - Each file must contain only a single 2D array in the primary HDU, and must contain the FITS header keywords:
+   - `XSHIFT`: shift in pixels along x-axis of the calibration mask for that exposure (from some nominal "home" position),
+   - `YSHIFT`: shift in pixels along y-axis of the calibration mask for that exposure (from some nominal "home" position),
+### 2) Defining coordinates
+ - The internal mavdac pipeline will fit a set of distortion basis functions (e.g., 2D bivariate polynomials) to the measured distortions. If you run the CLI without a second argument, e.g.:
+ ```bash
+mavdac "./some/dir/imgs_*.fits"
+ ```
+ then the software will print some status updates to stderr, and eventually the coefficients of the fitted basis functions to stdout. This is mostly unusable, unless you write your own software to parse these coefficients. The intended usage is to provide a set of coordinates in pixel space, in which case you will get the distortions at those coordinates printed to stdout. E.g., with a coordinates file, `./some/coords`:
+ ```
+0,0,
+0,2000,
+2000,0,
+2000,2000,
+ ```
+ you can run:
+ ```bash
+ mavdac "./some/dir/imgs_*.fits" ./some/coords
+ ```
+ which will output:
+ ```
+0,0,0.0966850540113886,0.3300609989212976,
+0,2000,0.07252484678180265,0.007773570343281411,
+2000,0,0.004342546567941594,0.26372826456406706,
+2000,2000,-0.0018909241959087534,-0.0036487693998900166,
+ ```
+ to stdout. The expected usage is something like:
+ ```bash
+mavdac "./imgs_*.fits" ./coords > ./out
+ ```
+ which cats only the stdout to a file, `./out`. I've spent a bit of time thinking about the simplest user interface with this software, and that is the best I can come up with, but if you have a better idea (perhaps inspired by experience using similar pipelines), please contact me or raise an issue in this repo.
+
+### 3) Writing a `grid.yaml` file
+ The coordinates of the pinhole grid are specified by a `grid.yaml` file, which currently only supports the "Hex" geometry. Here is an example:
+  ```yaml
+ !Hex
+ pitch: 133.33333333333334
+ rotation: 0.0
+ offset:
+   x: 0.0
+   y: 0.0
+  ```
+  All of these parameters are required, they are described below:
+   - `pitch`: distance between adjacent pinholes (in pixel units),
+   - `rotation`: orientation of hexagonal pinhole geometry (in radians),
+   - `offset`: shift of overall grid in pixel units, in `x` and `y` dimensions. Note that for `x=0` and `y=0`, there is a point in the grid at the exact middle of the image.
+
+### 4) Running `mavdac`
+Finally, once you have:
+ - the images in `./some/dir`,
+ - a `grid.yaml` file,
+ - and optionally, a `./coords` text file,
+ 
+ you can run mavdac from the command line by:
+```bash
+mavdac "./some/dir/*.fits" ./coords > ./out
+```
+For a reminder, you can check `mavdac --help`:
+mavis_acm/lab/inputs 
+```
+$ python -m mavdac --help
+usage: process mavis differential astrometric calibrations [-h] pattern [coordinates]
+
+positional arguments:
+  pattern      glob pattern for matching fits files to use for calibration
+  coordinates  file containing coordinates to map through distortions
+
+options:
+  -h, --help   show this help message and exit
+```
+
+
+# Validating mavdac
+All versions of mavdac < 1.0.0 are subject to breaking changes between any two releases. The tests and validations are being developed side-by-side with the software, so will become populated as the pipeline matures. It is envisioned that there will be 3 levels of validation for the software that should be re-satisfied for any release:
+ - unit tests,
+ - system tests, and
+ - hardware tests.
+
+## Unit tests
+Unit tests for the rust package will be found within each rust module, e.g., within `./rust/src/lib.rs`, under `mod tests {...}`.
+
+## System tests
+System tests for the rust + python package (e.g., the CLI), can be found in `./src/tests`. These will include tests that generate images with known distortions that are piped through mavdac, and the measured distortion field can be checked for quality.
+
+## Hardware tests
+The final validation of this software package is through it's use on data acquired in a real optical system with introduced distortions. To demonstrate the challenges associated with this kind of validation - try to construct an experiment which can verify that the software works as intended. The following are the most direct:
+- **measurement of known distortion**: By injecting a known distortion source into the optical path of a simple imaging system, one can estimate the level of distortion introduced by this optic, and compare it to the truth. The challenge here is finding some optical component that introduces a known distortion with some high level of absolute precision.
+- **measurement of known calibration source**: By using a highly calibrated pinhole mask at the entrance of the optical system, one can perform a classical distortion calibration, measuring the observed spots and taking their offset as the "true distortion", then compare this with the recovered distortion using the differential calibration method. The challenge here is finding a calibration source which is accurate and stable down to 10s of nanometres. It is worth noting that the calibration object does not need to be the full pinhole grid, but could be a smaller set of sources with precise relative positioning that are moved around the field through different exposures.
+- The third and most flexible method we considered allows the use of an **unknown distortion** with an **unknown calibration source** - that is, a calibration source with some unknown pinhole position errors, then to characterise the distortions with this mask. This will allow us to estimate the true positions of the pinholes (assuming that the mavdac pipeline works well). Then, we can insert the pinhole mask in a different orientation (e.g., rotated by 180 degrees), and measure the observed pinhole positions. These new pinhole positions should be the original estimated pinhole positions with distortion removed, rotated by 180 degress, plus the recovered distortion evaluated at the new pinhole positions. This should first be done without a strong distortion in the system, and then repeated with stronger distortions to assess the practical limitations of this method.

mavdac-0.1.1/README.md

@@ -0,0 +1,142 @@
+# mavdac
+MAVIS Differential Astrometric Calibration. This image processing pipeline is designed to ingest images acquired during astrometric calibration of (e.g.) [MAVIS](https://mavis-ao.org), and output the estimated distortion field derived from these images. The bulk of the pipeline is written in Rust, but the most useful functions and the CLI tool are wrapped in Python.
+
+# Install
+Eventually this will be pip installable, but for now, the easiest way to get going
+is through `maturin` (which is `pip install`able). Clone this repo then build it:
+```bash
+git clone https://github.com/jcranney/mavdac
+cd mavdac
+maturin develop --release
+```
+If successful, you should now be able to run `mavdac` as per [Usage](#usage)
+
+# Quickstart
+The simplest usage of this software is by calling
+```bash
+mavdac "./some/pattern*.fits"
+```
+which tries to parse all images on disk matching the glob pattern argument, and prints the distortion basis-function coefficients to stdout. These are not very useful on their own, so a user can also provide a set of coordinates to evaluate the distortion functions:
+```bash
+mavdac "./some/pattern*.fits" ./coordinates.txt
+```
+In this case, `mavdac` prints the sampled distortion function values to stdout, which the user can (e.g.) cat to some output file:
+```bash
+mavdac "./some/pattern*.fits" ./coordinates.txt > output.txt
+```
+
+# Usage
+
+There are a few ways to use `mavdac`, e.g.,:
+ - Through the Python CLI,
+ - As a Python module (part of another python-based application),
+ - As a Rust crate (part of another rust-based application),
+
+Here, I assume you are a standard user, hoping to perform astrometric calibration of an optical system using the MAVIS Differential Astrometric Calibration technique. In that case, the workflow is as follows:
+
+ 1) **Collect images** of calibration source into `./some/dir/img_0*.fits`,
+ 2) **Define coordinates** in pixel space to evaluate the measured distortions, e.g., `./some/dir/coords`:
+ ```
+ 0,0,
+ 0,8,
+ 0,16,
+ 0,32,
+ ...
+ 4096,4088,
+ 4096,4096,
+ ```
+ 3) **Write a `grid.yaml` file**, specifying the geometry of the pinhole mask/calibration source.
+ 4) **Run `mavdac`** pipeline, specifying image filename pattern and coordinates filename path, piping `stdout` to some `./results` file.
+
+Below we provide the requirements for each of the above tasks.
+### 1) Collecting images
+ - Each image must be saved in FITS format, as any of: `{u8, f32, f64, i32, u32}` type. Note that internally, all images are converted to `f64` upon loading.
+ - Each file must contain only a single 2D array in the primary HDU, and must contain the FITS header keywords:
+   - `XSHIFT`: shift in pixels along x-axis of the calibration mask for that exposure (from some nominal "home" position),
+   - `YSHIFT`: shift in pixels along y-axis of the calibration mask for that exposure (from some nominal "home" position),
+### 2) Defining coordinates
+ - The internal mavdac pipeline will fit a set of distortion basis functions (e.g., 2D bivariate polynomials) to the measured distortions. If you run the CLI without a second argument, e.g.:
+ ```bash
+mavdac "./some/dir/imgs_*.fits"
+ ```
+ then the software will print some status updates to stderr, and eventually the coefficients of the fitted basis functions to stdout. This is mostly unusable, unless you write your own software to parse these coefficients. The intended usage is to provide a set of coordinates in pixel space, in which case you will get the distortions at those coordinates printed to stdout. E.g., with a coordinates file, `./some/coords`:
+ ```
+0,0,
+0,2000,
+2000,0,
+2000,2000,
+ ```
+ you can run:
+ ```bash
+ mavdac "./some/dir/imgs_*.fits" ./some/coords
+ ```
+ which will output:
+ ```
+0,0,0.0966850540113886,0.3300609989212976,
+0,2000,0.07252484678180265,0.007773570343281411,
+2000,0,0.004342546567941594,0.26372826456406706,
+2000,2000,-0.0018909241959087534,-0.0036487693998900166,
+ ```
+ to stdout. The expected usage is something like:
+ ```bash
+mavdac "./imgs_*.fits" ./coords > ./out
+ ```
+ which cats only the stdout to a file, `./out`. I've spent a bit of time thinking about the simplest user interface with this software, and that is the best I can come up with, but if you have a better idea (perhaps inspired by experience using similar pipelines), please contact me or raise an issue in this repo.
+
+### 3) Writing a `grid.yaml` file
+ The coordinates of the pinhole grid are specified by a `grid.yaml` file, which currently only supports the "Hex" geometry. Here is an example:
+  ```yaml
+ !Hex
+ pitch: 133.33333333333334
+ rotation: 0.0
+ offset:
+   x: 0.0
+   y: 0.0
+  ```
+  All of these parameters are required, they are described below:
+   - `pitch`: distance between adjacent pinholes (in pixel units),
+   - `rotation`: orientation of hexagonal pinhole geometry (in radians),
+   - `offset`: shift of overall grid in pixel units, in `x` and `y` dimensions. Note that for `x=0` and `y=0`, there is a point in the grid at the exact middle of the image.
+
+### 4) Running `mavdac`
+Finally, once you have:
+ - the images in `./some/dir`,
+ - a `grid.yaml` file,
+ - and optionally, a `./coords` text file,
+ 
+ you can run mavdac from the command line by:
+```bash
+mavdac "./some/dir/*.fits" ./coords > ./out
+```
+For a reminder, you can check `mavdac --help`:
+mavis_acm/lab/inputs 
+```
+$ python -m mavdac --help
+usage: process mavis differential astrometric calibrations [-h] pattern [coordinates]
+
+positional arguments:
+  pattern      glob pattern for matching fits files to use for calibration
+  coordinates  file containing coordinates to map through distortions
+
+options:
+  -h, --help   show this help message and exit
+```
+
+
+# Validating mavdac
+All versions of mavdac < 1.0.0 are subject to breaking changes between any two releases. The tests and validations are being developed side-by-side with the software, so will become populated as the pipeline matures. It is envisioned that there will be 3 levels of validation for the software that should be re-satisfied for any release:
+ - unit tests,
+ - system tests, and
+ - hardware tests.
+
+## Unit tests
+Unit tests for the rust package will be found within each rust module, e.g., within `./rust/src/lib.rs`, under `mod tests {...}`.
+
+## System tests
+System tests for the rust + python package (e.g., the CLI), can be found in `./src/tests`. These will include tests that generate images with known distortions that are piped through mavdac, and the measured distortion field can be checked for quality.
+
+## Hardware tests
+The final validation of this software package is through it's use on data acquired in a real optical system with introduced distortions. To demonstrate the challenges associated with this kind of validation - try to construct an experiment which can verify that the software works as intended. The following are the most direct:
+- **measurement of known distortion**: By injecting a known distortion source into the optical path of a simple imaging system, one can estimate the level of distortion introduced by this optic, and compare it to the truth. The challenge here is finding some optical component that introduces a known distortion with some high level of absolute precision.
+- **measurement of known calibration source**: By using a highly calibrated pinhole mask at the entrance of the optical system, one can perform a classical distortion calibration, measuring the observed spots and taking their offset as the "true distortion", then compare this with the recovered distortion using the differential calibration method. The challenge here is finding a calibration source which is accurate and stable down to 10s of nanometres. It is worth noting that the calibration object does not need to be the full pinhole grid, but could be a smaller set of sources with precise relative positioning that are moved around the field through different exposures.
+- The third and most flexible method we considered allows the use of an **unknown distortion** with an **unknown calibration source** - that is, a calibration source with some unknown pinhole position errors, then to characterise the distortions with this mask. This will allow us to estimate the true positions of the pinholes (assuming that the mavdac pipeline works well). Then, we can insert the pinhole mask in a different orientation (e.g., rotated by 180 degrees), and measure the observed pinhole positions. These new pinhole positions should be the original estimated pinhole positions with distortion removed, rotated by 180 degress, plus the recovered distortion evaluated at the new pinhole positions. This should first be done without a strong distortion in the system, and then repeated with stronger distortions to assess the practical limitations of this method.

mavdac-0.1.1/pyproject.toml

@@ -0,0 +1,18 @@
+[build-system]
+requires = ["maturin>=1.7,<2.0"]
+build-backend = "maturin"
+
+[project]
+name = "mavdac"
+requires-python = ">=3.8"
+classifiers = [
+    "Programming Language :: Rust",
+    "Programming Language :: Python :: Implementation :: CPython",
+    "Programming Language :: Python :: Implementation :: PyPy",
+]
+dynamic = ["version"]
+[tool.maturin]
+features = ["pyo3/extension-module"]
+
+[project.scripts]
+mavdac = "mavdac.__main__:__name__"