dkist-processing-cryonirsp 0.0.55__tar.gz → 0.0.56__tar.gz

{dkist_processing_cryonirsp-0.0.55 → dkist_processing_cryonirsp-0.0.56}/CHANGELOG.rst

@@ -1,3 +1,12 @@
+v0.0.56 (2024-04-26)
+====================
+
+Documentation
+-------------
+
+- Update online documentation for Cryo-NIRSP bad pixel correction, beam angle calculation, and beam boundary calculation. (`#117 <https://bitbucket.org/dkistdc/dkist-processing-cryonirsp/pull-requests/117>`__)
+
+
 v0.0.55 (2024-04-12)
 ====================
 

{dkist_processing_cryonirsp-0.0.55 → dkist_processing_cryonirsp-0.0.56}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dkist_processing_cryonirsp
-Version: 0.0.55
+Version: 0.0.56
 Summary: Science processing code for the Cryo-NIRSP instrument on DKIST
 Home-page: https://bitbucket.org/dkistdc/dkist_processing_cryonirsp/src/main/
 Author: NSO / AURA

{dkist_processing_cryonirsp-0.0.55 → dkist_processing_cryonirsp-0.0.56}/dkist_processing_cryonirsp/tasks/linearity_correction.py

@@ -376,7 +376,7 @@ class LinearityCorrection(CryonirspTaskBase):
                 c. Load the frame slices into the chunk chunk_stack
                 d. Linearize the chunk chunk_stack
                 e. Store the linearized chunk in the proper location in the final linearized array
-            4. Return the linearized ramp, reshaping it to the original frame shape
+            3. Return the linearized ramp, reshaping it to the original frame shape
         """
         thresholds_flattened = thresholds.flatten()
         frame_shape = ramp_set.frame_shape

{dkist_processing_cryonirsp-0.0.55 → dkist_processing_cryonirsp-0.0.56}/dkist_processing_cryonirsp.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dkist-processing-cryonirsp
-Version: 0.0.55
+Version: 0.0.56
 Summary: Science processing code for the Cryo-NIRSP instrument on DKIST
 Home-page: https://bitbucket.org/dkistdc/dkist_processing_cryonirsp/src/main/
 Author: NSO / AURA

{dkist_processing_cryonirsp-0.0.55 → dkist_processing_cryonirsp-0.0.56}/dkist_processing_cryonirsp.egg-info/SOURCES.txt

@@ -102,13 +102,19 @@ dkist_processing_cryonirsp/workflows/ci_l0_processing.py
 dkist_processing_cryonirsp/workflows/sp_l0_processing.py
 dkist_processing_cryonirsp/workflows/trial_workflows.py
 docs/Makefile
+docs/bad_pixel_calibration.rst
+docs/beam_angle_calculation.rst
+docs/beam_boundary_computation.rst
 docs/changelog.rst
+docs/ci_science_calibration.rst
 docs/conf.py
 docs/index.rst
 docs/l0_to_l1_cryonirsp_ci.rst
 docs/l0_to_l1_cryonirsp_sp.rst
+docs/linearization.rst
 docs/make.bat
 docs/requirements.txt
 docs/requirements_table.rst
 docs/scientific_changelog.rst
+docs/sp_science_calibration.rst
 licenses/LICENSE.rst

dkist_processing_cryonirsp-0.0.56/docs/bad_pixel_calibration.rst

@@ -0,0 +1,47 @@
+Bad Pixel Calibration
+============================
+
+Introduction
+------------
+
+Both of the Cryo-NIRSP instrument cameras are known to have significant numbers of bad pixels.
+The effects of these are made worse by the linearization algorithm, which can force some pixels
+to be exactly zero. The algorithm described below is used to identify bad pixels and create a map
+of their locations. The map is an integer array of the same size as the camera output array, with
+zeros for pixels that are good and ones for the pixels that are bad.
+
+The Bad Pixel Map Algorithm
+------------------------------
+
+Compute Average Solar Gain Image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+First we will compute an average solar gain image. The solar gain is used because it has high flux
+and the beam illumination pattern is the same as during normal observing.
+
+Smooth the Average Gain Image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We can remove the effects of any bad pixels by smoothing the average gain image using a median filter method,
+filtering only along the spatial axis of the slit for SP data so as to not broaden the spectral lines, and
+filtering along both the spatial and spectral axes for CI data. After the gain image has been smoothed,
+the bad pixel areas should no longer be visible.
+
+Threshold the Difference Image to Find Bad Pixels
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Next, we subtract the smoothed image from the average gain image. Doing this allows us to identify all
+pixels in the difference image whose absolute value is larger than a set threshold value times the standard
+deviation of the average image (i.e., a difference larger than N standard deviations of the original image is
+considered a bad pixel). The threshold value is a pipeline settable parameter and was derived
+empirically. With a too-low threshold, we start to pick up the solar spectrum in the bad pixel image. With a
+too-large threshold, we start to miss bad pixels. Moreover, we currently use bad pixel corrections only for the
+gain images and not for the observe images, so any potential impacts are limited. This may change in the
+future.
+
+Apply Bad Pixel Map
+^^^^^^^^^^^^^^^^^^^
+
+This bad pixel map is then applied to the lamp gain and solar gain by replacing the identified
+bad pixels with the median value of the input image over a specified region. This prevents the
+gain image from causing “hot” or “cold” pixels in the resulting gain corrected input images.

dkist_processing_cryonirsp-0.0.56/docs/beam_angle_calculation.rst

@@ -0,0 +1,53 @@
+Beam Angle Computation
+============================
+
+Introduction
+------------
+
+As part of the geometric calibration, the angular rotation of the slit relative to the image axes
+must be computed and then corrected. There is no fiducial mark or hairline present in the Cryo-NIRSP
+images, so some other method must be used. The angular rotation is derived from the lamp gain images using
+the algorithm described below. It is described for only a single beam, but is equally applicable to use for both
+beams.
+
+The Beam Angle Algorithm
+------------------------------
+
+Compute Average Lamp Gain Image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We start by using the normalized lamp gain image that is computed as part of the lamp gain calibration.
+The image has been corrected for bad pixels (hot or cold) using the bad pixel map derived from average
+solar gain images (See :doc:`bad pixel calibration <bad_pixel_calibration>`).
+
+Compute the Normalized Spatial Gradient Image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The normalized lamp gain image contains some some faint horizontal (i.e. along the spectral axis) lines. These are the result of slight
+imperfections in the slit and in the grating optics, and they can be used to derive the rotation of the slit axis
+relative to the image axes. By computing a normalized spatial gradient along the spatial axis (along the slit),
+we can enhance the faint horizontal lines to make them more pronounced. The gradient is computed by shifting the
+image along the spatial axis in both the + and - directions, computing the difference of the two images, and
+normalizing by the sum of the two at each pixel position. The resulting gradient image contains horizontal
+lines that are more emphasized.
+
+Compute the Angular Rotation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Next we identify two long, narrow regions, or strips along the spatial axis. The length of each of the strips
+is one half of the slit length in pixels and they are centered about the mid-point of the length of the slit
+as seen on the sensor.
+
+We then compute the median value along the spectral axis (or rows). This condenses each strip into a 1D signal
+that represents the static fluctuations in the image along the spatial axis. The next step is to compute the
+cross correlation of the right signal relative to the left signal (the reference). The result is the measured
+shift of the right signal relative to the left. Using the measured shift between the signals and the known
+separation along the spectral axis between the midpoints of the strips, we can then compute the angular
+rotation using a simple arc-tangent.
+
+Correcting the Rotational Offset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Finally, we can remove this rotational offset. The slit axis is now aligned with the spatial axis of the image.
+The angular rotation measured here using the lamp gain image is then used to correct all of the
+observe images as part of the geometric corrections.

dkist_processing_cryonirsp-0.0.56/docs/beam_boundary_computation.rst

@@ -0,0 +1,88 @@
+Beam Boundary Computation
+============================
+
+Introduction
+------------
+
+Raw Cryo-NIRSP frames are not fully illuminated; in some areas, there is no illumination at all.
+These areas should not be used for science processing and
+must be accounted for. Moreover, the illuminated region(s) will change slightly depending
+on the optical alignment of the instrument. Hence, we cannot use a particular set
+of beam boundary constants from which to extract the illuminated regions. Instead we need to
+compute these regions as part of the calibration process.
+
+The Beam Boundary Algorithm
+------------------------------
+
+The basic idea of the algorithm is to use an average solar gain image and apply a segmentation
+algorithm to it that separates the image into illuminated pixels and non-illuminated pixels.
+(Note: We use solar gain images because they have larger flux than the lamp gain images
+and the lamp gain images do not have the same illumination pattern as the solar
+gain images. Therefore, in order to make sure the beam boundaries match the on-sky data,
+and are as correct as they can be, we must use solar gain images. We only use a single frame because the
+illuminated portion of the CCD is always constant.)
+The segmentation must be robust enough to have all pixels within a well defined outer boundary
+considered to be illuminated. This is sufficient to extract an illuminated region from a single
+beam for the Context Imager (CI). For the Spectropolarimeter (SP), however, we need to identify
+the separate beam regions and perform an initial alignment prior to computing the final individual
+beam boundaries.
+
+Compute the Average Solar Gain Image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We start by computing an average solar gain image. Because we are interested only in identifying
+the regions of the sensor that are illuminated, we do not need to perform any dark correction.
+Simply averaging the input solar gain images is sufficient.
+
+
+Correct Bad Pixels Using the Median Filter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The effects of the bad pixels must be corrected, otherwise they might be mistaken for non-illuminated
+regions. We correct these bad pixels using the :doc:`bad pixel calibration algorithm <bad_pixel_calibration>`
+and by applying a median filter to only those pixels that are flagged as bad. We use the masked array feature
+of numpy to achieve this filtering. For the SP, the correction algorithm is applied only in the spatial
+direction along the slit. For the CI, it is applied along both axes. For this application, the
+difference does not matter, as all we are doing is identifying illuminated pixels. However, later on when
+the solar gain array is computed, the correction must not broaden the spectral lines and so the restriction
+is important.
+
+
+Segment the Image Into Illuminated and Non-Illuminated Regions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Next we smooth the entire array so the image segmentation algorithm does not recognize the absorption
+lines as part of the non-illuminated region. The smoothed solar gain array is then used as the input to an
+image segmentation algorithm. The `Scikit Image threshold minimum <https://scikit-image.org/docs/stable/api/skimage.filters.html#skimage.filters.threshold_minimum>`_
+method is used to find the threshold between light and dark and this threshold is then
+used to generate a boolean map describing which pixels are illuminated (True) and which pixels are non-illuminated
+(False). However, this map cannot be used to easily extract illuminated regions because it is not guaranteed to be
+contiguous. Ideally, we would like to be able to identify a slice that can be used to extract the illuminated
+region as a rectangular array.
+
+
+Split Into Two Separate beam Images
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Now that we know the illuminated boundaries, we can identify and extract the largest inscribed rectangular
+region within the illuminated map. This is done using the `largestinteriorrectangle package <https://pypi.org/project/largestinteriorrectangle/>`_.
+We split the image into two beam images, avoiding a 10% spectral region
+surrounding the beam boundary in the middle of the image.
+
+Compute Relative Horizontal Shift of Right Beam to Left Beam
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There is a relative shift between these two images, so we need to determine that shift and compute the
+boundaries of each so that they overlap properly to within a single horizontal pixel. Note that the rotational
+differences and any additional shifts required to align the images are computed later as part of the geometric
+calibration. The horizontal shift of the right beam relative to the left is computed using the
+`Scikit Image phase cross correlation <https://scikit-image.org/docs/stable/api/skimage.registration.html#skimage.registration.phase_cross_correlation>`_
+method.
+
+Compute Beam Boundaries for Identical Size Overlapping Regions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The final step is to use the shift to adjust the boundaries of each beam image such that both images
+have the same horizontal dimension and represent essentially the same spatial and spectral regions
+(as mentioned above, final adjustments will come later in the geometric calibration). Moreover the beam
+boundaries must be defined so that they represent slices into the original image and can then be used
+later to extract each beam from any type of input image (dark, gain, polcal, observe).

dkist_processing_cryonirsp-0.0.56/docs/ci_science_calibration.rst

@@ -0,0 +1,33 @@
+CI L1 Science Calibration
+=========================
+
+Introduction
+------------
+
+The `~dkist_processing_cryonirsp.tasks.ci_science` module takes L0 input science frames and fully calibrates
+them into L1 science products. This page describes the basic steps in this processes as well as important
+features of the Cryo-NIRSP Context Imager (CI) algorithm that may not be obvious.
+
+L1 Coordinate System
+^^^^^^^^^^^^^^^^^^^^
+
+The final step of the science pipeline places L1 data into a coordinate frame that matches the coordinates used by
+SDO/HMI and HINDOE-SP. Namely, -Q and +Q will be aligned parallel and perpendicular to the central meridian of the Sun,
+respectively.
+
+Algorithm
+---------
+
+Input CI science data is processed into L1 science data via the following steps:
+
+
+#.  Dark signals are subtracted from linearized input data.
+
+#.  :doc:`Bad Pixel correction <bad_pixel_calibration>` is done.
+
+#.  A solar gain calibration frame is divided from the data.
+
+#.  If data is polarimetric, demodulation matricies are applied.
+
+#.  If data is polarimetric, the Telescope Polarization is removed. This removes the polarization effects of all DKIST mirrors upstream
+    of Cryo-NIRSP. This step also includes the rotation into the coordinate frame described above.

{dkist_processing_cryonirsp-0.0.55 → dkist_processing_cryonirsp-0.0.56}/docs/index.rst

@@ -11,3 +11,9 @@
     requirements_table
     changelog
     scientific_changelog
+    linearization
+    bad_pixel_calibration
+    beam_boundary_computation
+    beam_angle_calculation
+    ci_science_calibration
+    sp_science_calibration

dkist_processing_cryonirsp-0.0.56/docs/l0_to_l1_cryonirsp_ci.rst

@@ -0,0 +1,10 @@
+l0_to_l1_cryonirsp_ci
+=====================
+
+In the normal Cryo-NIRSP Context Imager (CI) operating mode, raw Cryo-NIRSP data are gathered at the summit and
+delivered to the Data Center. The Data Center then calibrates this data and prepares it for storage using the
+following workflow.
+
+For more detail on each workflow task, you can click on the task in the diagram.
+
+.. workflow_diagram:: dkist_processing_cryonirsp.workflows.ci_l0_processing.l0_pipeline

dkist_processing_cryonirsp-0.0.56/docs/l0_to_l1_cryonirsp_sp.rst

@@ -0,0 +1,10 @@
+l0_to_l1_cryonirsp_sp
+=====================
+
+In the normal Cryo-NIRSP Spectropolarimetric (SP) operating mode, raw Cryo-NIRSP data are gathered at the summit
+and delivered to the Data Center. The Data Center then calibrates this data and prepares it for storage using
+the following workflow.
+
+For more detail on each workflow task, you can click on the task in the diagram.
+
+.. workflow_diagram:: dkist_processing_cryonirsp.workflows.sp_l0_processing.l0_pipeline

dkist_processing_cryonirsp-0.0.56/docs/linearization.rst

@@ -0,0 +1,43 @@
+Linearization
+=============
+
+Introduction
+------------
+
+The Cryo-NIRSP camera has an H2RG detector, which has a non-linear response to light with
+increasing exposure time. This can vary from pixel to pixel. Because of the non-linear response, the count
+values at the final exposure do not accurately represent the light falling on the chip and therefore need to
+be corrected. We call this correction ‘linearization’. During an exposure, the camera
+reads out multiple frames and saves them while it continues to expose. The accumulated charge on the chip is
+not erased when this happens, so they are referred to as Non-Destructive Readouts (NDRs). The NDRs are
+saved at a pre-selected rate that is a fraction of the overall desired exposure time for a
+complete NDR. A set of NDRs with exposure times varying from 0 to the final desired exposure time are
+called a ramp. The linearization process reads all of the NDRs associated with a particular ramp and then
+applies an algorithm to compute what the final count values at the desired exposure time should be if the
+response of the detector was linear. Basically it takes the ramp set and "linearizes" it to produce a single
+NDR that has the correct counts at each pixel.
+
+A ramp can have many frames (anywhere from 10 to as much as 100, or more), and the linearization
+algorithm is agnostic to just about everything except the final exposure time. So after linearization, we
+have over 10x fewer frames to process, and the pipeline proceeds as though the linearized frames are raw input frames.
+
+The Linearization Algorithm
+---------------------------
+
+The Algorithm
+^^^^^^^^^^^^^
+
+First, we identify all NDRs from a single exposure of the Cryo-NIRSP cameras (i.e., a ramp set). A ramp is
+identified as all the files having the same DATE-OBS value. Note: If a ramp set contains only a single frame,
+it is discarded.
+
+Next, we need to identify the NDRs that fall in the “linear” portion of the chip response. These are those
+NDRs whose pixel value is below a pre-computed threshold value. I.e., the threshold values define the upper-limit
+of the linear response of each pixel.
+
+Then, we linearize this ramp set by normalizing the measured raw flux by the final ramp exposure time of the ramp set.
+Essentially, this removes the non-linear response of each pixel in the array. The resulting ramp set is essentially
+linear in ADUs v exposure time.
+
+Finally, we normalize the array by converting to counts per second and correct the counts for the Optical Density
+filter used during observations.

dkist_processing_cryonirsp-0.0.56/docs/sp_science_calibration.rst

@@ -0,0 +1,59 @@
+SP L1 Science Calibration
+=========================
+
+Introduction
+------------
+
+The `~dkist_processing_cryonirsp.tasks.sp_science` module takes L0 input science frames and fully calibrates
+them into L1 science products. This page describes the basic steps in this processes as well as important
+features of the Cryo-NIRSP Spectropolarimetric (SP) algorithm that may not be obvious.
+
+Important Features
+------------------
+
+Beam Combination
+^^^^^^^^^^^^^^^^
+
+Apart from the order in which the basic corrections are applied (described below), it is important to state how the two
+polarimetric beams of Cryo-NIRSP are combined to produce a single L1 data frame. After demodulation the 4 Stokes components of
+the two beams are combined thusly:
+
+.. math::
+
+  I_{comb} &= (I_1 + I_2) / 2 \\
+  Q_{comb} &= I_{comb} \left(\frac{Q_1}{I_1} + \frac{Q_2}{I_2}\right) / 2 \\
+  U_{comb} &= I_{comb} \left(\frac{U_1}{I_1} + \frac{U_2}{I_2}\right) / 2 \\
+  V_{comb} &= I_{comb} \left(\frac{V_1}{I_1} + \frac{V_2}{I_2}\right) / 2,
+
+where numbered subscripts correspond to beam number. This combination scheme improves the signal-to-noise of the data
+and mitigates residual polarization artifacts caused by temporal-based modulation (e.g., atmospheric seeing).
+
+L1 Coordinate System
+^^^^^^^^^^^^^^^^^^^^
+
+The final step of the science pipeline places L1 data into a coordinate frame that matches the coordinates used by
+SDO/HMI and HINDOE-SP. Namely, -Q and +Q will be aligned parallel and perpendicular to the central meridian of the Sun,
+respectively.
+
+Algorithm
+---------
+
+Input SP science data is processed into L1 science data via the following steps:
+
+#.  Dark signals are subtracted from linearized input data.
+
+#.  :doc:`Bad Pixel correction <bad_pixel_calibration>` is done.
+
+#.  A solar gain calibration frame is divided from the data.
+
+#.  If data is polarimetric, demodulation matrices are applied.
+
+#.  Geometric distortions (spectral rotation, x/y shift, spectral curvature) are removed via interpolation.
+    This step aligns the dispersion axis with a pixel axis, places both beams on the same pixel grid, and
+    straightens the spectra so that a single spectral pixel corresponds to the same physical wavelength for
+    all locations along the slit.
+
+#.  The beams are combined as described above. For non-polarimetric data, the combination is a simple average.
+
+#.  If data is polarimetric, the Telescope Polarization is removed. This removes the polarization effects of all DKIST mirrors upstream
+    of Cryo-NIRSP. This step also includes the rotation into the coordinate frame described above.

dkist_processing_cryonirsp-0.0.55/docs/l0_to_l1_cryonirsp_ci.rst

@@ -1,11 +0,0 @@
-l0_to_l1_cryonirsp_ci
-=====================
-
-In the normal CryoNIRSP operating mode, raw CryoNIRSP data is gathered at the summit and delivered to the Data Center.
-The Data Center then calibrates this data and prepares it for storage using the following workflow.
-
-For more detail on each workflow task, you can click on the task in the diagram.
-
-.. workflow_diagram:: dkist_processing_cryonirsp.workflows.ci_l0_processing.l0_pipeline
-
-In this workflow, raw dark, gain, and polarization calibration data is used to generate calibration products that are then applied to the science frames before repackaging them for storage and delivery to a science user.

dkist_processing_cryonirsp-0.0.55/docs/l0_to_l1_cryonirsp_sp.rst

@@ -1,11 +0,0 @@
-l0_to_l1_cryonirsp_sp
-=====================
-
-In the normal CryoNIRSP operating mode, raw CryoNIRSP data is gathered at the summit and delivered to the Data Center.
-The Data Center then calibrates this data and prepares it for storage using the following workflow.
-
-For more detail on each workflow task, you can click on the task in the diagram.
-
-.. workflow_diagram:: dkist_processing_cryonirsp.workflows.sp_l0_processing.l0_pipeline
-
-In this workflow, raw dark, gain, and polarization calibration data is used to generate calibration products that are then applied to the science frames before repackaging them for storage and delivery to a science user.