SpecProc 1.0.0__py3-none-any.whl

specproc-1.0.0.dist-info/METADATA

@@ -0,0 +1,668 @@
+Metadata-Version: 2.4
+Name: SpecProc
+Version: 1.0.0
+Summary: A spectral processing tool
+Author-email: "Niu, Hubiao" <niuhubiao@xao.ac.cn>
+License: MIT
+Project-URL: Homepage, https://github.com/yourusername/specproc
+Project-URL: Repository, https://github.com/yourusername/specproc
+Keywords: spectroscopy,FITS,data-reduction
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: astropy
+Requires-Dist: matplotlib
+Requires-Dist: PyQt5
+Requires-Dist: configparser
+Dynamic: license-file
+
+# SpecProc: PyQt GUI for Spectrograph Data Reduction
+
+![SpecProc GUI](docs/SpecProc.png)
+
+A complete PyQt-based graphical interface for reducing slit/chelle spectrograph FITS data.
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration](#configuration)
+- [Processing Pipeline](#processing-pipeline)
+- [Usage](#usage)
+- [Calibration Data](#calibration-data)
+- [Troubleshooting](#troubleshooting)
+- [Documentation](#documentation)
+
+## Features
+
+### Complete Reduction Pipeline
+
+**8-stage automated spectral reduction**:
+
+1. **Basic Pre-processing** - Overscan, bias subtraction, and cosmic ray correction
+2. **Apertures Tracing** - Master flat generation and echelle aperture tracing
+3. **Scattered Light Subtraction** - Inter-aperture background modeling and removal (astropy convolution or spline)
+4. **2D Flat-Field Correction** - 2D pixel flat correction
+5. **1D Spectrum Extraction** - 1D spectrum extraction for each aperture (sum or optimal)
+6. **De-blazing** - Blaze function correction
+7. **Wavelength Calibration** - Wavelength calibration applied to both sum and optimal de-blazed 1D spectra
+8. **Order Stitching** - Merge overlapping neighboring orders into a continuous 1D spectrum
+
+### Interactive GUI
+
+PyQt5-based user interface with:
+- File management for bias, flat, and science frames
+- Real-time progress tracking
+- Processing logs and diagnostics
+- One-click pipeline execution or step-by-step processing
+
+### Key Features
+
+- **Configuration-Driven**: INI-format configuration files for easy parameter adjustment
+- **Generic Spectrograph Support**: Configurable for different echelle spectrographs
+- **Flexible Output**: Control which intermediate results to save
+- **Command Line Interface**: Alternative to GUI for batch processing
+
+## Installation
+
+Choose one of the following installation methods:
+
+### Method 1: Using pip
+
+Install SpecProc directly from PyPI.
+
+```bash
+# Install SpecProc
+pip install specproc
+
+# Launch application
+specproc
+```
+
+**Notes:**
+- Requires Python 3.7+
+- Dependencies will be automatically installed from PyPI
+- Installation is permanent; uninstall with `pip uninstall specproc`
+
+### Method 2: Using conda
+
+Conda provides a complete environment with all dependencies. Two installation options:
+
+#### Option 2.1: Install in existing conda environment
+
+```bash
+# Activate your conda environment
+conda activate your_environment
+
+# Install SpecProc from conda-forge
+conda install -c conda-forge specproc
+
+# Launch application
+specproc
+```
+
+#### Option 2.2: Create new conda environment
+
+```bash
+# Create new conda environment for SpecProc
+conda create -n specproc python=3.8
+conda activate specproc
+
+# Install SpecProc and all dependencies
+conda install -c conda-forge specproc
+
+# Launch application
+specproc
+```
+
+**Notes:**
+- Recommended for users who want an isolated environment
+- All dependencies are managed by conda
+- Python 3.7-3.11 are supported
+
+### Method 3: Installing from Source
+
+Run the installation script to install SpecProc and all dependencies from local source.
+
+```bash
+# Navigate to SpecProc directory
+cd /path/to/SpecProc
+
+# Make the script executable (if needed)
+chmod +x install.sh
+
+# Run installation script
+./install.sh
+
+# Launch application
+specproc
+```
+
+**Notes:**
+- Installation script handles all dependencies automatically
+- Detects available package manager (pip or conda)
+- Installs SpecProc to your system
+- Use this method for automated setup from local source
+
+## Quick Start
+
+### Working Directory Setup
+
+**Important**: SpecProc should be run in your working directory, NOT in the source code directory.
+
+### Correct Workflow
+
+```bash
+# 1. Create your working directory (e.g., for an observation project)
+mkdir -p /myworkspace
+cd /myworkspace
+
+# 2. Create subdirectories for data processing
+mkdir -p 20241102_hrs output
+
+# 3. Copy/move FITS data files to 20241102_hrs directory
+cp /somewhere/bias_*.fits ./20241102_hrs/
+cp /somewhere/flat_*.fits ./20241102_hrs/
+cp /somewhere/thar_*.fits ./20241102_hrs/
+cp /somewhere/science_*.fits ./20241102_hrs/
+
+# 4. Create user config file (optional)
+cp /path/to/SpecProc/src/specproc/config/default_config.cfg ./specproc.cfg
+
+# 5. Run SpecProc in your working directory
+specproc --config ./specproc.cfg
+```
+
+### Directory Structure
+
+**Working directory** (where you process data):
+```
+/myworkspace/                  # Your working directory
+├── 20241102_hrs/              # Input FITS files
+│   ├── bias_*.fits
+│   ├── flat_*.fits
+│   ├── thar_*.fits      # ThAr lamp spectrum
+│   └── science_*.fits   # Science images
+├── output/                    # Processing results (auto-created)
+│   ├── step1_basic/            # Step 1: Basic pre-processing
+│   │   ├── overscan_corrected/ # Overscan-corrected images and diagnostic plots
+│   │   ├── bias_subtracted/    # Master bias and bias-corrected images
+│   │   └── cosmic_corrected/   # Cosmic ray corrected science images (if enabled)
+│   ├── step2_flat/             # Step 2: Master flat field, blaze profiles, and diagnostic plots
+│   ├── step3_background/       # Step 3: Background model and diagnostic plots
+│   ├── step5_extraction/       # Step 5: Extracted 1D spectra and diagnostic plots
+│   ├── step6_deblazing/        # Step 6: De-blazed spectra and diagnostic plots (if saved)
+│   └── step7_wavelength/       # Step 7: Final wavelength calibrated 1D spectra and plots
+│   └── step8_stitching/        # Step 8: Final stitched continuous 1D spectra and plots
+├── specproc.cfg              # User config file (optional)
+└── ...
+```
+
+**Note**:
+- ❌ Do NOT run `specproc` in SpecProc source directory (`/path/to/SpecProc`)
+- ✅ Run `specproc` in your working directory
+- ✅ `rawdata` and `output` will be created in your working directory
+
+## Configuration
+
+### Config File Types
+
+#### Default Configuration
+
+**Location**: `SpecProc/src/specproc/config/default_config.cfg`
+**Purpose**: Provides default parameter values
+**Modification**: Not recommended to modify directly
+
+#### User Configuration
+
+**Location**: `specproc.cfg` in working directory
+**Purpose**: Override default configuration, customize parameters
+**Priority**: User config > Default config
+
+### Path Configuration
+
+#### Data Paths
+
+```ini
+[data]
+# Raw FITS data directory path
+# Example: If running in /myworkspace/ and rawdata_path=20241102_hrs,
+# data will be loaded from /myworkspace/20241102_hrs/
+#
+# Path behavior:
+# - rawdata_path = /data/20241102_hrs  → Absolute path, loads from /data/20241102_hrs/
+# - rawdata_path = ./20241102_hrs       → Relative path, loads from working_directory/20241102_hrs/
+# - rawdata_path = 20241102_hrs         → Same as above, also relative to working directory
+#   (e.g., if working in /myworkspace/, loads from /myworkspace/20241102_hrs/)
+#
+# Examples (assuming working directory is /myworkspace/):
+#   rawdata_path = ./20241102_hrs      → Data from /myworkspace/20241102_hrs/
+#   rawdata_path = 20241102_hrs        → Same as above, data from /myworkspace/20241102_hrs/
+#   rawdata_path = /data/20241102_hrs  → Data from /data/20241102_hrs/
+rawdata_path = ./20241102_hrs
+```
+
+#### Output Path
+
+```ini
+[reduce]
+# Output directory path for all processing results
+# Example: If running in /myworkspace/ and output=output,
+# results will be saved in /myworkspace/output/
+#
+# Path behavior:
+# - output_path = /data/output      → Absolute path, saves to /data/output/
+# - output_path = ./output          → Relative path, saves to working_directory/output/
+# - output_path = output            → Same as above, also relative to working directory
+#   (e.g., if working in /myworkspace/, saves to /myworkspace/output/)
+#
+# Examples (assuming working directory is /myworkspace/):
+#   output_path = ./output          → Results to /myworkspace/output/
+#   output_path = output            → Same as above, results to /myworkspace/output/
+#   output_path = /data/output      → Results to /data/output/
+#
+# Output directory structure:
+# output/
+#   ├── step1_basic/            # Step 1: Basic pre-processing
+#   │   ├── overscan_corrected/ # Overscan-corrected images
+#   │   ├── bias_subtracted/    # Master bias and bias-corrected images
+#   │   └── cosmic_corrected/   # Cosmic ray corrected science images (if enabled)
+#   ├── step2_flat/             # Step 2: Master flat field with blaze profiles
+#   ├── step3_background/       # Step 3: Background model
+#   ├── step5_extraction/       # Step 5: Extracted 1D spectra (pixel space)
+#   ├── step6_deblazing/        # Step 6: De-blazed spectra (if saved)
+#   └── step7_wavelength/       # Step 7: Final wavelength calibrated 1D spectra
+output_path = output
+
+#### Path Examples
+
+```bash
+# Assume working directory is /myworkspace/
+cd /myworkspace/
+
+# Config file (relative path example):
+[data]
+rawdata_path = ./20241102_hrs
+[reduce]
+output_path = ./output
+
+# Actual paths used:
+# Input:  /myworkspace/20241102_hrs/
+# Output: /myworkspace/output/
+
+# Config file (absolute path example):
+[data]
+rawdata_path = /data/20241102_hrs
+[reduce]
+output_path = /data/output
+
+# Actual paths used:
+# Input:  /data/20241102_hrs/
+# Output: /data/output/
+```
+
+### Intermediate Results Saving
+
+Control which processing steps to save intermediate results:
+
+```ini
+[reduce.save_intermediate]
+# Whether to save intermediate results for each step
+# Set to 'yes' or 'no' for each step independently
+# Default is 'yes' for all steps
+save_overscan = yes        # Step 1: Overscan correction (saves to output/step1_basic/overscan_corrected/)
+save_bias = yes             # Step 1: Bias correction (saves to output/step1_basic/bias_subtracted/)
+save_flat = yes              # Step 2: Flat fielding (saves master flat to output/step2_flat/)
+save_background = yes        # Step 3: Background subtraction (saves model to output/step3_background/)
+save_cosmic = yes           # Step 1: Cosmic ray correction (saves to output/step1_basic/cosmic_corrected/)
+save_extraction = yes       # Step 5: Spectrum extraction (saves to output/step5_extraction/)
+save_deblaze = yes          # Step 6: De-blazing (saves to output/step6_deblazing/)
+save_wlcalib = yes          # Step 7: Wavelength calibration (saves solution to output/step7_wavelength/)
+```
+
+**Effect**:
+- If a step is set to `no`, the corresponding output subdirectory will NOT be created
+- Final spectra are always saved to `output/step8_final_spectra/` regardless of these settings
+- Diagnostic plots can be saved in corresponding step subdirectories
+- In GUI, there should be corresponding checkboxes to enable/disable saving
+- Default: All steps save intermediate results
+
+### Telescope and Calibration Configuration
+
+```ini
+[telescope]
+# Telescope name for calibration lookup
+name = xinglong216hrs
+
+# Spectrograph instrument name
+instrument = hrs
+
+[telescope.linelist]
+# Lamp linelist type
+linelist_type = ThAr
+
+# Path to linelist files
+linelist_path = calib_data/linelists/
+
+# Specific linelist file to use (optional)
+# For Xinglong 2.16m HRS: thar-noao.dat is recommended
+linelist_file = thar-noao.dat
+
+# Use pre-identified calibration files (optional)
+use_precomputed_calibration = yes
+calibration_path = calib_data/telescopes/xinglong216hrs/
+
+# Specific calibration file to use (optional)
+# Use latest: wlcalib_20211123011_A.fits
+calibration_file = wlcalib_20211123011_A.fits
+```
+
+## Processing Pipeline
+
+### Stage Descriptions
+
+#### STEP 1: Basic Pre-processing
+- **Input**: Raw FITS files (bias, flat, ThAr, science)
+- **Processing**:
+  - Extract overscan region (readout bias area)
+  - Calculate median or polynomial fit
+  - Subtract overscan bias from image
+  - Combine multiple bias frames (mean/median)
+  - Generate master bias
+  - Subtract master bias from science/flat/ThAr images
+  - Detect and remove cosmic rays using L.A.Cosmic (science images only)
+- **Output**: Pre-processed images (overscan, bias, cosmic-ray corrected)
+- **Note**: Fundamental corrections applied to prepare data for tracing and extraction.
+
+#### STEP 2: Apertures Tracing
+- **Input**: Pre-processed flat frames
+- **Processing**:
+  - Combine flat frames
+  - Generate master flat
+  - Detect echelle apertures
+  - Fit polynomial traces for each aperture
+  - Extract blaze profiles
+- **Output**: Master flat, apertures, and blaze profiles
+- **Note**: Provides apertures and blaze profiles for later stages
+
+#### STEP 3: Scattered Light Subtraction
+- **Input**: Pre-processed science image
+- **Processing**:
+  - Estimate background scattered light using 2D convolution or splines
+  - Subtract background model from science image
+- **Output**: Background subtracted image
+- **Note**: Removes inter-aperture stray light.
+
+#### STEP 4: 2D Flat-Field Correction
+- **Input**: Background subtracted science image
+- **Processing**:
+  - Generate 2D pixel-to-pixel flat correction map
+  - Apply 2D flat correction to science image
+- **Output**: 2D Flat-fielded image
+- **Note**: Corrects for pixel-to-pixel sensitivity variations.
+
+#### STEP 5: 1D Spectrum Extraction
+- **Input**: 2D Flat-fielded image
+- **Processing**:
+  - Extract 1D spectrum for each aperture
+  - Method: Sum extraction or Optimal extraction (Horne 1986)
+  - Calculate extraction errors
+- **Output**: SpectraSet (pixel space)
+- **Note**: Converts 2D traces to 1D pixel space spectra.
+
+#### STEP 6: De-blazing
+- **Input**: Extracted 1D spectra (pixel space)
+- **Processing**:
+  - Read flat spectrum blaze function (in pixel space)
+  - Match apertures
+  - Divide by blaze function: F_corrected(λ) = F_observed(λ) / B(λ)
+  - Normalize to unit continuum
+- **Output**: De-blazed spectra
+- **Note**: Corrects for the blaze function of the spectrograph grating. Must be performed after wavelength calibration.
+
+#### STEP 7: Wavelength Calibration
+- **Input**: De-blazed 1D spectra (pixel space)
+- **Processing**:
+  - Step 1: Calibrate ThAr lamp spectrum
+    - Extract 1D spectrum
+    - Identify emission lines
+    - Fit 2D wavelength polynomial: λ(x,y) = Σ p_ij·x^i·y^j
+  - Step 2: Apply to science spectra
+    - Convert pixel coordinates to wavelength units
+- **Output**: Wavelength-calibrated 1D spectra
+- **Note**: Establishes the physical wavelength scale for each order.
+
+
+## Usage
+
+### GUI Mode (Default)
+
+```bash
+# Launch GUI (default mode)
+specproc
+
+# Or explicitly specify GUI mode
+specproc --mode gui
+
+# With custom config file
+specproc --config /path/to/config.cfg
+```
+
+**GUI Workflow**:
+1. Select bias files
+2. Select flat files
+3. Select calibration files (ThAr lamp)
+4. Select science files
+5. Click "Run Full Pipeline" or execute stages step-by-step
+6. View progress in real-time
+7. Check results in output directory
+
+### CLI Mode (Command Line)
+
+```bash
+# Run CLI mode
+specproc --mode cli
+
+# Or with custom config file
+specproc --mode cli --config /path/to/config.cfg
+
+# Or select a packaged telescope profile by name, and process data with it
+specproc --mode cli --config XL216_hrs 20241102_hrs
+
+# Without input data, the named profile is copied to ./specproc.cfg instead
+# of being used directly, so it can be reviewed/customized first
+specproc --config MSTG193_slit
+```
+
+**Telescope profiles**: SpecProc ships one `default_config_<profile>.cfg` file
+per supported telescope/instrument under `src/specproc/config/` (e.g.
+`XL216_hrs` for the Xinglong 2.16m HRS, `MSTG193_slit` for the Muztagh-Ata
+1.93m slit spectrograph). Passing the profile name to `--config` resolves it
+to that packaged file automatically — no need to type the full path. Run
+`specproc --help` to see the list of available profiles.
+
+**Slit mode** (`[telescope] instrument = slit`, used by `MSTG193_slit`): a
+separate single-aperture pipeline (`core/slit_pipeline.py`) replaces the echelle
+stages: overscan/bias → pixel flat → cosmic rays → star trace + sky subtraction →
+sum/optimal extraction → 1-D wavelength solution. Frame types are taken from file
+names (`frame_type_source = filename`; bias / flat / FeAr / dark, rest = science).
+Run the red and blue cameras separately (one directory each). Wavelength
+calibration needs `[slit] identification_file` (pixel, wavelength rows; candidate
+pixels are written to `step7_wavelength/<lamp frame>_peaks.csv`) or `wave_start`/`wave_end`
+hints; otherwise spectra are saved on a pixel axis.
+
+Slit-mode output layout (same numbering as HRS; steps not needed are omitted):
+`step1_basic/{overscan_corrected,bias_subtracted,cosmic_corrected}`, `step2_trace` (slit trace of the master flat: centre and both edges, plus each
+science frame's trace = flat centre line + a constant star offset; the same trace
+is applied to the science and lamp frames),
+`step4_flat_corrected` (master/pixel flat and flat-fielded images), `step5_extraction`
+(sky-subtracted images and 1-D spectra of the targets, every lamp frame and the master flat,
+all extracted along the master-flat trace), `step7_wavelength`. No scattered-light
+(step3) or de-blazing (step6) directories are created.
+
+**Extraction aperture (slit mode)**: with `[slit] aperture_mode = auto` the spatial FWHM of each
+science frame is measured from the profile stacked along the whole dispersion axis
+(usable for faint targets); the median over frames sets the extraction half-width
+(`aperture_fwhm_factor` × FWHM, default 1.7) and the start of the sky region
+(`sky_fwhm_factor` × FWHM, default 3). If the FWHM cannot be measured the fixed
+`extract_half_width` / `sky_inner_offset` are used; `aperture_mode = fixed` always uses them.
+
+**1-D SNR (slit mode)**: SNR is estimated at 25/50/75 % of the effective spectral
+range (window ±`snr_half_window` px; signal = median flux, noise = standard deviation
+of the residual to a quadratic fit). SNR is a dimensionless ratio, labelled
+`SNR=xx@position`: the position is in pixels before wavelength calibration
+(`step5_extraction/*_spec_pixel.pdf`) and in Å after it (`step7_wavelength/*_spec_wave.pdf`).
+Values are annotated on the plots and stored as `SNRPxx`/`SNRWxx` keywords in the `*_spec.fits` table.
+
+**CLI Workflow**:
+1. Follow prompts to select files
+2. Select processing stages (0-7, or Enter for all)
+3. Monitor console progress
+4. Check results in output directory
+
+## Calibration Data
+
+### Directory Structure
+
+```
+calib_data/
+├── linelists/              # Lamp emission line catalogs
+│   ├── thar-noao.dat      # ThAr lamp lines (Xinglong 2.16m HRS recommended)
+│   ├── thar.dat           # Standard ThAr lamp lines
+│   ├── FeAr.dat           # FeAr lamp lines
+└── ...
+└── telescopes/             # Telescope-specific calibration files
+    ├── generic/           # Generic configuration template
+    └── xinglong216hrs/    # Xinglong 2.16m telescope
+        ├── wlcalib_20141103049.fits
+        ├── wlcalib_20171202012.fits
+        ├── wlcalib_20190905028_A.fits
+        └── wlcalib_20211123011_A.fits
+```
+
+### Lamp Line Lists
+
+**Available linelist files**:
+- `thar-noao.dat` - ThAr lamp lines (Xinglong 2.16m HRS recommended)
+- `thar.dat` - Standard ThAr lamp lines
+- `FeAr.dat` - FeAr lamp lines
+
+**Supported lamp types**:
+- `ThAr` - Thorium-Argon (most common for echelle spectrographs)
+- `FeAr` - Iron-Argon
+- `Ar` - Argon
+- `Ne` - Neon
+- `He` - Helium
+- `Fe` - Iron
+
+### Telescope Calibrations
+
+**Available calibration files for Xinglong 2.16m HRS**:
+- `wlcalib_20141103049.fits` - 2014-11-03 04:50
+- `wlcalib_20171202012.fits` - 2017-12-02 01:20
+- `wlcalib_20190905028_A.fits` - 2019-09-05 02:50 (version A)
+- `wlcalib_20211123011_A.fits` - 2021-11-23 01:10 (version A) - **Latest**
+
+### Configuration
+
+```ini
+[telescope]
+name = xinglong216hrs
+instrument = hrs
+
+[telescope.linelist]
+linelist_type = ThAr
+linelist_path = calib_data/linelists/
+linelist_file = thar-noao.dat
+use_precomputed_calibration = yes
+calibration_path = calib_data/telescopes/xinglong216hrs/
+calibration_file = wlcalib_20211123011_A.fits
+```
+
+## Troubleshooting
+
+### ImportError: No module named 'PyQt5'
+
+```bash
+# Install PyQt5
+pip install PyQt5
+```
+
+### specproc command not found
+
+```bash
+conda activate specproc
+pip install -e .
+```
+
+### Configuration file not found
+
+```bash
+# Copy default config
+cp /path/to/SpecProc/src/specproc/config/default_config.cfg ./specproc.cfg
+```
+
+### Large file errors on GitHub
+
+**Error**: `File exceeds GitHub's file size limit of 100.00 MB`
+
+**Solution**: Large FITS files should not be committed. Use `.gitignore` to exclude them.
+
+**Prevent future additions**:
+- Add output directories to `.gitignore`
+- Run SpecProc in separate working directory, not in source directory
+
+## Documentation
+
+- See [calib_data/README.md](calib_data/README.md) for calibration data configuration
+- See [README_CN.md](README_CN.md) for Chinese documentation
+- See [PIPELINE_FLOWCHART.md](PIPELINE_FLOWCHART.md) for detailed processing workflow
+
+## Project Structure
+
+```
+SpecProc/
+├── README.md                    # Main documentation
+├── README_CN.md                # Main documentation (Chinese)
+├── specproc.cfg.example         # Example user configuration
+├── install.sh                   # Installation script
+├── requirements.txt              # Python dependencies
+├── run.py                      # Main entry point
+├── setup.py                     # Installation configuration
+├── LICENSE                      # License
+├── .gitignore                  # Git ignore rules
+├── src/
+│   └── specproc/               # Source code
+│       ├── calib_data/              # Calibration data
+│       │   ├── linelists/
+│       │   └── telescopes/
+│       ├── default_config.cfg       # Default configuration
+│       ├── gui/                     # GUI modules
+│       ├── core/                    # Core processing
+│       ├── config/                  # Configuration management
+│       ├── utils/                   # Utility functions
+│       └── plotting/                # Plotting functions
+└── test_*.py                    # Test files
+```
+
+## License
+
+See LICENSE file for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## Acknowledgments
+
+- Inspired by [gamse](https://github.com/wangleon/gamse) package
+- Built with PyQt5, NumPy, SciPy and Astropy
+
+## Support
+
+For issues and questions, please open an issue on GitHub.