pyreduce-astro 0.4.37__tar.gz → 0.7__tar.gz

pyreduce_astro-0.7/.gitattributes

@@ -0,0 +1 @@
+pyreduce/_version.py export-subst

pyreduce_astro-0.7/.gitignore

@@ -0,0 +1,26 @@
+datasets
+DATA
+__pycache__
+*.sw[op]
+.coverage
+cov.xml
+*.so
+*.o
+pyreduce/clib/_slitfunc_2d.c
+pyreduce/clib/_slitfunc_bd.c
+pyreduce_astro.egg*
+.vscode
+*.py[cod]
+.venv
+.uv
+esorex.log
+pyesorex.log
+test/test_outputs
+docs/_build/
+.claude
+debug/
+idl/
+logs/
+*.ipynb
+tmp/*
+.obsidian/

pyreduce_astro-0.7/.pre-commit-config.yaml

@@ -0,0 +1,33 @@
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v6.0.0
+    hooks:
+    # Identify invalid files
+    -   id: check-ast
+    -   id: check-yaml
+    -   id: check-json
+    -   id: check-toml
+    # git checks
+    -   id: check-merge-conflict
+    -   id: check-added-large-files
+    -   id: detect-private-key
+    -   id: check-case-conflict
+    # Python checks
+    -   id: check-docstring-first
+    -   id: debug-statements
+    # General checks
+    -   id: mixed-line-ending
+        exclude: ".fits"
+    -   id: trailing-whitespace
+        exclude: ".fits"
+    -   id: end-of-file-fixer
+
+# Ruff - Modern Python linter and formatter (replaces isort, black, flake8, pyupgrade)
+-   repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.13.2
+    hooks:
+    # Run the linter
+    -   id: ruff
+        args: [--fix]
+    # Run the formatter
+    -   id: ruff-format

pyreduce_astro-0.7/.python-version

@@ -0,0 +1 @@
+3.14

pyreduce_astro-0.7/.readthedocs.yaml

@@ -0,0 +1,25 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF and ePub
+formats: all
+
+# Set the version of Python and install from pyproject.toml
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.13"
+
+python:
+  install:
+    - method: pip
+      path: .
+    - requirements: docs/requirements.txt

pyreduce_astro-0.7/AGENTS.md

@@ -0,0 +1 @@
+CLAUDE.md

pyreduce_astro-0.7/ANDES_plan.md

@@ -0,0 +1,182 @@
+# ANDES Instrument Configuration Design
+
+## Background
+
+### ANDES Instrument Overview
+
+ANDES is a future ELT high-resolution spectrograph with multiple spectrographs, each containing multiple arms:
+
+| Spectrograph | Arms | Detector Type |
+|--------------|------|---------------|
+| 1 | U, B, V | Optical CCD |
+| 2 | R, IZ | Red optical CCD |
+| 3 | Y, J, H | Near-IR (H2RG or similar) |
+| 4 | K | Thermal IR |
+
+Key characteristics:
+- All arms fed from the same fibers on sky
+- Each arm has its own independent detector
+- FITS files organized **per spectrograph** (one file with arms as extensions)
+- Detectors are similar within each spectrograph
+
+### Terminology
+
+| ANDES term | PyReduce term |
+|------------|---------------|
+| Spectrograph | Instrument |
+| Arm | Channel |
+
+### Current State
+
+- `ANDES_YJH/` - Working instrument for near-IR spectrograph (Y, J, H arms)
+- Old `ANDES/` directory removed (was empty)
+
+---
+
+## PyReduce Architecture Context
+
+### How Channels Work
+
+Channels in PyReduce handle multi-extension FITS files. The `getter` class in `instruments/common.py` automatically indexes list values by channel position:
+
+```yaml
+# Example: UVES with 3 channels
+channels: [BLUE, MIDDLE, RED]
+extension: [0, 2, 1]        # BLUE→ext0, MIDDLE→ext2, RED→ext1
+orientation: [2, 1, 1]      # BLUE rotated differently
+```
+
+When processing channel "RED" (index 2), PyReduce automatically uses `extension[2]=1`, `orientation[2]=1`.
+
+### Per-Channel Settings
+
+PyReduce supports per-channel settings files. When loading configuration for a channel, it looks for `settings_{channel}.json` first, falling back to `settings.json`.
+
+Example from MOSAIC:
+```
+MOSAIC/
+├── settings.json           # base: __inherits__: "defaults"
+├── settings_nir.json       # __inherits__: "MOSAIC", NIR-specific tuning
+├── settings_VIS1.json      # __inherits__: "MOSAIC", VIS1-specific tuning
+└── ...
+```
+
+This allows different arms to have different reduction parameters while sharing a single instrument config.
+
+### Channel-Based File Filtering
+
+Files can be filtered by channel using `kw_channel` and `id_channel`:
+
+```yaml
+# From ANDES_YJH config
+kw_channel: BAND
+id_channel: ["^Y$", "^J$", "^H$"]
+```
+
+When reducing channel `J`, only files with `BAND=J` header are selected.
+
+---
+
+## Design Options
+
+### Option A: Single ANDES Instrument (9 channels)
+
+```yaml
+# andes/config.yaml
+channels: [U, B, V, R, IZ, Y, J, H, K]
+extension: [1, 2, 3, 1, 2, 1, 2, 3, 1]
+kw_channel: BAND
+id_channel: ["^U$", "^B$", "^V$", "^R$", "^IZ$", "^Y$", "^J$", "^H$", "^K$"]
+```
+
+With per-channel settings:
+```
+ANDES/
+├── config.yaml
+├── settings.json           # base settings
+├── settings_U.json         # optical CCD tuning
+├── settings_B.json
+├── settings_V.json
+├── settings_R.json
+├── settings_IZ.json
+├── settings_Y.json         # near-IR tuning
+├── settings_J.json
+├── settings_H.json
+└── settings_K.json         # thermal IR tuning
+```
+
+| Pros | Cons |
+|------|------|
+| Single instrument, unified management | Larger config directory |
+| Can reduce any/all arms in one command | Requires consistent BAND header across spectrographs |
+| Conceptually cleaner (ANDES is one instrument) | |
+| Per-arm tuning via settings files | |
+
+**Requirements**: All ANDES FITS files must have a `BAND` header (or equivalent) identifying the arm.
+
+### Option B: Per-Spectrograph Instruments (4 instruments)
+
+```
+ANDES_UBV/   channels=[U,B,V]   extension=[1,2,3]
+ANDES_RIZ/   channels=[R,IZ]    extension=[1,2]
+ANDES_YJH/   channels=[Y,J,H]   extension=[1,2,3]
+ANDES_K/     channels=[K]       extension=1
+```
+
+| Pros | Cons |
+|------|------|
+| Smaller, focused config files | 4 instruments to manage |
+| Natural file separation (each spectrograph = separate FITS) | Cannot reduce "all ANDES" in one command |
+| Independent development per spectrograph | |
+| Already working for ANDES_YJH | |
+
+---
+
+## Recommendation
+
+**Option A (single instrument)** is now viable and cleaner, provided FITS headers are consistent across spectrographs.
+
+**Option B (per-spectrograph)** remains a practical alternative if:
+- Different spectrographs have incompatible header conventions
+- Independent development of each spectrograph config is preferred
+- ANDES_YJH is already working and migration isn't worth the effort
+
+### For ANDES_YJH Now
+
+The current setup works. If Y/J/H need different tuning, add per-channel settings:
+```
+ANDES_YJH/
+├── settings.json       # shared base
+├── settings_Y.json     # __inherits__: "ANDES_YJH"
+├── settings_J.json
+└── settings_H.json
+```
+
+### Future Migration to Single Instrument
+
+When other spectrographs (UBV, RIZ, K) are needed:
+1. Verify all use consistent `BAND` header convention
+2. If yes: create single `ANDES/` instrument with all 9 channels
+3. If no: create additional per-spectrograph instruments (ANDES_UBV, etc.)
+
+---
+
+## File Filtering Details
+
+For a single ANDES instrument to work, file classification must correctly select files by arm:
+
+1. **Channel identification**: `kw_channel: BAND` with `id_channel` patterns
+2. **File type identification**: `kw_flat`, `kw_wave`, etc. must work consistently across spectrographs
+
+If spectrographs use different header conventions for file types, separate instruments are cleaner.
+
+---
+
+## Summary
+
+| Approach | Per-arm tuning | File filtering | Complexity |
+|----------|----------------|----------------|------------|
+| Single ANDES | Via settings_*.json | Via kw_channel/id_channel | One instrument |
+| Per-spectrograph | Via settings_*.json | Natural (separate FITS) | Multiple instruments |
+
+Both approaches now support per-arm tuning via per-channel settings files. The choice depends on FITS header consistency and organizational preference.

pyreduce_astro-0.7/CHANGELOG.md

@@ -0,0 +1,252 @@
+# Changelog
+
+
+## [0.7] - 2026-01-25
+
+First stable release of the 0.7 series!
+
+### Changed
+- Rename `OrderTracing` class to `Trace`, save file from `.orders.npz` to `.traces.npz`
+- Store per-trace extraction heights in traces.npz
+- Compute extraction heights for fiber bundles automatically
+- CLI: consistent `-t/--target` option across all commands
+- Allow explicit paths in settings `__inherits__` directive
+
+### Fixed
+- Limit trace overlay to column_range in science step plot
+
+### Documentation
+- Add WhatsNew.md summarizing 0.7 series improvements
+- Update docs for traces.npz rename and mask convention
+
+## [0.7b4] - 2026-01-22
+
+### Added
+- Per-channel settings support via `settings_{channel}.json` files
+- Fiber grouping configuration for multi-fiber instruments (bundle_centers, fiber_groups)
+- `preset_slitfunc` parameter for single-pass extraction using pre-computed slit function
+- `noise_relative` parameter for trace threshold scaling
+- `extraction_reject` threshold parameter for outlier rejection
+- ANDES_YJH instrument with multi-channel support
+- MOSAIC VIS quadrant support (VIS1-VIS4 channels) and NIR settings
+- HARPN fiber B wavecal file
+- Extraction residual panel in ProgressPlot
+- `tools/plot_swath_debug.py` for analyzing extraction debug data
+
+### Changed
+- **Mask convention**: Switch to numpy convention (True/1=bad) on Python side; C code uses inverted (1=good)
+- Extraction convergence now based on spectrum change with global RMS for rejection
+- Improved outlier rejection using 6*sigma instead of MAD in 2D extraction
+- Norm_flat step saves slit function with metadata for reuse
+- Refactor Pipeline trace API: add `trace_raw()` and `organize()` methods
+- Remove `extraction_cutoff` parameter (dead code)
+
+### Fixed
+- `smooth_spectrum` normalization (port from cr2res)
+- Masked array handling in swath debug plotting
+- Minimum 4 iterations enforced in curved extraction
+
+## [0.7b3] - 2026-01-11
+
+### Added
+- MOSAIC instrument support with fiber group detection and curvature step
+- Extraction animation controls: pause/step buttons and speed control (`PYREDUCE_PLOT_ANIMATION_SPEED`)
+- `--plot-dir` and `--plot-show` CLI options for flexible plot output
+
+### Changed
+- Rename `orders` to `traces` in rectify and slit_curve modules for consistency
+- Downgrade extraction max-iterations message from ERROR to WARNING
+- Only warn about missing files for steps that are actually requested
+
+### Fixed
+- Handle channel mismatch gracefully in CLI
+- Curvature plotting index error when peaks need int casting
+
+## [0.7b2] - 2026-01-09
+
+### Changed
+- **Reorganize instrument files**: All instrument-related files now in per-instrument directories
+  - `instruments/{name}.py` → `instruments/{NAME}/__init__.py`
+  - `instruments/{name}.yaml` → `instruments/{NAME}/config.yaml`
+  - `settings/settings_{NAME}.json` → `instruments/{NAME}/settings.json`
+  - `wavecal/{name}_*.npz` → `instruments/{NAME}/wavecal_*.npz`
+  - `masks/mask_{name}_*.fits.gz` → `instruments/{NAME}/mask_*.fits.gz`
+- Base settings and schema moved to `instruments/defaults/`
+- Wavelength atlas files moved to `instruments/defaults/atlas/`
+
+### Removed
+- Orphan mask files for undefined instruments (elodie, sarg, hds, etc.)
+
+## [0.7b1] - 2026-01-06
+
+### Changed
+- **Curvature algorithm rewrite**: Replace 2D model fitting with row-tracking method for better robustness
+- Rename `make_shear.py` to `slit_curve.py`
+- Rename curvature coefficients: `tilt`/`shear` → `p1`/`p2` throughout codebase
+- Rename `arc_extraction` to `simple_extraction`
+- Split curvature `extraction_height` into separate `extraction_height` and `curve_height` parameters
+- Save file renamed from `.shear.npz` to `.curve.npz`
+
+### Added
+- `discover_channels()` for automatic channel detection from data files
+- CLI: `--target` is now optional; loops over all targets if not specified
+- CLI: Uses `$REDUCE_DATA` for base_dir, reads default input_dir from config
+- Comprehensive CLI test coverage
+
+### Fixed
+- Curvature step index error when traces are removed during processing
+- Validate base_dir and input_dir exist with clear error messages
+- File sorting to correctly loop over all nights when not specified
+- CLI dynamic dependency loading for step commands
+
+## [0.7a7] - 2026-01-04
+
+### Added
+- `--settings` option for CLI to override reduction parameters from JSON file
+
+### Changed
+- Rename `extraction_width` to `extraction_height` in settings (clarifies coordinate system)
+- Rename `orders` to `traces` in extract.py internal API
+
+## [0.7a6] - 2026-01-03
+
+### Added
+- `--file` option for CLI step commands to bypass file discovery
+- NEID instrument with multi-amplifier support
+
+### Changed
+- Rename `orders` step to `trace` throughout codebase (CLI, API, configs)
+
+### Fixed
+- test_normflat to use column_range returned by extract
+
+## [0.7a5] - 2025-12-30
+
+### Changed
+- Rename `arm` to `channel` throughout codebase (API, CLI, configs)
+- Remove decker from CRIRES+ channel format (`J1228_det1` instead of `J1228_Open_det1`)
+- Example scripts now use `$REDUCE_DATA` env var instead of hardcoded paths
+
+## [0.7a4] - 2025-12-23
+
+### Added
+- `reduce examples --run` flag to download and execute examples directly
+- PEP 723 inline metadata in examples for `uv run` compatibility
+
+## [0.7a3] - 2025-12-23
+
+### Added
+- `reduce examples` command to list/download examples from GitHub matching installed version
+
+### Changed
+- CLI startup 12x faster via lazy imports (1.2s -> 0.1s)
+
+## [0.7a2] - 2025-12-23
+
+### Added
+- Manual API calls documentation (manual_calls.md)
+
+### Fixed
+- ReadTheDocs build failing due to missing myst_parser
+- Trace module reference in documentation
+
+### Changed
+- Docs Makefile now uses uv
+- pyproject.toml cleanup
+
+## [0.7a1] - 2025-12-22
+
+### Added
+- New Pipeline API with `Pipeline.from_instrument()` for simplified usage
+- Click-based CLI replacing argparse (`uv run reduce run UVES HD132205`)
+- Pydantic models for instrument configuration validation
+- YAML instrument configs replacing JSON
+- IPython startup script for interactive development
+- `plot_dir` option to save plots as PNG files
+- Fiber bundle tracing support for multi-fiber instruments
+- `filter_x` and `filter_type` options for order tracing
+
+### Changed
+- Rename `mode` to `channel` terminology throughout
+- Output extension changed from `.ech` to `.fits`
+- Documentation converted from RST to Markdown
+- Trace detection parameters renamed (`opower` -> `degree`, `filter_size` -> `filter_y`)
+
+### Fixed
+- Plotting issues with non-finite values
+- Use `interpolate_replace_nans` for masked pixels
+
+## [0.6.0] - 2025-12-22
+
+### Added
+- JSON schema validation test for instrument configurations
+- GitHub workflow creates GitHub Release on tag push
+
+### Fixed
+- Fix test_wavecal to match WavelengthCalibration.execute() signature
+- Fix spec fixture to match ScienceExtraction.run() signature
+- Fix instrument schema: replace invalid 'value' keyword with standard JSON Schema 'type'
+
+## [0.6.0b5] - 2025-10-04
+
+### Fixed
+- Include *.pyd files for Windows builds
+- Remove redundant build test from workflow
+
+## [0.6.0b4] - 2025-10-03
+
+### Changed
+- workflow builds multi-arch wheels
+- minor fixes to make build pass
+
+## [0.6.0b3] - 2025-10-02
+
+### Added
+- Test organization with pytest markers (unit, instrument, slow, downloads)
+- ANDES instrument configuration and settings
+
+### Changed
+- CI now runs fast tests (~7s) on every push to master
+- Test suite optimized with slow marker for tests >3s
+
+### Fixed
+- ANDES instrument configuration errors (missing decker fields, regex pattern)
+- NIRSPEC FITS header errors with illegal keywords containing dots
+- Pre-commit hooks now documented and working correctly
+
+## [0.6.0b2] - 2025-10-02
+
+### Changed
+- lazy imports
+- cosmetic fixes like version string
+
+## [0.6.0b1] - 2025-10-02
+
+### Changed
+- Modern build system using uv package manager
+- Minimum Python version now 3.11, default 3.13
+- Migrated build system from setuptools to Hatchling (PEP 517)
+- Replaced black, isort, flake8, and pyupgrade with Ruff
+- Improved code formatting across entire codebase using Ruff
+- Modernized GitHub Actions workflow (uv-based, Python 3.11-3.13 matrix)
+- Updated documentation to reflect modern installation with uv
+- Consolidated legacy build files (setup.py, requirements.txt) into pyproject.toml
+- Removed automatic PyPI publishing from GitHub Actions (now manual)
+- GitHub Actions tests now manual-only on master branch
+- Modernized ReadTheDocs build configuration
+
+### Fixed
+- Close FITS file handles after reading to prevent resource leaks ([#28](https://github.com/ivh/PyReduce/pull/28))
+- Store wavelength calibration in double precision instead of single ([#30](https://github.com/ivh/PyReduce/pull/30))
+- Add fallback value for E_ORIENT header keyword ([#31](https://github.com/ivh/PyReduce/pull/31))
+- Return final wavelength linelist with flags from calibration ([#33](https://github.com/ivh/PyReduce/pull/33))
+- Fix MaskedArray filling in normflat routine
+- Fix undefined TypeFilter import in NEID instrument
+- Fix Sphinx documentation build warnings
+- Fix pre-commit hook configuration issues
+
+### Documentation
+- Updated README badges for GitHub Actions and Python versions
+- Updated installation instructions to use uv instead of pip
+- Improved documentation build process
+- CLAUDE.md / AGENTS.md for AI-assisted development guidance