spectro-kernel 0.1.6__tar.gz → 0.2.1__tar.gz

{spectro_kernel-0.1.6 → spectro_kernel-0.2.1}/CHANGELOG.md

@@ -6,6 +6,176 @@ Until `1.0.0` the public API may change between minor versions.
 
 ## [Unreleased]
 
+## [0.2.1] — 2026-06-09
+
+End-to-end aurora-parity patch. Diagnosed and validated bit-exactly on
+real M42 long-slit data: with these two changes a downstream aurora
+reduction pipeline running entirely on spectro-kernel matches its
+science-core baseline at **Pearson 0.99996** (median |Δ| 0.82 %),
+where v0.2.0 alone scored 0.107 (output silently shifted by +32768 ADU
+on every step).
+
+### Fixed — uint16 ``BZERO``/``BSCALE`` leak in the easyspec staging helpers (BUG)
+
+The shared helper ``stage_target`` in
+[`reduction/_easyspec_apply.py`](src/spectro_kernel/algorithms/reduction/_easyspec_apply.py)
+wrote ``ctx.image.data`` as float64 on a new ``PrimaryHDU`` and copied
+the source header verbatim. When the source frame originated from a
+**uint16 raw file** (``BITPIX=16``, ``BZERO=32768``, ``BSCALE=1`` —
+astropy's convention), the scaling keywords landed on the float64 HDU
+and astropy then re-applied them on read-back, **silently adding 32768
+ADU to every pixel**. The companion ``build_corrected_imageframe``
+re-attached the same scaling block to the corrected ``ImageFrame``, so
+the next pipeline step that re-staged it inherited the leak.
+
+The fix strips a small set of reserved / scaling keywords —
+``SIMPLE, BITPIX, EXTEND, BZERO, BSCALE, BLANK, END, NAXIS*, PCOUNT,
+GCOUNT, XTENSION`` — before they reach either the staged HDU or the
+header re-attached downstream. astropy re-derives the structural keys
+from the data itself; the non-structural metadata (``EXPTIME``,
+``OBSERVER``, etc.) survives.
+
+Affects all four easyspec apply wrappers in one shot:
+``subtract_bias_easyspec``, ``subtract_dark_easyspec``,
+``flat_normalize_easyspec``, ``extract_spectrum_easyspec``. Native-numpy
+algorithms (``dark_subtract``, ``flat_normalize``, ``subtract_sky_2d``,
+the v0.2.0 geometry / denoising suite) were never affected — they
+operate directly on ``ctx.image.data`` without round-tripping through a
+FITS file.
+
+5 regression tests added in
+[`tests/unit/test_easyspec_apply_staging.py`](tests/unit/test_easyspec_apply_staging.py).
+
+### Added — ``extract_spectrum_boxcar`` (literature aperture extraction)
+
+New algorithm in
+[`algorithms/extraction/boxcar.py`](src/spectro_kernel/algorithms/extraction/boxcar.py).
+Same easyspec tracing step as ``extract_spectrum_easyspec``, but
+extraction is a **pure-numpy aperture sum** (clamped to the detector
+bounds on every column).
+
+Why a new brick: ``extract_spectrum_easyspec`` delegates the extraction
+to ``easyspec.extracting``, whose per-column internal sky window
+crashes with ``ValueError: broadcast (W,) vs (S,)`` when the aperture
+half-width is wide (≥ a few hundred px) and the trace sits near a
+detector edge — the typical aurora / extended-source geometry. The new
+brick keeps easyspec's argmax tracing (the trace itself is fine) but
+performs the aperture sum in numpy, with ``max(0, …)`` / ``min(rows,
+…)`` clipping on every column so it is **edge-safe by construction**.
+
+Supports two weightings:
+
+- ``"tophat"`` — unweighted box sum (standard IRAF ``apall``).
+- ``"gaussian"`` — profile-weighted Horne 1986 estimator (the average
+  column profile is fit with ``astropy.modeling`` to derive the
+  weights).
+
+``shift_y_pixels = 0`` disables the background subtraction (pure
+column sum); a positive value samples sky from two symmetric flanking
+windows. References: **Horne 1986, PASP 98, 609** ; **Tody 1986, Proc.
+SPIE 627, 733** (IRAF ``apall`` / ``aptrace``). easyspec is imported
+lazily inside ``run()`` (v0.1.5+ pattern).
+
+6 tests in
+[`tests/unit/test_extract_boxcar.py`](tests/unit/test_extract_boxcar.py),
+including the wide-aperture edge case that motivated the brick.
+
+### Numbers
+
+**98 algorithms** registered (was 97). 254 tests pass (11 new), 3
+skipped. ``ruff check`` clean, ``mkdocs build --strict`` clean,
+``twine check --strict`` clean.
+
+## [0.2.0] — 2026-06-09
+
+A library expansion focused on long-slit reduction (aurora, stellar,
+solar — wherever a 2-D spectrograph frame becomes a 1-D spectrum).
+**13 new algorithms** ported from a downstream aurora-reduction codebase
+into the kernel's standard pattern: same ``BaseAlgorithm`` / ``@register_algorithm``
+shape, every reference cited, every parameter documented. Existing
+behaviour is untouched; this is a pure additive bump.
+
+The minor-version bump (``0.1.x`` → ``0.2.0``) signals the broader scope
+even though no API breaks landed.
+
+### Added — geometric long-slit corrections (3 algorithms)
+
+- ``correct_tilt_affine`` — vertical shear re-aligning the slit with
+  the detector columns (Howell 2006 §5.2 ; Tody 1986 IRAF heritage).
+- ``correct_slant_affine`` — horizontal shear pivoted on the trace row,
+  making monochromatic lines parallel to the rows.
+- ``correct_smile_polynomial`` — closed-form Schroeder 2000 even-order
+  expansion ``dx(dy) = dy²/(2R) + dy⁴/(8R³) + 5·dy⁶/(16R⁵)`` undoing
+  the cylindrical "smile" of concave-grating spectrographs (Bottema 1980).
+
+All three are no-ops when their amplitude parameter is zero, so they
+can be composed unconditionally in a preset.
+
+### Added — 2D image denoising (3 algorithms)
+
+- ``outlier_rejection_mad_adaptive`` — vectorised Hwang & Haddad 1995
+  rank-conditional median filter with MAD as the local dispersion
+  estimator. Suppresses impulsive noise (CMOS RTS, hot pixels) without
+  smearing narrow spectral lines.
+- ``denoise_gaussian_2d`` — 2D companion of the existing
+  ``smooth_gaussian`` (which is 1D only).
+- ``denoise_median_2d`` — square median filter, optionally banded by
+  rows so the science aperture is cleaned without disturbing the sky.
+
+### Added — sky extraction + in-situ wavelength calibration
+
+- ``extract_sky_lateral_bands`` — median-combine two off-trace bands
+  into a 1-D pixel-axis sky reference, stored at
+  ``ctx.extras["sky_spectrum"]``. Builds the input for in-situ
+  calibration when no arc lamp is available.
+- ``fit_emission_lines_gaussian`` — multi-line centroid fitter
+  (Gaussian + constant continuum). Distinct from ``fit_gaussian_line``
+  (single line, linear continuum) — both have their use.
+- ``wavelength_calibration_in_situ`` — Stoughton et al. 2002 recipe:
+  centroid sky lines, derive Δλ vs. the catalogue, resample science on
+  a uniform Å grid and apply the shift.
+- ``wavelength_calibration_solar`` — built-in catalogue of 15 strong
+  Fraunhofer lines (Ca II K/H, Hβ, Mg b triplet, Na D, Hα, O₂ telluric
+  bands), peak detection on the inverted spectrum, polynomial fit.
+  Lets daylight / twilight / solar spectra be calibrated without an arc
+  lamp.
+
+### Added — 1D postprocessing (2 algorithms)
+
+- ``combine_spectra_arithmetic`` — element-wise ``add``/``sub``/``mul``/``div``
+  between ``ctx.spectrum`` and a reference. The reference comes from
+  ``reference_path`` (FITS) or ``ctx.extras["reference_spectrum"]`` —
+  the same idiom ``remove_telluric_division`` uses. IRAF ``sarith``
+  pass-through outside the reference's domain.
+- ``normalize_to_region`` — divide the flux by its NaN-safe mean over
+  ``[wave_lo, wave_hi]``. Sits beside the existing ``normalize_edges`` /
+  ``normalize_polynomial`` / ``normalize_max`` / ``normalize_percentile``
+  family with the explicit "normalise to *this* band" semantic.
+
+### Added — BeSS / ARAS-compliant FITS exporter
+
+- ``export_fits_bess`` — writes ``ctx.spectrum`` (linear Å grid) with
+  the full BeSS observation-header convention (Teyssier 2015 §3.2 ;
+  Buil 2012 ARAS Observation Guide): OBJNAME, BSS_INST, BSS_SITE,
+  OBSERVER, DATE-OBS, MJD-OBS, JD-OBS (computed offline via Meeus 1991
+  §7), EXPTIME, BSS_VHEL, BSS_ESRC, plus the ``-32000`` masked-pixel
+  sentinel. Generates the canonical filename
+  ``_<object>_<date>_<time>_<observer>.fits`` automatically.
+
+### Improved — catalogue page
+
+``docs/gen_catalogue.py`` now has friendly titles for every category
+the kernel registers — the categories that joined the catalogue with
+v0.1 (preprocessing, extraction, wavelength_calibration, embedding, …)
+no longer fall back to the raw ``snake_case`` rendering.
+
+### Numbers
+
+**97 algorithms** are registered after this release, up from 84 in
+v0.1.6. 243 tests pass (47 new). ``ruff check`` is clean, ``mkdocs
+build --strict`` is clean. Boot-time discovery stays at ~1.9 s thanks
+to v0.1.5/v0.1.6's lazy-import work.
+
 ## [0.1.6] — 2026-06-09
 
 A hygiene pass on top of v0.1.5: same algorithms, same numerics, less

{spectro_kernel-0.1.6 → spectro_kernel-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spectro-kernel
-Version: 0.1.6
+Version: 0.2.1
 Summary: A shared catalogue of astronomical spectroscopy algorithms, composable into reproducible pipelines, usable as a Python library, a CLI, or an MCP server.
 Project-URL: Homepage, https://github.com/matthieulel/spectro-kernel
 Project-URL: Documentation, https://matthieulel.github.io/spectro-kernel/

{spectro_kernel-0.1.6 → spectro_kernel-0.2.1}/docs/gen_catalogue.py

@@ -12,6 +12,12 @@ from spectro_kernel.registry import list_algorithms
 
 _CATEGORY_TITLES = {
     "io": "Input / output",
+    "master_creation": "Master frames",
+    "preprocessing": "Preprocessing (2D image)",
+    "cosmic_ray": "Cosmic-ray rejection",
+    "extraction": "Extraction (2D → 1D)",
+    "wavelength_calibration": "Wavelength calibration",
+    "flux_calibration": "Flux calibration",
     "continuum": "Continuum",
     "smoothing": "Smoothing",
     "resampling": "Resampling",
@@ -23,9 +29,11 @@ _CATEGORY_TITLES = {
     "radial_velocity": "Radial velocity",
     "timeseries": "Time series",
     "stacking": "Stacking",
+    "embedding": "Embeddings",
     "catalog": "External catalogues",
     "visualization": "Visualisation",
     "export": "Export",
+    "advanced": "Advanced",
 }
 
 _BACKEND_NOTE = {

{spectro_kernel-0.1.6 → spectro_kernel-0.2.1}/docs/index.md

@@ -58,6 +58,12 @@ through MCP, you can do from a terminal, and vice-versa.
 
 ## Status
 
-`v0.1.0` — alpha. The API may change until `v1.0.0`. See the
+`v0.2.0` — alpha. The catalogue covers ~97 algorithms across image-frame
+reduction (bias / dark / flat, geometric rectification, denoising,
+cosmic-ray clipping, sky subtraction, trace extraction), wavelength &
+flux calibration (arc-lamp polynomial, in-situ from sky lines, solar
+Fraunhofer), 1-D analysis (continuum, smoothing, line fitting, radial
+velocity, time series), embeddings, exports (FITS / CSV / VOTable / HDF5
+/ BeSS), and visualisation. The API may change until `v1.0.0`. See the
 [changelog](https://github.com/matthieulel/spectro-kernel/blob/main/CHANGELOG.md)
 for release notes.

spectro_kernel-0.2.1/docs/tutorials/long-slit-reduction.md

@@ -0,0 +1,171 @@
+# Tutorial: long-slit reduction (aurora, stellar, solar)
+
+Take a raw 2-D detector frame and walk it all the way to a flux-calibrated
+1-D spectrum on a uniform Å grid, ready for line fitting or BeSS submission.
+This tutorial shows how the **13 long-slit bricks added in v0.2.0** compose
+into a complete reduction. The example uses an aurora-style setup (sky
+emission lines as wavelength reference), but the same recipe works for
+stellar long-slit, solar twilight, or any small-spectrograph field setup.
+
+> If you only need 1-D analysis on an already-reduced spectrum, see
+> [Analyse a spectrum](analyse-a-spectrum.md) instead.
+
+## Pipeline overview
+
+The new bricks form three groups, with one entry point each:
+
+| Group | Bricks | When to use |
+|---|---|---|
+| **Geometry** | `correct_tilt_affine`, `correct_slant_affine`, `correct_smile_polynomial` | Once per instrument: align slit with columns, rectify monochromatic lines, undo cylindrical smile of concave gratings. |
+| **2-D denoising** | `outlier_rejection_mad_adaptive`, `denoise_gaussian_2d`, `denoise_median_2d` | After cosmic-ray clipping, before extraction. CMOS RTS / hot pixels. |
+| **In-situ λ** | `extract_sky_lateral_bands` + `fit_emission_lines_gaussian` + `wavelength_calibration_in_situ` (or `wavelength_calibration_solar`) | When no arc lamp is available — field aurora monitors, twilight, daytime solar. |
+
+Two postprocessing helpers and the BeSS exporter round out the set:
+
+| Group | Bricks |
+|---|---|
+| **Postprocessing 1-D** | `combine_spectra_arithmetic` (response/std-star division), `normalize_to_region` (normalise to a chosen continuum band) |
+| **Export** | `export_fits_bess` (BeSS / ARAS-compliant FITS with full header, JD/MJD, canonical filename) |
+
+## A complete preset
+
+The kernel's preset format strings these bricks together. The example
+below is the field-aurora workflow; substitute parameters for stellar or
+solar setups.
+
+```yaml
+# presets/aurora_long_slit.yaml
+name: aurora_long_slit
+description: |
+  Long-slit aurora reduction: geometry → denoising → extraction →
+  in-situ wavelength calibration → continuum normalisation.
+steps:
+  # ── 1. Geometric rectification (instrument-specific, one-shot values).
+  - algorithm: correct_tilt_affine
+    params: {tilt_deg: 0.4}
+  - algorithm: correct_slant_affine
+    params: {slant_deg: 0.6, pivot_row: 120}
+  - algorithm: correct_smile_polynomial
+    params: {reference_row: 120, smile_radius: 1500.0, polynomial_order: 4}
+
+  # ── 2. Per-frame denoising (skip the science band when running median).
+  - algorithm: outlier_rejection_mad_adaptive
+    params: {kernel_size: 3, threshold: 0.6}
+  - algorithm: denoise_median_2d
+    params: {kernel_size: 3, row_lo: 0, row_hi: 100}      # sky strip only
+
+  # ── 3. Sky-aware extraction.
+  - algorithm: extract_sky_lateral_bands
+    params:
+      band_above_lo: 80
+      band_above_hi: 110
+      band_below_lo: 130
+      band_below_hi: 160
+  - algorithm: subtract_sky_2d
+    params: {trace_row: 120, trace_half_width: 6, sky_offset: 4, sky_half_width: 10}
+  # extract_spectrum_boxcar is the edge-safe boxcar for extended sources
+  # (aurora, nebulae). For a stellar point source, extract_spectrum_sum
+  # or extract_spectrum_easyspec are also valid choices.
+  - algorithm: extract_spectrum_boxcar
+    params: {trace_method: "argmax", trace_half_width: 6, extraction_weights: "tophat", shift_y_pixels: 0}
+
+  # ── 4. In-situ wavelength calibration from the simultaneously-acquired
+  #     sky reference (no arc lamp needed).
+  - algorithm: wavelength_calibration_in_situ
+    params:
+      polynomial_coef: [1.0, 4000.0]          # initial λ(x) ≈ 4000 + x
+      reference_wavelengths: [5577.34, 6300.30, 6363.78]
+      guess_positions: [1577, 2300, 2363]
+      search_width: 40.0
+      oversampling: 2.0
+
+  # ── 5. Continuum normalisation to a known line-free band.
+  - algorithm: normalize_to_region
+    params: {wave_lo: 6400.0, wave_hi: 6500.0}
+
+  # ── 6. BeSS-compliant FITS for sharing.
+  - algorithm: export_fits_bess
+    params:
+      object_name: "aurora_2026-06-09"
+      instrument: "Alpy-600"
+      site: "Skibotn"
+      observer: "field-monitor"
+      date_obs_utc: "2026-06-09T22:13:45"
+      exposure_seconds: 30.0
+      vhelio_kms: 0.0
+```
+
+Save and run with the CLI:
+
+```sh
+spectro run aurora_long_slit.yaml --image my_raw_frame.fits
+```
+
+… or in Python:
+
+```python
+from spectro_kernel import WorkContext, run_preset
+from spectro_kernel.io import read_fits_image  # 2D variant
+
+ctx = WorkContext(image=read_fits_image("my_raw_frame.fits"))
+result = run_preset("aurora_long_slit", ctx)
+print(result.history)         # full audit trail
+bess_blob = ctx.exports["fits_bess"]
+```
+
+## Variants
+
+### Stellar long-slit (arc-lamp calibration)
+
+Replace step 4 with `wavelength_calibrate_polynomial` (existing brick)
+on identifications from your arc-lamp frame. Steps 1–3, 5 and 6 stay
+as-is.
+
+### Solar / twilight (no sky lines, no arc lamp)
+
+Use `wavelength_calibration_solar` in place of step 3+4. It needs only
+two instrument-specific values (an approximate ``λ_min`` and a
+dispersion guess in Å/pixel) and the built-in Fraunhofer catalogue:
+
+```yaml
+- algorithm: wavelength_calibration_solar
+  params:
+    approx_wavelength_min_angstrom: 3800.0
+    approx_dispersion_angstrom_per_pixel: 1.0
+    poly_order: 3
+    min_lines_for_fit: 6
+```
+
+### Response correction with a standard star
+
+After step 5 (or in place of it), divide by the instrumental response:
+
+```yaml
+- algorithm: combine_spectra_arithmetic
+  params:
+    operation: div
+    reference_path: "/path/to/response.fits"
+    min_denominator: 1.0e-5
+```
+
+The reference can also live in `ctx.extras["reference_spectrum"]` if
+your pipeline derives it on the fly — same idiom as the existing
+`remove_telluric_division`.
+
+## What the bricks share
+
+Every algorithm above follows the same kernel contract:
+
+- **No new context fields**: 2-D frames go through `ctx.image`, the
+  science spectrum through `ctx.spectrum`, second spectra (sky, response,
+  standard) through `ctx.extras[...]`.
+- **Literature references** on every class (`references = […]`).
+- **No-op when zero**: geometric corrections and the Gaussian denoiser
+  pass through unchanged when their amplitude parameter is zero, so a
+  single preset can run on instruments with and without those distortions.
+- **Provenance**: every step writes a fingerprint of what it did into
+  `ctx.image.meta` (for 2-D) or `ctx.spectrum.meta` (for 1-D), and the
+  per-step `metrics` are JSON-serialisable for later auditing.
+
+See [Algorithm catalogue](../catalogue.md) for the full parameter and
+reference list of each brick.

spectro_kernel-0.2.1/src/spectro_kernel/algorithms/continuum/normalize_region.py

@@ -0,0 +1,97 @@
+"""Algorithm: normalise a spectrum by its mean flux over a chosen wavelength window."""
+
+from __future__ import annotations
+
+from typing import Any
+
+import numpy as np
+
+from ...base import AlgorithmOutput, BaseAlgorithm
+from ...errors import InvalidParameterError
+from ...registry import register_algorithm
+from ...types import AlgorithmCategory, WorkContext
+
+
+@register_algorithm(
+    "normalize_to_region",
+    category=AlgorithmCategory.CONTINUUM,
+    version="1.0.0",
+)
+class NormalizeToRegion(BaseAlgorithm):
+    """Divide the flux by its NaN-safe mean over ``[wave_lo, wave_hi]``.
+
+    The window should be a line-free continuum region so the output sits
+    near unity inside it. The companion of ``normalize_edges`` /
+    ``normalize_polynomial`` / ``normalize_max`` /
+    ``normalize_percentile`` with the explicit "normalise to *this*
+    band" convention — exactly what the IRAF ``continuum`` task does
+    when scaled by a measured band mean.
+    """
+
+    backend = "numpy"
+    references = [
+        "Tody 1986, Proc. SPIE 627, 733 — IRAF continuum heritage.",
+    ]
+    long_description = (
+        "out = flux / nanmean(flux[wave_lo:wave_hi]). The window is "
+        "closed at both ends and clamped to the spectrum's wavelength "
+        "range. Uncertainty (if present) is scaled by |region_mean|. "
+        "If the region mean is zero or non-finite the algorithm fails "
+        "with a clear message rather than emitting NaN."
+    )
+    default_params = {"wave_lo": 0.0, "wave_hi": 0.0}
+    required_params = ["wave_lo", "wave_hi"]
+    param_descriptions = {
+        "wave_lo": "Lower bound of the reference window (same unit as the wavelength axis).",
+        "wave_hi": "Upper bound of the reference window.",
+    }
+    input_requirements = ["spectrum"]
+    output_produces = ["spectrum", "metrics.region_mean"]
+
+    def run(self, ctx: WorkContext, params: dict[str, Any]) -> AlgorithmOutput:
+        wave_lo = float(params["wave_lo"])
+        wave_hi = float(params["wave_hi"])
+        if wave_hi <= wave_lo:
+            raise InvalidParameterError("wave_hi must be strictly greater than wave_lo.")
+        spec = ctx.spectrum
+        if spec is None:
+            return AlgorithmOutput.fail("No ctx.spectrum to normalise.")
+
+        wave = np.asarray(spec.wavelength, dtype=np.float64)
+        flux = np.asarray(spec.flux, dtype=np.float64)
+        # Nearest-sample window so users don't need to know the grid step.
+        lo_idx = int(np.argmin(np.abs(wave - wave_lo)))
+        hi_idx = int(np.argmin(np.abs(wave - wave_hi)))
+        lo_idx, hi_idx = sorted((lo_idx, hi_idx))
+        if hi_idx == lo_idx:
+            return AlgorithmOutput.fail(
+                f"Reference window [{wave_lo}, {wave_hi}] is narrower than "
+                "one sample of the spectrum's wavelength grid."
+            )
+
+        region_mean = float(np.nanmean(flux[lo_idx : hi_idx + 1]))
+        if not np.isfinite(region_mean) or region_mean == 0.0:
+            return AlgorithmOutput.fail(
+                f"Reference region mean is {region_mean} — pick a window "
+                "with finite, non-zero flux."
+            )
+
+        out = spec.with_flux(flux / region_mean, flux_unit="normalized")
+        if spec.uncertainty is not None:
+            out.uncertainty = spec.uncertainty / abs(region_mean)
+        out.meta["normalized_to_region"] = (wave_lo, wave_hi)
+        out.meta["normalization_factor"] = region_mean
+        ctx.spectrum = out
+
+        return AlgorithmOutput.ok(
+            metrics={
+                "region_mean": region_mean,
+                "wave_lo": wave_lo,
+                "wave_hi": wave_hi,
+                "n_samples": float(hi_idx - lo_idx + 1),
+            },
+            message=(
+                f"Normalised to mean {region_mean:.4g} over "
+                f"[{wave_lo:g}, {wave_hi:g}] ({hi_idx - lo_idx + 1} samples)."
+            ),
+        )