stdpipe 0.4.1__tar.gz → 0.5.0__tar.gz

stdpipe-0.5.0/CHANGELOG.md

@@ -0,0 +1,102 @@
+# Changelog
+
+All notable changes to STDPipe are documented in this file. Changes in
+releases up to 0.4.0 are only available in the
+[commit history](https://github.com/karpov-sv/stdpipe/commits/master).
+
+## 0.5.0 - 2026-10-31
+
+Also published as 0.4.2, which was yanked.
+
+### Changed
+
+- PSF fitters (`photometry_psf.measure_objects_psf()`,
+  `photometry_measure.measure_objects_sep()` and
+  `photometry_iraf.measure_objects_psf()`) now store fitted positions in
+  `x`/`y` and keep the input ones in `x_orig`/`y_orig`, as centroiding does.
+  The `x_psf`/`y_psf` columns are gone; `get_objects_sextractor()` still
+  reports SExtractor's own `x_psf`/`y_psf`.
+- `psf.create_psf_model()` no longer uses photutils `EPSFBuilder`. The new
+  builder places star pixels at their true sub-pixel phases, iteratively
+  recentres and renormalizes the model, rejects outliers per pixel, subtracts
+  neighbours with the model itself, and sizes the stamp automatically from the
+  stacked stellar profile. This removes a ~0.8% PSF flux bias and 2-6% flux
+  losses for Moffat-like PSFs, and is several times faster. Resulting models
+  differ slightly from previous versions.
+- `measure_objects_psf()` default `maxiters` raised from 3 to 100 (it is a
+  limit on function evaluations, and fits did not converge before), and
+  non-convergence is now detected using the correct photutils flag bit.
+- `measure_objects_sep()` now uses a 5-sigma clipped mean for the local
+  background by default, for both aperture photometry (was unclipped median)
+  and optimal extraction (was 3-sigma clipped mean).
+- Legacy Survey templates are taken from DR11 instead of DR10. Previously
+  cached DR10 bricks are reused as is, as their filenames do not encode the
+  data release.
+- `resolve.simbadResolve()` queries Simbad only (instead of Simbad, NED and
+  VizieR), using the current Sesame URL.
+- Requires `sep-x>=1.5.2`, which correctly resamples point-sampled
+  supersampled PSF models (PSFEx, `create_psf_model()`).
+
+### Added
+
+- `create_psf_model(degree='auto')` selects the spatial polynomial degree of
+  the PSF model by spatial cross-validation, up to `max_degree`.
+- `order='auto'` for SIP order selection in `refine_wcs_quadhash()` and
+  `refine_wcs_simple()`, as already supported by `refine_wcs_scamp()`.
+- `bg_estimator` (`'mode'`, `'median'`, `'mean'`) with `clip_sigma`/`clip_iters`
+  for the local background in `measure_objects()` and `measure_objects_sep()`.
+- `filter_type='matched'` in `get_objects_sextractor()`, so that the detection
+  threshold is in units of the filtered noise, as in `sep.extract()`.
+- `get_objects_sep()` reports background flux in the aperture (`bg_flux`,
+  `bg_fluxerr`), and per-source `fwhm_model` and `aper` columns with a
+  position-dependent FWHM model. `measure_objects_sep()` also reports
+  `bg_fluxerr`.
+- VizieR mirror fallback in `get_cat_vizier()` (`catalogs.vizier_mirrors`,
+  used on network errors only) and a `timeout` argument.
+- Sesame mirror fallback in `resolve` (`resolve.sesame_mirrors`), and
+  `resolve()` falls through to the next method when Simbad or TNS lookups
+  fail.
+- `cache` option in `templates.get_hips_image()`.
+- `utils.file_lock()` and inter-process locking of skycell cache downloads in
+  `templates.get_skycells()`.
+- `plots.render_lock` serializing Matplotlib rendering in `figure_saver`
+  across threads.
+- Support for photutils 3.0.
+
+### Fixed
+
+- Local background in `measure_objects()` was truncated to integers.
+- Background aperture noise in `get_objects_sep()` counted the sky Poisson
+  noise twice.
+- Masked object positions were measured at the values hidden under the mask.
+- PSF fits of stars with masked pixels nearby were discarded as failed.
+- PSF sub-pixel phase errors: centre of even-sized PSFEx models, conversion
+  of sub-pixel integrated models in the photutils backend, and resampling of
+  point-sampled models in the SEP backend.
+- `get_psf_stamp()` lost >5% of the peak of even-sized point-sampled models.
+- Robust `photometry_model.match()` with statsmodels 0.15, and huge cancelling
+  coefficients for rank-deficient designs; errors of terms not constrained by
+  the data are now NaN.
+- Catalogue band augmentation: Gaia DR2 produced no Johnson-Cousins
+  magnitudes since astroquery 0.4.8, Gaia EDR3 was treated as DR2, and
+  `augment_bands=False` was ignored in `get_cat_vizier()`.
+- Legacy Survey bricks present in both northern and southern lists were
+  always taken from the south, and bricks lacking the requested band produced
+  failed downloads.
+- Remote FITS downloads no longer leave temporary files or fill the astropy
+  download cache.
+- Three-color images in `plots.imshow()` are now properly normalized.
+- `refine_wcs_simple(method='astrometrynet')` passed unmatched objects to
+  `fit-wcs`.
+
+## 0.4.1 - 2026-08-07
+
+### Fixed
+
+- NumPy 2 incompatibilities in flag handling and set membership.
+- A perfect photometric fit in `photometry_model.match()` is treated as
+  success rather than a degenerate model.
+
+## 0.4.0 - 2026-08-02
+
+First release published through the automated PyPI release workflow.

{stdpipe-0.4.1 → stdpipe-0.5.0}/MANIFEST.in

@@ -1,3 +1,4 @@
 include README.md
+include CHANGELOG.md
 recursive-include data *
 recursive-exclude notebooks *

{stdpipe-0.4.1/stdpipe.egg-info → stdpipe-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: stdpipe
-Version: 0.4.1
+Version: 0.5.0
 Summary: Simple Transient Detection Pipeline
 Author-email: Sergey Karpov <karpov.sv@gmail.com>
 Maintainer-email: Sergey Karpov <karpov.sv@gmail.com>
@@ -30,7 +30,7 @@ Project-URL: Homepage, https://github.com/karpov-sv/stdpipe
 Project-URL: Documentation, https://stdpipe.readthedocs.io/
 Project-URL: Repository, https://github.com/karpov-sv/stdpipe
 Project-URL: Bug Tracker, https://github.com/karpov-sv/stdpipe/issues
-Project-URL: Changelog, https://github.com/karpov-sv/stdpipe/commits/master
+Project-URL: Changelog, https://github.com/karpov-sv/stdpipe/blob/master/CHANGELOG.md
 Keywords: astronomy,astrophysics,transients,photometry,astrometry,image-processing,pipeline
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Science/Research
@@ -53,7 +53,7 @@ Requires-Dist: scipy
 Requires-Dist: astropy>=4.1
 Requires-Dist: matplotlib
 Requires-Dist: astroquery>0.4.1
-Requires-Dist: sep-x
+Requires-Dist: sep-x>=1.5.2
 Requires-Dist: astroscrappy
 Requires-Dist: photutils>=2.2
 Requires-Dist: scikit-image>=0.22.0

{stdpipe-0.4.1 → stdpipe-0.5.0}/pyproject.toml

@@ -47,7 +47,7 @@ dependencies = [
     "astropy>=4.1",
     "matplotlib",
     "astroquery>0.4.1",
-    "sep-x",
+    "sep-x>=1.5.2",
     "astroscrappy",
     "photutils>=2.2",
     "scikit-image>=0.22.0",  # Requires >=0.22.0 for NumPy 2.0 compatibility
@@ -103,7 +103,7 @@ Homepage = "https://github.com/karpov-sv/stdpipe"
 Documentation = "https://stdpipe.readthedocs.io/"
 Repository = "https://github.com/karpov-sv/stdpipe"
 "Bug Tracker" = "https://github.com/karpov-sv/stdpipe/issues"
-Changelog = "https://github.com/karpov-sv/stdpipe/commits/master"
+Changelog = "https://github.com/karpov-sv/stdpipe/blob/master/CHANGELOG.md"
 
 # ============================================================================
 # Tool configurations

{stdpipe-0.4.1 → stdpipe-0.5.0}/stdpipe/__init__.py

@@ -6,4 +6,4 @@ detection related tasks, intended for quick and easy implementation of custom
 pipelines, as well as for interactive data analysis.
 """
 
-__version__ = "0.4.1"
+__version__ = "0.5.0"

{stdpipe-0.4.1 → stdpipe-0.5.0}/stdpipe/astrometry.py

@@ -594,6 +594,8 @@ def refine_wcs_simple(
     cat_col_ra='RAJ2000',
     cat_col_dec='DEJ2000',
     method='astropy',
+    auto_max_order=5,
+    auto_threshold=0.05,
     _tmpdir=None,
     verbose=False,
 ):
@@ -605,8 +607,12 @@ def refine_wcs_simple(
         List of detected objects with at least ``x``, ``y``, ``ra``, ``dec`` columns.
     cat : astropy.table.Table
         Reference astrometric catalogue.
-    order : int, optional
-        SIP polynomial order for distortion solution.
+    order : int or str, optional
+        SIP polynomial order for distortion solution, or ``'auto'`` to select it
+        based on hold-out residuals: the matched pairs are split in two halves,
+        every order is fitted on one and its residuals measured on the other,
+        and the lowest order that is within ``auto_threshold`` from the best one
+        is used.
     match : bool, optional
         If True, perform nearest-neighbor matching between objects and catalogue.
         If False, assume they are already matched line by line.
@@ -620,6 +626,11 @@ def refine_wcs_simple(
         Catalogue column name for Declination.
     method : str, optional
         Fitting method. Either ``'astropy'`` or ``'astrometrynet'``.
+    auto_max_order : int, optional
+        Largest SIP order to try when ``order='auto'``. Default is 5.
+    auto_threshold : float, optional
+        Minimal relative improvement of hold-out residuals for accepting a
+        higher SIP order when ``order='auto'``. Default is 0.05, i.e. 5 percent.
     _tmpdir : str, optional
         If specified, temporary files will be created inside this path.
     verbose : bool or callable, optional
@@ -634,6 +645,9 @@ def refine_wcs_simple(
     # Simple wrapper around print for logging in verbose mode only
     log = (verbose if callable(verbose) else print) if verbose else lambda *args, **kwargs: None
 
+    if isinstance(order, str) and order != 'auto':
+        raise ValueError("order must be an integer or 'auto', not '%s'" % order)
+
     if match:
         # Perform simple nearest-neighbor matching within given radius
         oidx, cidx, dist = spherical_match(
@@ -646,34 +660,39 @@ def refine_wcs_simple(
         _obj = obj
         _cat = cat
 
-    wcs = None
-
-    if method == 'astropy':
-        wcs = fit_wcs_from_points(
-            [_obj['x'], _obj['y']],
-            SkyCoord(_cat[cat_col_ra], _cat[cat_col_dec], unit='deg'),
-            sip_degree=order,
-        )
-    elif method == 'astrometrynet':
-        binname = None
-
-        # Rough estimate of frame dimensions
-        width = np.max(obj['x'])
-        height = np.max(obj['y'])
-
+    binname = None
+    if method == 'astrometrynet':
         for path in ['.', '/usr/local', '/opt/local']:
             if os.path.isfile(os.path.join(path, 'astrometry', 'bin', 'fit-wcs')):
                 binname = os.path.join(path, 'astrometry', 'bin', 'fit-wcs')
                 break
 
-        if binname:
+        if not binname:
+            log("Astrometry.Net fit-wcs binary not found")
+
+    def fit(order, _obj, _cat, quiet=False):
+        """Fit WCS of a given SIP order to matched pairs of objects and catalogue stars"""
+        flog = (lambda *args, **kwargs: None) if quiet else log
+        wcs = None
+
+        if method == 'astropy':
+            wcs = fit_wcs_from_points(
+                [_obj['x'], _obj['y']],
+                SkyCoord(_cat[cat_col_ra], _cat[cat_col_dec], unit='deg'),
+                sip_degree=int(order),
+            )
+        elif method == 'astrometrynet' and binname:
+            # Rough estimate of frame dimensions
+            width = np.max(obj['x'])
+            height = np.max(obj['y'])
+
             dirname = tempfile.mkdtemp(prefix='astrometry', dir=_tmpdir)
 
             columns = [
-                fits.Column(name='FIELD_X', format='1D', array=obj['x'] + 1),
-                fits.Column(name='FIELD_Y', format='1D', array=obj['y'] + 1),
-                fits.Column(name='INDEX_RA', format='1D', array=cat[cat_col_ra]),
-                fits.Column(name='INDEX_DEC', format='1D', array=cat[cat_col_dec]),
+                fits.Column(name='FIELD_X', format='1D', array=_obj['x'] + 1),
+                fits.Column(name='FIELD_Y', format='1D', array=_obj['y'] + 1),
+                fits.Column(name='INDEX_RA', format='1D', array=_cat[cat_col_ra]),
+                fits.Column(name='INDEX_DEC', format='1D', array=_cat[cat_col_dec]),
             ]
             tbhdu = fits.BinTableHDU.from_columns(columns)
             filename = os.path.join(dirname, 'list.fits')
@@ -690,22 +709,79 @@ def refine_wcs_simple(
                 order,
             )
 
-            log('Running Astrometry.Net WCS fitter like that:')
-            log(command)
+            flog('Running Astrometry.Net WCS fitter like that:')
+            flog(command)
 
             os.system(command)
 
             if os.path.isfile(wcsname):
                 header = fits.getheader(wcsname)
                 wcs = WCS(header)
-                log('WCS fitter run succeeded')
+                flog('WCS fitter run succeeded')
             else:
-                log('WCS fitter run failed')
+                flog('WCS fitter run failed')
 
             shutil.rmtree(dirname)
 
-        else:
-            log("Astrometry.Net fit-wcs binary not found")
+        return wcs
+
+    def select_order():
+        """Choose the SIP order by fitting one half of the matched pairs and
+        measuring the residuals on the other, unused one. Unlike the residuals
+        of the fit itself, these get worse as soon as it starts to overfit, so
+        we may just pick the lowest order that reaches the best of them. All
+        the orders are checked, as the improvement is not monotonic. Order 1
+        is skipped, as SIP only has terms of second order and above."""
+        log('Selecting the SIP order using half of the matches as a test set')
+
+        obj_fit, cat_fit = _obj[::2], _cat[::2]
+        obj_test, cat_test = _obj[1::2], _cat[1::2]
+
+        results = {}
+
+        for o in [0] + list(range(2, int(auto_max_order) + 1)):
+            # Linear part (CRPIX + CD) plus SIP terms with 2 <= i + j <= order,
+            # for both axes. Require at least 3 pairs per free parameter
+            n_sip = (o + 1) * (o + 2) // 2 - 3 if o >= 2 else 0
+            if len(obj_fit) < 3 * (6 + 2 * n_sip):
+                log('  order %d: %d matches, less than %d needed' % (o, len(obj_fit), 3 * (6 + 2 * n_sip)))
+                break
+
+            try:
+                w = fit(o, obj_fit, cat_fit, quiet=True)
+            except Exception:
+                w = None
+            if w is None:
+                break
+
+            ra, dec = w.all_pix2world(obj_test['x'], obj_test['y'], 0)
+            # Median is robust to the outliers among the test pairs
+            res = 3600 * np.nanmedian(
+                spherical_distance(ra, dec, cat_test[cat_col_ra], cat_test[cat_col_dec])
+            )
+            if not np.isfinite(res):
+                break
+
+            results[o] = res
+
+            log('  order %d: test residual %.3f arcsec' % (o, res))
+
+        if not results:
+            log('Order selection failed, using order 0')
+            return 0
+
+        # Lowest order that is within the threshold from the best residual
+        best_res = min(results.values())
+        best_order = min(o for o, res in results.items() if res <= best_res * (1 + auto_threshold))
+
+        log('Using SIP order %d' % best_order)
+
+        return best_order
+
+    if order == 'auto':
+        order = select_order()
+
+    wcs = fit(order, _obj, _cat)
 
     if wcs:
         if update: