pygspice 2.0.0__tar.gz

pygspice-2.0.0/.gitignore

@@ -0,0 +1,15 @@
+dist
+.vscode
+# The npm packages and cache of the site, and the API pages of docs/scripts/api.py
+docs/node_modules/
+docs/.astro/
+docs/src/content/docs/reference/
+# The 2021 draft of the paper, kept out of the repository
+paper/
+# Local environments and caches
+.venv/
+.pytest_cache/
+.coverage
+__pycache__/
+.ruff_cache/
+.hypothesis/

pygspice-2.0.0/CHANGELOG.md

@@ -0,0 +1,97 @@
+# Changelog
+
+## 2.0.0 - 2026-10-29
+
+### Added
+
+- Golden tests in `tests/golden/`: the outputs of every public function of the 2022 package (commit `1fbab9f`) on synthetic inputs, and tests that compare an implementation with them.
+- Continuous integration on Linux with Python 3.11 to 3.15, on macOS and Windows with Python 3.11 and 3.15, and on the oldest supported numpy and scipy.
+- A CI job that builds the wheel and the sdist, checks that the sdist holds the sources and tests and that the wheel holds `py.typed` and no tests, and runs a smoke test of the wheel in a clean environment.
+- The package is on PyPI as `pygspice`, because PyPI does not allow the name `gspice`: `pip install pygspice`. The import name is `gspice`. Before you install it, run `pip uninstall gspice` if you installed GSPICE from GitHub, because both packages write the same files.
+- A workflow that, for each tag `vX.Y.Z`, runs CI, uploads the wheel and the sdist to PyPI once the lead approves, and publishes a GitHub release with them and the CHANGELOG section of X.Y.Z. It checks that the tag names the version of the built package.
+- `pyproject.toml`: `pip install .` installs the package. It requires Python 3.11 or later, numpy 1.26 or later and scipy 1.11 or later.
+- `gspice.__version__`, which is `2.0.0`.
+- `CITATION.cff`, the citation metadata for the software and the paper, which the sdist holds.
+- The licence is declared as GPL-3.0-or-later, the GPL v3 of the 2022 code in its "or later" form.
+- Type checks with mypy in strict mode for the code outside `gspice.legacy`.
+- A test coverage floor of 90% for the code outside `gspice.legacy`.
+- pre-commit hooks that run ruff.
+- Log records on the `gspice` logger: DEBUG lines on the time and size of the work of the main functions, INFO lines on the clipping passes of `fit`, and a WARNING from `fit` for each clipping pass whose threshold exceeds the in-sample ceiling of |z|. The package adds a `logging.NullHandler` to the logger, so the application decides where the records go.
+- A documentation site in `docs/`, built with Sphinx and deployed to GitHub Pages, with a quickstart, tutorials on fitting, pixelwise prediction, infilling, repair, calibration, the low-rank and noise-aware models, saving and loading, and migrating from the 2022 interface, background pages on the method, prediction, repair, fitting and diagnostics, a glossary, an API reference of every public name, and the GSPICE logo. The tutorials work on simulated LAMOST-like spectra from `docs/scripts/spectra.py`, and `docs/scripts/figures.py` makes their figures.
+- `benchmarks/run.py`, which times `pixelwise`, `infill`, `fit_blocks`, `repair`, `fit`, the noise models, the 2022 `gp_interp` and the 2022 Schur route against `solve_pattern`, and prints the timings and the setup as JSON.
+- `gspice.scale`, `gspice.unscale` and `gspice.snr`. `scale` interpolates each spectrum over its bad pixels and scales it to unit mean noise, `unscale` returns scaled values and variances to flux units, and `snr` gives the median signal-to-noise ratio of each spectrum.
+- `gspice.Model`, which holds the mean, the covariance and the training count of each S/N bin, and `gspice.save` and `gspice.load`, which write and read a model as one .npz file.
+- `gspice.pixelwise`, which predicts each pixel of each spectrum from the pixels farther than `nguard` from it, and its result `gspice.Prediction`. A model keeps the precision matrix and the guard weights of each bin after their first use.
+- `gspice.simulate`, which draws synthetic continuum-normalized spectra for tests and examples, and its result `gspice.Simulation`. The spectra have absorption lines whose depths follow two latent parameters in two populations, Gaussian noise, and optional cosmic rays, bad columns and shifted blocks. With `gaussian=True` and no artifacts they are exactly Gaussian.
+- `gspice.infill`, which replaces chosen pixels of each spectrum by their Gaussian conditional mean given the other pixels, and its result `gspice.Repaired`. It reports the replaced and ignored pixels and χ², with `covariance=True` the conditional covariance of the replaced pixels of each spectrum, and with a `numpy.random.Generator` as `rng` a draw from the conditional distribution in `Repaired.draws`. Without `rng`, spectra of one S/N bin with the same replaced pixels share one factorization of the precision matrix at the pixels that all of them exclude, also where each spectrum has a few bad pixels of its own; `repair` and `calibration` take the same route where their patterns allow it.
+- `gspice.fit`, which fits a model to training spectra with the iterative outlier clipping of paper Alg. 4, and returns the model and the final training mask.
+- `gspice.finite_sample_factor`, which gives f_s(N_s, N_k, N_*), the mean χ² per predicted pixel of a held-out Gaussian spectrum when a model fitted to N_s spectra predicts N_* pixels from N_k others.
+- The option `inflate` of `gspice.pixelwise` and `gspice.infill`, which multiplies the variances and covariances of a mean-centred, unweighted model by f_s(N_s, N_k, 1). The field `Repaired.fs` holds the factor of each spectrum by which `chi2 / dof` is divided, so that `chi2 / (dof * fs)` has the mean 1 for held-out Gaussian spectra of such a model, with or without `inflate`.
+- `gspice/py.typed`, the PEP 561 marker, so that type checkers read the annotations of an installed `gspice`. The functions of `gspice.legacy` keep their 2022 signatures without annotations.
+- `gspice.fit_blocks`, which computes from spectra without artifacts a χ² threshold for each block of consecutive pixels and each S/N bin (paper section 3.2), and returns the model with them. `gspice.Blocks` holds the blocks and their thresholds in the new field `Model.blocks`, which `save` and `load` keep.
+- `gspice.noise_audit`, which compares the residuals of `gspice.pixelwise` with the reported noise at each pixel. It returns rho, the root mean of (f − f̂)² ivar over the spectra that are good at the pixel; on spectra not used in the fit and away from bad runs longer than nguard, rho² has a mean above 1 even where ivar is right.
+- `gspice.calibration`, which predicts a run or a random set of n good pixels of each held-out spectrum from its other good pixels, for each of several sizes n, with masks drawn independently of the data, and its result `gspice.Calibration`. `Calibration` holds χ², n and the finite-sample factor f_s of each prediction, and also takes the results of `infill` and `pixelwise`. Its methods give the mean of χ²/(n·f_s) and its standard error, the fraction of χ²/f_s above a quantile of the χ² distribution with n degrees of freedom, and the normal scores of a QQ plot, for all spectra or a selection of them. An error of a prediction names the row of its spectrum in `flux`.
+- `gspice.repair`, which detects and replaces bad pixels with the procedure that the paper validates on simulated spectra: two pixelwise passes, a block pass with the thresholds of `fit_blocks`, an exclusion of the pixels around extended artifacts, and a final `infill`. The second pixelwise pass tests again each pixel that the first one flagged. The field `Repaired.blocks` holds the blocks that the block pass flagged in each spectrum.
+- The option `rank` of `gspice.fit`, which fits a low-rank model: the covariance C = V diag(values) Vᵀ + diag(d) holds the K leading eigenvectors V and eigenvalues of the sample covariance, and d keeps its diagonal. Where K ≤ npix/80, Lanczos finds them, with a check that falls back to the eigendecomposition of the K pairs and logs it at INFO. The clipping passes use the same rank, so they also run with fewer spectra than pixels. `gspice.LowRank` holds V, the values and d, and `save` and `load` keep it. A model computes the precision matrix of a low-rank covariance by the Woodbury identity, without forming C, and raises `numpy.linalg.LinAlgError` where its condition bound κ_W exceeds 1e13. `inflate` needs a dense model, and `Repaired.fs` and `Calibration.fs` are 1 for a low-rank one.
+- The option `snr_bins` of `gspice.fit`, which fits a model of several S/N bins: either their number n, with edges at the quantiles that give each bin the same count of the spectra that `fit` uses before clipping, or the interior edges. Each bin runs the clipping passes and the final fit on its own spectra, so it gets its own mask, mean, covariance and training count, as `fit` gives for those spectra alone.
+- The option `noise` of `gspice.fit`, which fits a noise model in flux units. Its covariance is that of the noise-free flux, from an estimator that is unbiased with per-pixel masks, weights and the ivar of each spectrum, with its negative eigenvalues set to 0; with `rank=K`, its K leading eigenpairs plus the diagonal of the other eigenpairs, with their negative eigenvalues set to 0. The clipping passes run as without `noise`. The new field `Model.noise` marks such a model, and `save` and `load` keep it. `pixelwise`, `infill`, `repair`, `fit_blocks`, `calibration` and `noise_audit` condition each spectrum on that covariance plus its own noise variances, 1/ivar at good pixels and 1/a_s² at bad pixels, with a_s² the mean ivar of the good pixels, and form its precision matrix once per call, or, for a dense noise model in a call that forms no precision matrix for other steps, such as `infill` and `calibration`, only its Cholesky factor. `pixelwise` of a noise model computes each mean from its precision matrix times the spectrum, so an outlier within nguard of a pixel cancels in its mean only to rounding, linearly in the outlier: 9.3e-10 in z at 1e6σ in the review of PR #103. Their infills of a noise model with `rank=K` use the K eigenpairs and the diagonal plus the noise variances instead, at about npix K² flops per pattern, without the precision matrix; they raise where κ_K = 1 + λ_max(Λ^½ Vᵀ D_s⁻¹ V Λ^½) exceeds 1e13, which accepts every C_s that κ_W accepts. `noise` takes no `center` or `snr_bins`, `inflate` needs a model without noise, and `Repaired.fs` and `Calibration.fs` are 1 for a noise model.
+
+### Changed
+
+- The documentation site is built with Astro Starlight instead of Sphinx. Its API pages come from the docstrings, as before. Page addresses end in `/` instead of `.html`, for example `https://joshspeagle.github.io/GSPICE/start/install/`, and the anchors of the API and glossary entries are unchanged.
+- The documentation site has the visual design of `docs/brand.md`: dark and light themes on a navy header, the Archivo and IBM Plex Mono fonts, square corners, a new logo and favicon drawn in pixel cells, output blocks joined to their code, and framed figures.
+- The documentation site opens with a Getting started group: Is GSPICE right for my data?, Install, Quickstart, Prepare your spectra, and What you get back. The home page has an interactive figure that predicts the pixels under a block that you drag over a spectrum. The installation and quickstart pages moved to `start/install/` and `start/quickstart/`, and their old addresses redirect there.
+- The documentation site has a Guides group of seven task pages on simulated spectra, which need no download: Fill known bad pixels, Find outliers, Find and repair artifacts, Choose a model, Check your model, Save and load, and Troubleshooting. They replace the tutorials on pixelwise prediction, infilling, repair, calibration, the low-rank and noise-aware models, and saving, whose old addresses redirect to them. The glossary adds artifact, clean spectrum, flagged pixel, outlier, scaled model and shifted segment.
+- The docstrings, and so the API pages, use the terms of the glossary and show their equations as rendered maths. `README.md` has a figure of the quickstart, an example, badges and links to the pages of the site.
+- The documentation site has a Concepts group of eight pages in place of the six Background pages: The model, Fitting, Prediction, Held-out uncertainty, Detection, Diagnostics, Numerics and Glossary. They hold the ideas, the equations and their limits, with new figures drawn from `gspice.simulate` by `docs/scripts/concepts.py`, and no code to run. Held-out uncertainty and Numerics are new, Numerics with benchmark times. The glossary adds a symbol table and terms such as conditional mean, covariance, ivar and S/N, and keeps its anchors. The old Background addresses redirect to the new pages.
+- The documentation site has interactive explainers: on The model, a covariance explorer, a two-pixel explainer and the full mask explainer; in Find outliers, the guard window; and in Find and repair artifacts, the steps of `repair`. A link to a glossary entry shows the entry in a card on hover or keyboard focus.
+- The documentation site replaces its two tutorials with four pages. A survey end to end runs GSPICE on the 10000 simulated LAMOST-like spectra of `docs/scripts/spectra.py`, with figures in the style of the site. Migrate from the 2022 package gives in one table the new function, the outputs and the units for each 2022 function, then the deprecation timeline. Cite GSPICE gives the paper, `CITATION.cff` and the data of the worked example, and the Changelog page shows this file. The old tutorial addresses redirect to the new pages.
+
+### Deprecated
+
+- An access to one of the 11 functions of the 2022 package through `gspice`, as `gspice.<name>` or `from gspice import <name>`, emits a `DeprecationWarning`. It names the replacement and the removal of the name in 3.0.0. The access returns `gspice.legacy.<name>`, which keeps the 2022 results, except for the behaviour changes below, and does not warn. Type checkers report `gspice.<name>` as an unknown attribute.
+- `gspice.standard_scale`: use `gspice.scale`.
+- `gspice.covar` and `gspice.covar_iter_mask`: use `gspice.fit`.
+- `gspice.get_chimask` and `gspice.gp_interp`: use `gspice.pixelwise`.
+- `gspice.gaussian_estimate`: use `gspice.infill`.
+- `gspice.cholesky_inv`, `gspice.submatrix_inv`, `gspice.submatrix_inv_mult`, `gspice.maskinterp1` and `gspice.maskinterp`: use `gspice.legacy.<name>`.
+
+### Removed
+
+- The IDL code in `pro/` and the Julia code in `julia/`.
+- The notebooks and notes of 2021 and 2022 in `notebooks/` and `notes/`, and the 2021 draft of the paper in `paper/`.
+- `setup.py`, which `pyproject.toml` replaces.
+- The egg in `dist/` and the 2022 tests in `gspice/tests/`, which could not run and which the golden tests replace.
+- The import of astropy, which the 2022 code did not use.
+- The 10 prints of `gspice.legacy`, which reported progress, timings, shapes and notices such as "imask does not remove any rows or columns." `covar(checkmean=True)` still prints its check of the column means.
+- The LAPACK message that `cholesky_inv` printed for a 0 × 0 matrix.
+
+### Behaviour changes
+
+- The second pixelwise pass of `repair` tests each pixel again against its prediction with the replaced pixels infilled. A pixel above `nsigma_replace` in the first pass stays flagged only while it exceeds `nsigma_ignore` there, and the exclusion of step 4 counts the new sets. The paper and r26 `gp_repair` keep every such pixel, which replaced artifact-free pixels whose first-pass prediction a strong artifact had disturbed; design decision A30 gives the measurements.
+- Step 5 of `repair` on a dense noise model agrees with `infill` to rounding, not bit for bit, where steps 1 to 4 form the precision matrix of the spectrum: `infill` then uses its Cholesky factor. In the test of the steps, flux differed by 1.4e-14σ.
+- The block χ² of `fit_blocks` and `repair` reads the guard pixels of each block, which the earlier unreleased code never read. An outlier in a guard cancels only to rounding, so χ²_A changes linearly with its size: by up to 6e-8 at κ 1e8 and 1e4σ, and by 8.6 at κ 1e10 and 1e10σ (measured in PR #99).
+- The golden tests compare 11 outputs at a relative tolerance looser than 1e-10: 1e-9 for 7 outputs and 1e-8 for 4. These outputs come from ill-conditioned inputs, on which the 2022 results differ from a direct Cholesky solve by up to 7.6e-10.
+- The 2022 modules `gspice.specUtils`, `gspice.matrixUtils` and `gspice.djs_maskinterp` moved to `gspice.legacy.specUtils`, `gspice.legacy.matrixUtils` and `gspice.legacy.djs_maskinterp`, and the unused `gspice.gspiceMain` is deleted. Imports of the old module paths fail, and pickles that name the old paths no longer load. To migrate, insert `legacy.` into the module path, for example `from gspice.legacy.specUtils import standard_scale`.
+- `gspice.__all__` holds the new public names and `legacy`, so `from gspice import *` binds them. It no longer binds the 11 functions and the 3 modules of the 2022 package. It can shadow names such as `load` and `save` of an earlier `from numpy import *`. To migrate, use `from gspice.legacy import *`, which binds the same 14 names as the 2022 star import, without a warning.
+- The 2022 asserts of `gspice.legacy` raise `_LegacyError`, a subclass of `AssertionError` and `ValueError`, with a message that names the argument. An `except ValueError` clause now catches them, and the checks also run under `python -O`.
+- `gp_interp` accepts `irange` as a list, a tuple or an array of at least two integers of any integer type, and raises unless 0 <= `irange[0]` <= `irange[1]` <= npix - 1. The 2022 code rejected a 2-element `irange` and a uint64 array, and for a negative start with `irange[1]` = npix - 1 it returned leading columns of zeros. A numpy bool in `irange[0]` or `irange[1]` raises `TypeError`, and spectra without pixels raise.
+- `gp_interp` raises for a negative `nguard`, where the 2022 code predicted each pixel from itself. Under numpy 2, fixed-width numpy integers now work for `nguard`, where the 2022 code gave wrong windows or raised OverflowError or TypeError. A numpy bool or an array of one or more dimensions raises `TypeError` for `nguard`.
+- When `bruteforce` is False and `imask` keeps every row, `submatrix_inv` and `submatrix_inv_mult` return a copy of `Minv` and `MinvY`. The 2022 code returned the argument itself, so a later change to the result changed the caller's array.
+- `gp_interp` with `bruteforce=True` returns `pred` and `predvar` in the layout of the default path, where the 2022 code raised ValueError. The golden tests compare them with the 2022 outputs of the default path at the tolerances of those outputs, which are 1e-8 for 4 of them.
+- In an array with one good entry, `maskinterp1` sets a non-finite value at a bad entry to the good value, and so do `maskinterp` and `djs_maskinterp.average_bilinear`, which call it. The 2022 code gave NaN there. `standard_scale` spread that NaN to a whole column of Dvec, and `get_chimask` and `covar_iter_mask` then raised ValueError.
+- `standard_scale` ignores a non-finite ivar at a bad pixel. The 2022 code made the refscale of that spectrum NaN, and with it every value of Dvec and refmean, so `get_chimask` and `covar_iter_mask` raised ValueError.
+- `standard_scale` and `get_chimask` raise `_LegacyError` for a spectrum without a good pixel, and name its row. `covar_iter_mask` raises it for a kept spectrum that its mask or the clipping masks completely, and names its row in `flux`. The 2022 code returned NaN or raised another error.
+- `covar_iter_mask` accepts `mask=None`, which marks the pixels where ivar == 0, also for `maxbadpix`. The 2022 code raised numpy's AxisError.
+- `scale` raises ValueError for a non-finite flux at a good pixel, a non-finite or negative ivar at a pixel not in `mask`, and zero spectra without `refmean`. For such inputs, the 2022 `standard_scale` returned NaN or a wrong refscale.
+- `scale` counts a pixel with ivar = 0 as bad also where `mask` is given. It interpolates the flux of such a pixel and leaves the pixel out of refscale. The 2022 `standard_scale` with a mask used that flux and counted the 0 in the mean ivar.
+- `scale` casts flux to float64 before it interpolates. The 2022 `standard_scale` interpolated in the dtype of the flux, so it truncated the interpolated values of integer flux and rounded those of float32 flux to float32.
+- `scale` checks and computes a long double flux or ivar as float64. It raises ValueError for a long double beyond the float64 range at a good pixel, and an ivar that rounds to 0 in float64 marks a bad pixel. The 2022 `standard_scale` computed in long double and returned a finite result for such values.
+- `scale` and `snr` raise ValueError for a spectrum without a good pixel, and name its row. For such a spectrum, the 2022 `standard_scale` returned NaN in every value of Dvec and refmean.
+- `pixelwise` computes the conditional variances through the precision matrix, where the 2022 `gp_interp` used the Schur updates of a precomputed inverse. On the golden data set D3, whose covariance has the condition number 8.6e4, the 2022 predvar differs from a 60-digit reference by up to 7.6e-10 at nguard 5 and 2.7e-10 at nguard 20, relative. `pixelwise` differs by at most 1.7e-13 and 8.7e-14.
+- `infill` computes the conditional covariance through the precision matrix, where the 2022 `gaussian_estimate` used the Schur updates of a precomputed inverse. On D3, the 2022 predcovar differs from a 60-digit reference by up to 3.0e-10 relative on the diagonal, and `infill` by at most 8.8e-14.
+- `fit` masks the flags of every clipping pass, as paper Alg. 4 does. The 2022 `covar_iter_mask` masked only the flags of the last two passes; `fit(cumulative=False)` keeps that rule. On the golden data sets D1 to D4, the final mask of the default `fit` differs from the 2022 mask in 157, 152, 106 and 8 entries.
+- `fit` flags a pixel whose |z| exceeds the threshold. The 2022 `get_chimask` also flagged a pixel whose |z| equals it.
+- `fit` drops a spectrum that has no good pixel, or that a clipping mask covers completely. For such a spectrum, the 2022 `covar_iter_mask` returned a covariance of NaN or raised an error.
+- `fit` needs at least npix + 1 spectra for a mean-centred model, and a good pixel in one of them at every pixel, and raises ValueError otherwise. The 2022 `covar_iter_mask` accepted npix spectra, whose covariance is singular, and then raised LinAlgError or flagged most pixels. It filled a pixel that is bad in every spectrum by interpolation, which also makes the covariance singular; on D1 it then raised LinAlgError.
+- `fit` counts a pixel with ivar = 0 as bad, also against `maxbadpix`. The 2022 `covar_iter_mask` counted only the pixels in its mask.

pygspice-2.0.0/CITATION.cff

@@ -0,0 +1,28 @@
+cff-version: 1.2.0
+message: If you use this software, please cite the paper below.
+title: GSPICE
+version: 2.0.0
+date-released: '2026-10-29'
+license: GPL-3.0-or-later
+repository-code: https://github.com/joshspeagle/GSPICE
+url: https://joshspeagle.github.io/GSPICE/
+abstract: >-
+  GSPICE models spectra on a common wavelength grid as a multivariate Gaussian with
+  an empirical covariance, and predicts any set of pixels from the others. The
+  prediction can replace bad pixels, and the distance of a pixel from its prediction
+  can flag artifacts.
+authors: &authors
+  - family-names: Finkbeiner
+    given-names: Douglas P.
+  - family-names: Speagle
+    given-names: Joshua S.
+  - family-names: Karim
+    given-names: Tanveer
+preferred-citation:
+  type: article
+  title: Data-Driven Stellar Spectral Modelling with GSPICE
+  authors: *authors
+  journal: arXiv e-prints
+  year: 2025
+  url: https://arxiv.org/abs/2511.16754
+  doi: 10.48550/arXiv.2511.16754