spectro-kernel 0.6.2__tar.gz → 0.8.0__tar.gz

{spectro_kernel-0.6.2 → spectro_kernel-0.8.0}/.gitignore

@@ -166,8 +166,12 @@ venv.bak/
 
 # documentation - generated by tools/docs/generate.py
 docs/algorithms/
+docs/recipes/catalogue/
+docs/validation/catalogue/
 docs/ai/
 docs/llms.txt
+docs/reference/cli.md
+docs/reference/mcp.md
 
 # mkdocs documentation
 /site

spectro_kernel-0.8.0/AGENTS.md

@@ -0,0 +1,419 @@
+# AGENTS.md — guidance for AI-assisted contributors
+
+> Read this file end-to-end before adding code or proposing changes.
+> It is the standing brief for **coding agents that modify spectro-kernel**
+> (Claude Code loads it through `CLAUDE.md`; other agents read it directly).
+> It supplements [CONTRIBUTING.md](CONTRIBUTING.md) (the human-oriented
+> contributor guide) with the guard-rails that matter most when an AI is at
+> the keyboard. The complementary file for agents that *use* the library is
+> `docs/llms.txt` (generated, published at https://docs.spectrokernel.io/llms.txt):
+> `AGENTS.md` says how an AI modifies spectro-kernel, `llms.txt` how an AI
+> understands and uses it.
+
+---
+
+## 1. What the project is
+
+`spectro-kernel` is a shared catalogue of **astronomical spectroscopy
+algorithms grounded in the published literature**. The whole library
+exists so that the *N*-th project that needs continuum normalisation
+doesn't write a slightly-different *N*-th version. Every contribution
+must keep that invariant true.
+
+One catalogue, three doors: the Python library, the `spectro` CLI and the
+`spectro-mcp` MCP server all read the same registry. **Functional parity
+between the three is an invariant**, and the documentation for all three is
+generated from the code (see §7).
+
+---
+
+## 2. The non-negotiables
+
+**Before** writing or proposing an algorithm :
+
+1. **Is it already in the catalogue?** Run :
+   ```bash
+   .venv/bin/python -c "from spectro_kernel.registry import ensure_discovered, list_algorithms; \
+     ensure_discovered(); \
+     print('\n'.join(sorted(a['name'] for a in list_algorithms())))"
+   ```
+   Search the printed list with grep for related keywords *and* for the
+   nearest astronomical idiom (`bpt`, `nebular`, `redshift`,
+   `template`, `tomogr`, etc.). When in doubt, ask. **Duplicates are
+   rejected.**
+
+2. **Is it in the literature?** Every algorithm must cite at least
+   **one peer-reviewed reference** (paper, textbook, IRAF / specutils
+   / pPXF documentation) in its `references = […]` class attribute.
+   Reproduce the published formula / coefficients **verbatim** when
+   possible — `stuff/spectro_kernel_spec_25_06_26.md` is a worked
+   example. If you have to derive a coefficient yourself, that's a
+   research project, not a kernel contribution.
+
+3. **Are the references real?** Verify each citation against the
+   paper's abstract (arXiv / ADS / publisher DOI). A reference that
+   doesn't exist (or that doesn't say what you claim it says) is an
+   immediate revert.
+
+4. **Is it overlapping with astropy / specutils / pPXF / pypeit?** If
+   one of those tools already does the job in importable form, prefer
+   to **wrap** rather than re-implement. Set `backend = "astropy"` /
+   `"specutils"` / `"easyspec"` and forward — the catalogue layer
+   makes the wrap the *one* canonical entry point.
+
+5. **Will the implementation fit on a shelf?** The library's contract
+   is "one algorithm = one Python file = one `BaseAlgorithm`
+   subclass". If the design needs new context fields, a new pipeline
+   primitive, or new MCP plumbing, stop and discuss — those changes
+   ripple through every other algorithm and need separate review.
+
+If you cannot answer **yes** to all five questions, do not push code.
+
+### Anti-patterns (rejected on sight)
+
+- **Fabricated references.** "Smith et al. 2021" that nobody can find
+  in ADS is an automatic revert. Verify before citing.
+- **Domain hand-waving.** "Standard astronomy practice" is not a
+  reference. Cite the paper that establishes the formula.
+- **Duplicates.** "It's slightly different" is rarely true. If the
+  result is genuinely different, *why* — and is that documented as a
+  caveat in both bricks' `long_description`? (Example of valid
+  coexistence : `extract_spectrum_boxcar` vs `extract_spectrum_optimal`
+  — both are Horne-flavoured but the optimal variant adds variance +
+  cosmic rejection. The two `long_description`s cross-reference each
+  other so users can pick.)
+- **API drift.** Don't introduce a new `ctx.something` field, a new
+  return-type shape, or a new registration mechanism. The triple
+  surface (Python library, CLI, MCP) is generated from the registry —
+  any drift breaks one of them silently.
+- **ML-flashy reinvention.** A "Bayesian-deep-something" wrapper of a
+  classical algorithm is not a new algorithm if it doesn't change the
+  published method. Keep the kernel boring on purpose.
+- **Inventing parameter names that don't match the literature.** If
+  the paper says ``R = F(6716)/F(6731)``, your variable is `R` and
+  your param is `lambda_6716` — not `lhs_line_a`.
+- **Hardcoded paths to home directories, network fetches at import
+  time, secret tokens.** None of those belong in a kernel algorithm.
+
+### The kernel philosophy (the "why")
+
+If you have to pick between two options, the right one is usually the
+one closest to these principles :
+
+- **Literature over invention.** The kernel is a curated archive of
+  *published* recipes. We do not invent algorithms here.
+- **One algorithm = one shelf brick.** Composability comes from
+  pipelines composing bricks, not from monolithic super-algorithms.
+- **No surprise.** Every algorithm declares its `backend`, its
+  `references`, its `default_params` and its `output_produces`. The
+  registry is a contract — code without metadata is rejected.
+- **Provenance is a feature.** Each `BaseAlgorithm.execute` writes a
+  `ProcessingStep` into `ctx.history` (parameters, input hash, output
+  hash, duration, success). Don't bypass it.
+- **Determinism by default.** Random sources are explicit (seeded
+  from `params`), file I/O is explicit, network calls are explicit
+  and lazy (inside `run()`, never at module import).
+- **Triple surface.** Every algorithm becomes a Python function, a
+  `spectro run` CLI command **and** an MCP tool — for free. Don't write
+  per-surface code.
+- **No data in the wheel.** Catalogues / atlases / templates are
+  loaded on demand from disk (see
+  [`algorithms/wavelength_calibration/lamp_atlas.py`](src/spectro_kernel/algorithms/wavelength_calibration/lamp_atlas.py)
+  for a small bundled one, and
+  [`algorithms/classification/pickles_atlas.py`](src/spectro_kernel/algorithms/classification/pickles_atlas.py)
+  for the externally-fetched variant). Keep the install size honest.
+
+---
+
+## 3. Architecture (memorise this)
+
+```
+src/spectro_kernel/        the importable library — the catalogue
+  types/                   Spectrum1D, WorkContext, ProcessingStep, LineFitResult, enums
+  base.py                  BaseAlgorithm + AlgorithmOutput (the algorithm contract)
+  registry.py              @register_algorithm, discovery, list/describe/run_algorithm
+  registry_export.py       the ONE builder of the catalogue JSON (algorithms + recipes)
+  pipeline.py              Pipeline, PipelineBuilder (from_preset), PipelineResult
+  presets/                 recipe loader, variables, validation, bundled catalog/*.yaml
+  io/                      FITS / ASCII / VOTable readers and writers
+  algorithms/<category>/   one file per algorithm — auto-discovered
+  cli.py                   the `spectro` command (argparse, build_parser())
+src/spectro_mcp/           the MCP server over the same registry
+  server.py                build_server(): FastMCP + tools + resources + /health, /metrics
+  auto_tools.py            13 session/discovery tools + one tool per algorithm, path/URL guards
+  resources.py             spectro://algorithms, spectro://recipes, spectro://llms.txt
+tools/docs/                documentation generators (§7), build.py = what CI runs
+docs/                      the documentation site source (Markdown, Zensical / MkDocs)
+examples/                  executable, deterministic examples run by CI
+tests/unit/                the test suite (one test file per brick or feature)
+website/                   the public landing page (React) — separate from docs/
+```
+
+### The algorithm pattern
+
+Every algorithm follows this pattern (see e.g.
+[`src/spectro_kernel/algorithms/continuum/normalize_polynomial.py`](src/spectro_kernel/algorithms/continuum/normalize_polynomial.py),
+[`src/spectro_kernel/algorithms/nebular/oiii_electron_temperature.py`](src/spectro_kernel/algorithms/nebular/oiii_electron_temperature.py)) :
+
+```python
+from __future__ import annotations
+from typing import Any
+
+import numpy as np
+
+from ...base import AlgorithmOutput, BaseAlgorithm
+from ...errors import InvalidParameterError
+from ...registry import register_algorithm
+from ...types import AlgorithmCategory, WorkContext
+
+
+@register_algorithm("name", category=AlgorithmCategory.SOMETHING, version="1.0.0")
+class MyAlgorithm(BaseAlgorithm):
+    """One-line summary (becomes the registry description)."""
+
+    backend = "numpy"                          # numpy | scipy | astropy | specutils | …
+    references = ["Author YYYY, Journal vol, page — short note."]
+    long_description = "Optional richer paragraph with caveats / regime of validity."
+    default_params = {"threshold": 3.0}
+    required_params = ["threshold"]            # only when there is no sensible default
+    param_descriptions = {"threshold": "…"}
+    input_requirements = ["spectrum"]          # "image" | "spectrum" | "spectra" | "images"
+    output_produces = ["metrics.x"]            # documentation only
+
+    def run(self, ctx: WorkContext, params: dict[str, Any]) -> AlgorithmOutput:
+        spec = ctx.spectrum
+        if spec is None:
+            return AlgorithmOutput.fail("…")
+        # …
+        return AlgorithmOutput.ok(metrics={...}, message="…")
+```
+
+**Auto-discovery** picks the file up from its category directory at
+import time — no registry edit required. The metadata is the contract:
+it feeds `spectro describe`, the MCP tool description, the catalogue
+page and the machine-readable JSON, all generated.
+
+### Context conventions (do NOT add new fields)
+
+| Data | Slot | Type |
+|---|---|---|
+| current 2-D frame | `ctx.image` | `ImageFrame` |
+| current 1-D spectrum | `ctx.spectrum` | `Spectrum1D` |
+| list of 2-D frames | `ctx.images` | `list[ImageFrame]` |
+| list of 1-D spectra | `ctx.spectra` | `list[Spectrum1D]` |
+| **second spectrum** (template, reference, sky, std) | `ctx.extras["<key>"]` | `Spectrum1D` |
+| scalar metrics (JSON-safe) | `ctx.metrics` | `dict[str, float]` |
+| line fits | `ctx.line_fits` | `dict[str, LineFitResult]` |
+| anything else (artefacts, intermediate state) | `ctx.extras["<key>"]` | any JSON-safe |
+
+**The "second spectrum lives in `ctx.extras`" rule is invariant.**
+Examples: `remove_telluric_division` reads
+`ctx.extras["telluric_spectrum"]`,
+`combine_spectra_arithmetic` reads `ctx.extras["reference_spectrum"]`,
+`response_from_standard` writes `ctx.extras["reference_spectrum"]`,
+`extract_sky_lateral_bands` writes `ctx.extras["sky_spectrum"]`. If
+your algorithm needs another spectrum, follow the same convention with
+a self-documenting key.
+
+### Categories
+
+`AlgorithmCategory` is the curated taxonomy
+([`src/spectro_kernel/types/enums.py`](src/spectro_kernel/types/enums.py)).
+Use an existing member when possible. New categories require :
+
+1. a real use case (3+ algorithms or a clear domain gap),
+2. an enum entry,
+3. a friendly title in `CATEGORY_TITLES` of
+   [`src/spectro_kernel/registry_export.py`](src/spectro_kernel/registry_export.py)
+   (shared by the docs generators and the MCP resources),
+4. an update to the website catalogue group if the brick is
+   user-facing (see
+   [`website/components/sections.jsx`](website/components/sections.jsx)).
+
+Adding a category is fine — the v0.4.0 release added `nebular`,
+`kinematics` and `classification`. Don't add a category for one algo
+unless the domain really is new.
+
+### The MCP layer
+
+`spectro_mcp` depends on `spectro_kernel`, never the reverse. Tools are
+generated from the registry in
+[`src/spectro_mcp/auto_tools.py`](src/spectro_mcp/auto_tools.py) (one tool
+per algorithm, plus 13 session / discovery tools); the catalogue is also
+readable as resources in
+[`src/spectro_mcp/resources.py`](src/spectro_mcp/resources.py)
+(`spectro://algorithms`, `spectro://algorithms/{name}`, `spectro://recipes`,
+`spectro://recipes/{name}`, `spectro://llms.txt`), backed by
+`registry_export.py`. Safety (path / URL guards, masking, deny mode) lives
+at the MCP layer; kernel algorithms stay policy-free. In HTTP deny mode
+nothing may disclose an install path.
+
+---
+
+## 4. Recipes (presets) and profiles
+
+A **recipe** (`src/spectro_kernel/presets/catalog/*.yaml`, or an installed
+collection) composes bricks with the scientific conventions fixed;
+everything that depends on an instrument or an observer is a declared
+**variable**, bound at run time by a **profile** (`--profile`, `--set`,
+`variables=`). The loader lives in
+[`src/spectro_kernel/presets/`](src/spectro_kernel/presets/) (variables,
+discovery through `SPECTRO_PRESET_PATH` and the `spectro_kernel.presets`
+entry-point group, validation, `requires` range), the engine records a
+`pipeline:<name>` provenance step at the end of every named recipe run.
+Rules: a recipe never hard-codes an instrument value, never introduces a
+brick, and campaign recipes cite references. The kernel bundles only
+generic recipes; campaign recipes belong to
+[spectro-kernel-recipes](https://github.com/matthieulel/spectro-kernel-recipes).
+`spectro preset validate` is the gate.
+
+---
+
+## 5. Commands and environment
+
+- Project root: `/Users/matthieu/lab/meta-spectro/spectro-kernel`
+- Python : 3.12 (managed via `pyenv` ; project venv at `.venv/`).
+- Run any tool via `.venv/bin/<tool>` (e.g. `.venv/bin/pytest`).
+- Install for development: `uv pip install -e ".[dev,mcp]"` (add `docs`
+  to build the site, `all` for every optional extra).
+- Lint / format: `.venv/bin/python -m ruff check src tests tools examples`
+  and `.venv/bin/python -m ruff format <files>` (line length 100, rules
+  `E F I UP B W`, see `pyproject.toml`). CI runs `ruff check src tests`.
+- Docs : `.venv/bin/python tools/docs/generate.py && .venv/bin/zensical serve`
+  then open http://127.0.0.1:8000. Published at https://docs.spectrokernel.io/.
+- MCP server stdio mode : `.venv/bin/spectro-mcp` (cloud HTTP variant
+  deployed on DigitalOcean).
+- Examples : `.venv/bin/python examples/<file>.py` (each is
+  self-contained, deterministic, offline).
+
+If you are inside Claude Code, your tools are : `Read`, `Edit`,
+`Write`, `Bash`, `Agent`. Stay inside the project working directory ;
+never run commands that touch `~` or `/etc`.
+
+---
+
+## 6. Tests (required for every contribution)
+
+Each new algorithm ships with **at least one** test in `tests/unit/`.
+The test must :
+
+1. Build a synthetic input where the right answer is **known
+   analytically from the cited literature** (e.g.
+   `tests/unit/test_nebular_diagnostics.py::test_oiii_inversion_acceptance_values`
+   exercises Eq. 1 from Aller 1984 directly).
+2. Assert on the numerical result with a tolerance that reflects the
+   method's known precision — not blindly tight, not generously loose.
+3. Cover at least one failure path (parameter validation, missing
+   input, regime out of validity).
+
+When the spec we are porting from already lists *acceptance values*
+(STAROS-style spec docs), put them in the test docstring verbatim.
+
+Scientific validation against real data lives in `tests/reference/`
+(known-answer tests on reference spectra); a brick that claims a published
+precision should be checked there when reference data exists.
+
+Run before pushing :
+
+```bash
+.venv/bin/python -m pytest tests/                     # the whole suite must pass
+.venv/bin/python -m ruff check src tests tools examples   # clean
+.venv/bin/python tools/docs/build.py                  # generators + zensical build --strict, clean
+.venv/bin/python -m build && .venv/bin/python -m twine check --strict dist/*
+```
+
+---
+
+## 7. Documentation requirements and how the generated docs work
+
+The documentation is generated from the code wherever a source of truth
+exists; **never edit a generated file**, edit the generator or the metadata
+it reads. Generated outputs are git-ignored (`.gitignore`, "documentation"
+block) and rebuilt by `tools/docs/generate.py` (called by `build.py`, by
+CI and by Netlify):
+
+| Generator | Reads | Writes |
+|---|---|---|
+| `tools/docs/generate_catalogue.py` | the registry | `docs/algorithms/**`, `docs/ai/algorithms.json` + schema |
+| `tools/docs/generate_recipes.py` | the presets loader | `docs/recipes/catalogue/**`, `docs/ai/presets.json` |
+| `tools/docs/generate_cli.py` | `spectro_kernel.cli.build_parser()` | `docs/reference/cli.md` |
+| `tools/docs/generate_mcp.py` | `spectro_mcp.server.build_server()` | `docs/reference/mcp.md`, `docs/ai/mcp-tools.json` |
+| `tools/docs/generate_llms.py` | `mkdocs.yml` nav + page descriptions | `docs/llms.txt` |
+
+The JSON builders live in `src/spectro_kernel/registry_export.py` so the
+published JSON and the MCP resources are one implementation. Hand-written
+pages (`docs/concepts`, `docs/usage`, `docs/tutorials`, `docs/reference/python.md`)
+link to the generated ones; the nav is in `mkdocs.yml`. `build.py` runs
+Zensical in `--strict` mode: a broken link or a missing page fails CI.
+
+A new algorithm needs **no** documentation edit: its docstring and
+metadata become its page, its MCP tool description and its JSON entry. A
+new CLI flag or MCP tool is documented on the next build. A new
+user-facing feature gets a paragraph in the relevant `docs/usage` or
+`docs/concepts` page and, when it can be shown in under 60 lines, an
+example in `examples/` (deterministic, offline, registered in
+`tests/unit/test_examples_run.py`, which CI runs).
+
+---
+
+## 8. Versioning
+
+- **PATCH** (`0.x.y → 0.x.(y+1)`) : bug fix + at most one small
+  incremental addition (cf. v0.2.1 — uint16 BZERO leak fix + one new
+  `extract_spectrum_boxcar` brick).
+- **MINOR** (`0.x → 0.(x+1)`) : substantive feature add (cf. v0.2.0 —
+  13 long-slit bricks ; v0.4.0 — 8 science bricks + 3 new categories).
+- **MAJOR** (`0.x → 1.0+`) : only after a `1.0.0` API freeze decision.
+
+Bump the **algorithm's own** `version` attribute on any behaviour
+change so old results stay reproducible. Numerical change → MAJOR
+attribute bump (`1.0.0 → 2.0.0`). Performance / docstring → MINOR /
+PATCH.
+
+---
+
+## 9. Release flow (when asked to ship)
+
+1. Update [CHANGELOG.md](CHANGELOG.md) with full per-algorithm notes
+   (every new brick named, every reference cited, numeric impact
+   measured when relevant — see the v0.2.1 / v0.3.0 / v0.4.0 entries
+   for the expected level of detail).
+2. Bump [`src/spectro_kernel/version.py`](src/spectro_kernel/version.py).
+3. Update the website counters / catalogue group entries in
+   [`website/app.jsx`](website/app.jsx) and
+   [`website/components/sections.jsx`](website/components/sections.jsx).
+4. Run the pre-flight quartet from section 6.
+5. Commit, push, tag (`git tag -a vX.Y.Z -m "…"`), push the tag.
+6. The
+   [`release.yml`](.github/workflows/release.yml)
+   GitHub Action runs the test matrix, builds the wheel + sdist, and
+   publishes to PyPI via Trusted Publisher OIDC (no token in the
+   repo). Watch the run ; PyPI propagation usually completes within
+   one minute of workflow success.
+
+---
+
+## 10. When to escalate to the user
+
+Surface a decision to the user (rather than guessing) when :
+
+- A new external dependency would land in the wheel (changes the
+  install footprint).
+- A change would affect the public registry contract (renaming an
+  algorithm, removing a parameter, changing a return shape).
+- A pipeline's deterministic output would shift numerically (algorithm
+  MAJOR bump scenario).
+- A release would skip a version on PyPI (cf. the v0.1.5 re-tag
+  scenario).
+- The proposed addition is genuinely on the boundary between "shelf
+  brick" and "downstream application" — when in doubt, the kernel
+  errs on the side of being smaller.
+
+For everything routine — implementing a brick whose spec is unambiguous,
+fixing a clear bug, refreshing docs — proceed and report.
+
+---
+
+*This file is the standing brief for AI contributors. Keep it short —
+human contributors read [CONTRIBUTING.md](CONTRIBUTING.md), AI
+contributors read both.*