cesm-hawc 0.3.0__tar.gz

cesm_hawc-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Victoria McDonald
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cesm_hawc-0.3.0/PKG-INFO

@@ -0,0 +1,205 @@
+Metadata-Version: 2.4
+Name: cesm-hawc
+Version: 0.3.0
+Summary: Simulate HAWC ALI limb observations and retrievals from CESM2/WACCM output
+Author: Victoria McDonald
+License-Expression: MIT
+Project-URL: Documentation, https://torimcd.github.io/cesm-hawc/
+Project-URL: Source, https://github.com/torimcd/cesm-hawc
+Project-URL: Issues, https://github.com/torimcd/cesm-hawc/issues
+Project-URL: Changelog, https://torimcd.github.io/cesm-hawc/changelog
+Keywords: CESM,WACCM,stratospheric aerosol,limb scattering,HAWC,ALI,remote sensing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: xarray
+Requires-Dist: scipy
+Requires-Dist: pandas
+Requires-Dist: netCDF4
+Requires-Dist: dask
+Provides-Extra: sim
+Requires-Dist: aliprocessing>=2026.10.0; extra == "sim"
+Requires-Dist: hawcsimulator>=2026.6.0; extra == "sim"
+Requires-Dist: sasktran2; extra == "sim"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: jupyter; extra == "dev"
+Requires-Dist: matplotlib; extra == "dev"
+Requires-Dist: cartopy; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: jupyter-book>=2; extra == "docs"
+Dynamic: license-file
+
+# cesm-hawc
+
+Feed CESM2/WACCM atmosphere model output into the
+[HAWC ALI simulator](https://github.com/usask-arg/hawc-simulator) to
+simulate what the HAWCSat Aerosol Limb Imager would observe.
+
+**Documentation:** <https://torimcd.github.io/cesm-hawc/>, including a step-by-step
+[tutorial](https://torimcd.github.io/cesm-hawc/tutorial) on an example column.
+
+## What this does
+
+Given CESM/WACCM history output this package:
+
+1. Extracts one or many atmospheric columns (T, P, O₃, MAM4 sulfate aerosol)
+2. Converts MAM4 modal aerosol to extinction profiles using mode-matched Mie databases
+3. Runs the `IdealALISimulator` forward model and, optionally, the L2 retrieval
+4. Outputs retrieved aerosol extinction and median radius profiles
+
+Requires Python >=3.11. It has two tiers:
+
+- **Base** (`pip install cesm-hawc`) — WACCM column extraction and saving
+  simulator-ready inputs. Only needs numpy, xarray, scipy, pandas, netCDF4
+  and dask.
+- **`[sim]` extra** — the full forward model and L2 retrieval, via
+  `hawcsimulator` + `sasktran2`. Install it into the project's conda
+  environment (see below).
+
+## Install
+
+To run the simulator, install into the project's conda environment
+(recommended):
+
+```bash
+curl -O https://raw.githubusercontent.com/torimcd/cesm-hawc/main/environment.yml
+conda env create -f environment.yml
+conda activate hawc_env
+pip install "cesm-hawc[sim]"
+```
+
+For the base tier only (no simulator), pip on its own is enough:
+
+```bash
+pip install cesm-hawc
+```
+
+**From source (development):**
+
+```bash
+git clone https://github.com/torimcd/cesm-hawc
+cd cesm-hawc
+conda env create -f environment.yml
+conda activate hawc_env
+pip install -e ".[sim,dev]"
+```
+
+See the [installation guide](https://torimcd.github.io/cesm-hawc/installation)
+for details.
+
+**Alliance Canada HPC (Fir/Rorqual/Narval):**
+
+```bash
+git clone https://github.com/torimcd/cesm-hawc
+cd cesm-hawc
+bash scripts/setup/create_env.sh
+```
+
+That script creates the `hawc_env` micromamba environment, installs all
+dependencies, and registers the package. It takes 5–10 minutes on first run.
+
+## Quick start
+
+### CLI
+
+Copy `config.example.toml` to `config.toml` and fill in your paths. Each run
+processes one model case, described by the `[case]` table, and writes under
+`out_dir/<case name>/`.
+
+```bash
+# Base tier: extract and save WACCM column inputs, no [sim] extra needed
+cesm-hawc save-inputs --config config.toml --mode fixed
+
+# [sim] tier: run the forward model (+ L2 retrieval if run_l2 = true)
+cesm-hawc run --config config.toml --mode fixed
+```
+
+`--mode` selects how the model is sampled:
+
+| Mode | Samples | Config table |
+|------|---------|--------------|
+| `fixed` | one column at a fixed tangent point and solar geometry, per history file | `[fixed]` |
+| `orbit` | columns along a real HAWC orbit ground track, moved onto the case's dates | `[orbit]` |
+
+Add `--dry-run` to see the job count without running anything, and
+`--case-name NAME` to run a different case from the same config. Run
+`cesm-hawc save-inputs --help` / `cesm-hawc run --help` for the full flag
+list.
+
+### Python API
+
+```python
+import cesm_hawc
+from cesm_hawc.noise import default_noise_model
+from cesm_hawc.simulation import run_ali_simulation
+
+cesm_hawc.configure_environment()   # once per process; the CLI does this for you
+
+result = run_ali_simulation(
+    "path/to/my_case.cam.h0.2035-02.nc",
+    lat=30.6, lon=180.0,
+    run_l2=True,
+    noise_model=default_noise_model(),
+)
+l2 = result["data"]["l2"]   # retrieved extinction, median radius, ...
+```
+
+Library users calling into `cesm_hawc.orbit_files`, `cesm_hawc.calibration`,
+or `cesm_hawc.simulation` directly (rather than through the CLI) should call
+`cesm_hawc.configure_environment()` once at startup — it disables astropy's
+IERS auto-download, silences noisy third-party logging, and patches a known
+`hawcsimulator` calibration-cache race condition.
+
+## Using saved inputs without cesm-hawc
+
+When `sasktran2` is installed, each file written by `save-inputs` contains
+everything needed to rebuild the simulator's aerosol and gas constituents
+with native `sasktran2` calls: per-mode reference and truth extinction,
+clipped median radius, and the Mie database build parameters as attributes.
+See [Using saved inputs](https://torimcd.github.io/cesm-hawc/saved-inputs) for a complete example.
+
+## Required WACCM output variables
+
+Add these to `fincl` in `user_nl_cam` if not already present:
+
+| Variable | Description |
+|----------|-------------|
+| `T`, `Q`, `PS` | Temperature, humidity, surface pressure (required) |
+| `hyam`, `hybm`, `P0` | Hybrid level coefficients and reference pressure (`hyam`, `hybm` required) |
+| `O3`, `NO2`, `SO2`, `H2O` | Gas chemistry (mol/mol) |
+| `so4_a1`, `so4_a3` | Dry sulfate mass mixing ratio (kg/kg) |
+| `num_a1`, `num_a3` | Aerosol number mixing ratio (#/kg) |
+| `dgnumwet1`, `dgnumwet3` | Aerosol wet median diameter (m) |
+
+See [WACCM output variables](https://torimcd.github.io/cesm-hawc/waccm-variables) for details.
+
+## Citing
+
+If you use cesm-hawc in published work, please cite it using the details in
+[`CITATION.cff`](CITATION.cff) (on GitHub, "Cite this repository"), and cite
+the simulator packages it builds on; see
+[Citing cesm-hawc](https://torimcd.github.io/cesm-hawc/citing).
+
+## Testing
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+Tests that need `sasktran2`/`hawcsimulator` are automatically skipped if
+those aren't installed. A small bundled example column
+(`src/cesm_hawc/data/example_column.nc`) is used for fixture-based tests
+and doesn't require any external data.

cesm_hawc-0.3.0/README.md

@@ -0,0 +1,162 @@
+# cesm-hawc
+
+Feed CESM2/WACCM atmosphere model output into the
+[HAWC ALI simulator](https://github.com/usask-arg/hawc-simulator) to
+simulate what the HAWCSat Aerosol Limb Imager would observe.
+
+**Documentation:** <https://torimcd.github.io/cesm-hawc/>, including a step-by-step
+[tutorial](https://torimcd.github.io/cesm-hawc/tutorial) on an example column.
+
+## What this does
+
+Given CESM/WACCM history output this package:
+
+1. Extracts one or many atmospheric columns (T, P, O₃, MAM4 sulfate aerosol)
+2. Converts MAM4 modal aerosol to extinction profiles using mode-matched Mie databases
+3. Runs the `IdealALISimulator` forward model and, optionally, the L2 retrieval
+4. Outputs retrieved aerosol extinction and median radius profiles
+
+Requires Python >=3.11. It has two tiers:
+
+- **Base** (`pip install cesm-hawc`) — WACCM column extraction and saving
+  simulator-ready inputs. Only needs numpy, xarray, scipy, pandas, netCDF4
+  and dask.
+- **`[sim]` extra** — the full forward model and L2 retrieval, via
+  `hawcsimulator` + `sasktran2`. Install it into the project's conda
+  environment (see below).
+
+## Install
+
+To run the simulator, install into the project's conda environment
+(recommended):
+
+```bash
+curl -O https://raw.githubusercontent.com/torimcd/cesm-hawc/main/environment.yml
+conda env create -f environment.yml
+conda activate hawc_env
+pip install "cesm-hawc[sim]"
+```
+
+For the base tier only (no simulator), pip on its own is enough:
+
+```bash
+pip install cesm-hawc
+```
+
+**From source (development):**
+
+```bash
+git clone https://github.com/torimcd/cesm-hawc
+cd cesm-hawc
+conda env create -f environment.yml
+conda activate hawc_env
+pip install -e ".[sim,dev]"
+```
+
+See the [installation guide](https://torimcd.github.io/cesm-hawc/installation)
+for details.
+
+**Alliance Canada HPC (Fir/Rorqual/Narval):**
+
+```bash
+git clone https://github.com/torimcd/cesm-hawc
+cd cesm-hawc
+bash scripts/setup/create_env.sh
+```
+
+That script creates the `hawc_env` micromamba environment, installs all
+dependencies, and registers the package. It takes 5–10 minutes on first run.
+
+## Quick start
+
+### CLI
+
+Copy `config.example.toml` to `config.toml` and fill in your paths. Each run
+processes one model case, described by the `[case]` table, and writes under
+`out_dir/<case name>/`.
+
+```bash
+# Base tier: extract and save WACCM column inputs, no [sim] extra needed
+cesm-hawc save-inputs --config config.toml --mode fixed
+
+# [sim] tier: run the forward model (+ L2 retrieval if run_l2 = true)
+cesm-hawc run --config config.toml --mode fixed
+```
+
+`--mode` selects how the model is sampled:
+
+| Mode | Samples | Config table |
+|------|---------|--------------|
+| `fixed` | one column at a fixed tangent point and solar geometry, per history file | `[fixed]` |
+| `orbit` | columns along a real HAWC orbit ground track, moved onto the case's dates | `[orbit]` |
+
+Add `--dry-run` to see the job count without running anything, and
+`--case-name NAME` to run a different case from the same config. Run
+`cesm-hawc save-inputs --help` / `cesm-hawc run --help` for the full flag
+list.
+
+### Python API
+
+```python
+import cesm_hawc
+from cesm_hawc.noise import default_noise_model
+from cesm_hawc.simulation import run_ali_simulation
+
+cesm_hawc.configure_environment()   # once per process; the CLI does this for you
+
+result = run_ali_simulation(
+    "path/to/my_case.cam.h0.2035-02.nc",
+    lat=30.6, lon=180.0,
+    run_l2=True,
+    noise_model=default_noise_model(),
+)
+l2 = result["data"]["l2"]   # retrieved extinction, median radius, ...
+```
+
+Library users calling into `cesm_hawc.orbit_files`, `cesm_hawc.calibration`,
+or `cesm_hawc.simulation` directly (rather than through the CLI) should call
+`cesm_hawc.configure_environment()` once at startup — it disables astropy's
+IERS auto-download, silences noisy third-party logging, and patches a known
+`hawcsimulator` calibration-cache race condition.
+
+## Using saved inputs without cesm-hawc
+
+When `sasktran2` is installed, each file written by `save-inputs` contains
+everything needed to rebuild the simulator's aerosol and gas constituents
+with native `sasktran2` calls: per-mode reference and truth extinction,
+clipped median radius, and the Mie database build parameters as attributes.
+See [Using saved inputs](https://torimcd.github.io/cesm-hawc/saved-inputs) for a complete example.
+
+## Required WACCM output variables
+
+Add these to `fincl` in `user_nl_cam` if not already present:
+
+| Variable | Description |
+|----------|-------------|
+| `T`, `Q`, `PS` | Temperature, humidity, surface pressure (required) |
+| `hyam`, `hybm`, `P0` | Hybrid level coefficients and reference pressure (`hyam`, `hybm` required) |
+| `O3`, `NO2`, `SO2`, `H2O` | Gas chemistry (mol/mol) |
+| `so4_a1`, `so4_a3` | Dry sulfate mass mixing ratio (kg/kg) |
+| `num_a1`, `num_a3` | Aerosol number mixing ratio (#/kg) |
+| `dgnumwet1`, `dgnumwet3` | Aerosol wet median diameter (m) |
+
+See [WACCM output variables](https://torimcd.github.io/cesm-hawc/waccm-variables) for details.
+
+## Citing
+
+If you use cesm-hawc in published work, please cite it using the details in
+[`CITATION.cff`](CITATION.cff) (on GitHub, "Cite this repository"), and cite
+the simulator packages it builds on; see
+[Citing cesm-hawc](https://torimcd.github.io/cesm-hawc/citing).
+
+## Testing
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+Tests that need `sasktran2`/`hawcsimulator` are automatically skipped if
+those aren't installed. A small bundled example column
+(`src/cesm_hawc/data/example_column.nc`) is used for fixture-based tests
+and doesn't require any external data.

cesm_hawc-0.3.0/pyproject.toml

@@ -0,0 +1,73 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cesm-hawc"
+dynamic = ["version"]
+description = "Simulate HAWC ALI limb observations and retrievals from CESM2/WACCM output"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Victoria McDonald" }]
+requires-python = ">=3.11"
+keywords = ["CESM", "WACCM", "stratospheric aerosol", "limb scattering", "HAWC", "ALI", "remote sensing"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering :: Atmospheric Science",
+]
+dependencies = [
+    "numpy",
+    "xarray",
+    "scipy",
+    "pandas",
+    "netCDF4",
+    "dask",
+]
+
+[project.urls]
+Documentation = "https://torimcd.github.io/cesm-hawc/"
+Source = "https://github.com/torimcd/cesm-hawc"
+Issues = "https://github.com/torimcd/cesm-hawc/issues"
+Changelog = "https://torimcd.github.io/cesm-hawc/changelog"
+
+[project.scripts]
+cesm-hawc = "cesm_hawc.cli:main"
+
+[project.optional-dependencies]
+# End-to-end simulation (the `cesm-hawc run` CLI command / cesm_hawc.simulation
+# / cesm_hawc.constituents).
+sim = [
+    "aliprocessing>=2026.10.0",
+    "hawcsimulator>=2026.6.0",
+    "sasktran2",
+]
+dev = ["pytest", "jupyter", "matplotlib", "cartopy"]
+docs = ["jupyter-book>=2"]
+
+[tool.setuptools.dynamic]
+version = { attr = "cesm_hawc.__version__" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+cesm_hawc = ["data/*.nc"]
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F", "D"]
+ignore = ["D105", "D107"]  # magic methods; __init__ is documented on the class
+
+[tool.ruff.lint.pydocstyle]
+convention = "numpy"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

cesm_hawc-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

cesm_hawc-0.3.0/src/cesm_hawc/__init__.py

@@ -0,0 +1,15 @@
+"""Simulate HAWC ALI observations and retrievals from CESM2/WACCM output.
+
+The package has two tiers. The base install extracts WACCM columns and
+saves simulator-ready inputs (:mod:`cesm_hawc.waccm`,
+:mod:`cesm_hawc.save_inputs`). The ``[sim]`` extra adds the ALI forward
+model and L2 retrieval (:mod:`cesm_hawc.simulation`,
+:mod:`cesm_hawc.constituents`). Call :func:`configure_environment` once
+before using the ``[sim]`` tier from your own code.
+"""
+
+from cesm_hawc.env import configure_environment
+
+__all__ = ["configure_environment"]
+
+__version__ = "0.3.0"  # the package version; pyproject.toml reads it from here

cesm_hawc-0.3.0/src/cesm_hawc/calibration.py

@@ -0,0 +1,111 @@
+"""Workarounds for cache races when many simulations run in parallel.
+
+``hawcsimulator`` and ``aliprocessing`` build cached files the first time a
+simulator is constructed. When many worker processes start at once, they
+race to write those files. The functions here pre-build the caches once in
+the main process and stop ``hawcsimulator`` rewriting its calibration file
+on every call. They do nothing if the ``[sim]`` extra isn't installed.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+
+log = logging.getLogger(__name__)
+
+_patched = False
+
+
+def _cache_file_path(name: str, version: str) -> str:
+    """Path of ``hawcsimulator``'s cached calibration file for ``(name, version)``.
+
+    Built the same way as ``hawcsimulator.ali.calibration``, from the
+    platform's user data directory (e.g. ``~/.local/share`` on Linux,
+    ``~/Library/Application Support`` on macOS).
+    """
+    import appdirs  # a hawcsimulator dependency
+
+    data_dir = appdirs.AppDirs(appname="hawc-simulator", appauthor="usask-arg").user_data_dir
+    return os.path.join(data_dir, "ali", "calibration", f"{name}_{version}.nc")
+
+
+def patch_calibration_database_race() -> None:
+    """Stop ``hawcsimulator`` rewriting its calibration cache on every call.
+
+    ``hawcsimulator``'s ``calibration_database()`` rewrites its cached NetCDF
+    file every time it is called, including whenever an
+    ``IdealALISimulator`` is constructed. With many worker processes using
+    the same (often network-mounted) file, this causes ``PermissionError``
+    and ``KeyError`` races. After patching, an existing cache file is reused
+    instead. This is safe because the file's contents depend only on its
+    ``(name, version)``.
+
+    Safe to call more than once; does nothing if ``hawcsimulator`` isn't
+    installed. :func:`cesm_hawc.configure_environment` calls it for you.
+    """
+    global _patched
+    if _patched:
+        return
+    try:
+        from hawcsimulator.ali import calibration as _cal_mod
+        from hawcsimulator.ali.configurations import (
+            ideal_dolp_imager as _ideal_dolp_imager_mod,
+        )
+    except ImportError:
+        return
+
+    _orig_calibration_database = _cal_mod.calibration_database
+
+    def _safe_calibration_database(name: str, version: str):
+        cache_file = _cache_file_path(name, version)
+        if os.path.exists(cache_file):
+            return cache_file
+        return _orig_calibration_database(name, version)
+
+    _cal_mod.calibration_database = _safe_calibration_database
+    _ideal_dolp_imager_mod.calibration_database = _safe_calibration_database
+    _patched = True
+
+
+def warm_calibration_database(name: str = "ideal_spectrograph", version: str = "v1") -> None:
+    """Build ``hawcsimulator``'s calibration cache file once.
+
+    Call this in the main process before starting worker processes, so they
+    don't all race to create the file. Failures are logged as warnings, not
+    raised.
+
+    Parameters
+    ----------
+    name, version : str, optional
+        Calibration dataset to build. The defaults are the dataset the
+        ``ideal_dolp_imager`` simulator uses.
+    """
+    try:
+        from hawcsimulator.ali.calibration import calibration_database
+    except ImportError:
+        return
+    try:
+        calibration_database(name, version)
+    except Exception as e:
+        log.warning("Could not pre-warm calibration database: %s", e)
+
+
+def warm_retrieval_optical_database() -> None:
+    """Build the retrieval's Mie database cache once.
+
+    Every ``IdealALISimulator`` builds ``aliprocessing``'s
+    ``aerosol_median_radius_db()`` when constructed. Call this in the main
+    process before starting worker processes, so they don't race to write
+    it (a worker reading a half-written file fails with an xarray "did not
+    find a match in any of xarray's currently installed IO backends"
+    error). Failures are logged as warnings, not raised.
+    """
+    try:
+        from aliprocessing.l2.optical import aerosol_median_radius_db
+    except ImportError:
+        return
+    try:
+        aerosol_median_radius_db()
+    except Exception as e:
+        log.warning("Could not pre-warm retrieval optical database: %s", e)