grid-process 1.0.0__tar.gz → 1.1.0__tar.gz

grid_process-1.1.0/CHANGELOG.rst

@@ -0,0 +1,96 @@
+=========
+Changelog
+=========
+
+1.1.0 (2026-11-12)
+------------------
+
+**Changes of results** (compared to 1.0.0, see the entries below for details)
+
+* ``interp2poi`` and ``interp2grid`` (outside of the regular grid interpolation) search nearest neighbors and
+  triangulate in meters instead of latitude/longitude degrees: neighbors, IDW weights and ``linear``/``cubic``
+  results change slightly for grids that were searched in degrees, including native LV95 grids with
+  latitude/longitude coordinates
+* grids in rotated pole or latitude/longitude on a sphere without datum (e.g. a CF grid mapping with
+  ``earth_radius``) now get the datum shift to Swiss coordinates (about 180 m)
+* wrong values or NaN are fixed for grids with integer coordinates and for latitude/longitude grids whose longitude
+  convention (0..360 or -180..180) differs from the one of the POIs or destination grid
+* ``create_dst_grid`` (with ``topright`` and ``resolution``) and ``create_dst_grid_from_src`` return different
+  destination grids: exact resolution, and the extent of the source domain reprojected in the destination CRS
+* ``grid_interp._get_buffer`` is renamed to ``get_buffer`` (now public)
+
+**Changes**
+
+* added a ``py.typed`` marker so that type checkers use the inline type hints of ``grid-process`` (PEP 561)
+* added example notebooks to the documentation illustrating ``interp2poi`` and ``interp2grid`` with open data, with
+  their dependencies in the optional ``example`` dependency group. Two of them are global: GFS 2 m temperature to
+  stations worldwide (compared with GHCN-hourly observations) and ERA5 2 m temperature to the IFS 0.1° grid
+* fixed ``interp2grid`` and ``interp2poi`` truncating reprojected coordinates when the grid coordinates are
+  integers (e.g. an LV95 destination grid built with ``np.arange``), which gave wrong values or NaN
+* fixed ``create_dst_grid`` with ``topright`` and ``resolution``: the resolution is now exact and ``topright``
+  a closed upper bound, instead of the grid spacing being stretched to end exactly on ``topright``
+* fixed ``interp2poi`` and ``interp2grid`` (outside of the regular grid interpolation) searching nearest neighbors
+  and triangulating (``linear``, ``cubic``) in latitude/longitude degrees, where a degree of longitude is shorter
+  than a degree of latitude (about 0.68 times in Switzerland). Distances are now always in meters: native
+  coordinates projected in meters are used even if latitude/longitude coordinates are present (e.g. LV95 grids),
+  and latitude/longitude grids are converted to Swiss LV95, or to earth-centered earth-fixed coordinates (ECEF) far
+  from Switzerland. Neighbors, IDW weights and ``linear``/``cubic`` results change slightly for grids that were
+  searched in latitude/longitude, including native LV95 grids with latitude/longitude coordinates
+* fixed the reprojection of grids in rotated pole (or latitude/longitude) on a sphere without datum, e.g. from a CF
+  grid mapping with ``earth_radius``: their latitudes and longitudes are now considered as WGS84 ones, as their
+  latitude/longitude coordinates usually are. The datum shift to Swiss coordinates (about 180 m) was left out.
+  ``parse_grid_mapping`` (and ``get_grid_crs``) now return such CF mappings directly on WGS84, without
+  ``pyproj.CRS.from_cf``, which took about 0.3 s for a rotated pole; other mappings parsed by ``from_cf`` are cached
+* added the support of global and other non-Swiss latitude/longitude grids by ``interp2poi`` and ``interp2grid``
+  (except ``linear``, ``linear_3D`` and ``cubic``, which raise an error), with nearest neighbor searches in ECEF
+  coordinates. The maximum distance of the destination to the LV95 central northing for using LV95 is set by the
+  new setting ``lv95_max_offset`` (environment variable ``GRID_PROCESS__lv95_max_offset``, default 700 km)
+* changed the methods with altitude, which no longer reproject to Swiss coordinates when the source grid or the
+  destination has another CRS projected in meters (e.g. UTM)
+* added an explicit error when the destination grid of ``interp2grid`` is unstructured, which is not supported
+* fixed ``interp2poi`` and ``interp2grid`` with latitude/longitude source grids whose longitude convention
+  (0..360 or -180..180) differs from the one of the POIs or destination grid: the source was sliced to the
+  wrong longitudes (wrong values in the generic path, e.g. for a global destination grid) and the regular grid
+  interpolation (``try_reg=True``) returned NaN. Global latitude/longitude grids are also interpolated across
+  their 360 degree seam. When slicing a latitude/longitude source grid, the longitude buffer is widened by
+  1 / cos(latitude) (as nearest neighbors are now searched in meters) and all longitudes are kept close to the
+  poles
+* changed the ``temp`` method of ``interp2poi``: POIs whose nearest grid points all have the same altitude (e.g.
+  close to the poles of a latitude/longitude grid, whose pole rows repeat the same point, or on flat orography)
+  use the fixed lapse rate fallback (as POIs far above or below their neighborhood) instead of raising an error
+* fixed ``create_dst_grid_from_src``: the destination grid now spans exactly the bounding box of the source domain
+  reprojected in the destination CRS (from native coordinates projected in meters when available), instead of the
+  reprojected corners of its latitude/longitude bounding box, which could be much larger (e.g. +80 km for ICON-CH2
+  in LV95) or miss parts of the domain (e.g. 28 km for IFS in LV95)
+* fixed ``set_grid_mapping`` with CF grid mappings holding numpy arrays or non-float64 numpy values (e.g. a
+  ``standard_parallel`` array read from a netCDF file), which raised a ``TypeError``
+* changed ``memoize``: the least recently used result is evicted (instead of the oldest one), the cache key can be
+  computed by an optional ``key`` function (only when caching is active), a cache size of 0 deactivates caching
+  without overhead, and the cache is safe to use from several threads, also when the process forks (the fork waits
+  until no thread is using the cache). The ``hkey`` keyword argument still works
+* fixed the in-memory caches of the POI and grid reprojections, whose keys left out the source CRS, and the cache
+  of the regular grid nearest interpolation, which failed with a POI dimension name other than ``poi``
+  (``GRID_PROCESS__data_model__poi__dim``)
+* documented the in-memory caches of ``interp2poi`` and ``interp2grid`` and how to set their sizes or deactivate
+  them with the environment variables ``GRID_PROCESS__cache_size__<name>`` (Usage page)
+* changed ``interp2poi`` and ``interp2grid`` outside of the regular grid interpolation to flatten structured
+  grids with numpy instead of ``xarray.stack``, and ``interp2poi`` to look for the grid points with NaN in all
+  fields (excluded, as before) only among those with NaN in the middle field. Later calls with cached neighbors are
+  faster on large grids (e.g. 0.13 s to 0.05 s for a global 0.25 degree grid to 33 POIs, 1.6 s to 0.5 s from this
+  grid to a global 0.1 degree grid), with identical results
+* fixed ``interp2poi`` raising an ``IndexError`` when all POIs are outside an unstructured grid (no grid point left
+  after slicing): they now get NaN, as outside of the regular grid interpolation
+* made ``grid_interp.get_buffer`` public (renamed from ``_get_buffer``): the slicing buffer in grid points of an
+  interpolation method and its arguments, e.g. to slice the input data with ``slice_boundingbox`` beforehand
+
+1.0.0
+-----
+
+First open-source release of ``grid-process``, the continuation of the internal MeteoSwiss library
+``gridefix-process`` (last internal version: 6.1.0).
+
+**Breaking Change** (compared to ``gridefix-process`` 6.1.0)
+
+* renamed the package from ``gridefix-process`` to ``grid-process`` (import name ``grid_process``)
+* renamed the environment variable prefix from ``GRIDEFIX_PROCESS__`` to ``GRID_PROCESS__``
+* versioning restarts at ``1.0.0``

{grid_process-1.0.0 → grid_process-1.1.0}/LICENSE

@@ -1,7 +1,6 @@
 BSD 3-Clause License
 
 Copyright (c) 2026, MeteoSwiss
-Authors: Mathieu Schaer <mathieu.schaer@meteoswiss.ch>
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:

{grid_process-1.0.0 → grid_process-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: grid-process
-Version: 1.0.0
+Version: 1.1.0
 Summary: Weather forecast oriented Python library to process xarray Datasets
 License-Expression: BSD-3-Clause
 License-File: AUTHORS
@@ -138,7 +138,7 @@ Migrating from gridefix-process
 * Package and import name: ``gridefix-process`` / ``gridefix_process`` became ``grid-process`` / ``grid_process``.
 * Environment variable prefix: ``GRIDEFIX_PROCESS__`` became ``GRID_PROCESS__``
   (e.g. ``GRID_PROCESS__data_model__validtime``).
-* Versioning restarts at ``1.0.0``; the last internal release was ``gridefix-process`` 6.0.1.
+* Versioning restarts at ``1.0.0``; the last internal release was ``gridefix-process`` 6.1.0.
 
 Release the Project
 -------------------

{grid_process-1.0.0 → grid_process-1.1.0}/README.rst

@@ -106,7 +106,7 @@ Migrating from gridefix-process
 * Package and import name: ``gridefix-process`` / ``gridefix_process`` became ``grid-process`` / ``grid_process``.
 * Environment variable prefix: ``GRIDEFIX_PROCESS__`` became ``GRID_PROCESS__``
   (e.g. ``GRID_PROCESS__data_model__validtime``).
-* Versioning restarts at ``1.0.0``; the last internal release was ``gridefix-process`` 6.0.1.
+* Versioning restarts at ``1.0.0``; the last internal release was ``gridefix-process`` 6.1.0.
 
 Release the Project
 -------------------

{grid_process-1.0.0 → grid_process-1.1.0}/grid_process/config/grid_process.yaml

@@ -1,9 +1,15 @@
+# number of results kept by each in-memory cache (least recently used evicted), 0 to deactivate it. Override with
+# the environment variables GRID_PROCESS__cache_size__<name>, set before importing grid_process
 cache_size:
   nn_lookup: 10
   nn_lookup_regular: 10
   reproject_poi: 6
   reproject_grid: 6
 
+# [m] max |N_LV95 - 1 200 000| of the destination for interpolating in LV95 when both grids are
+# in lat/lon (otherwise ECEF). 700 km ≈ 0.6 % scale error, covers ICON-CH1/CH2 (523 km)
+lv95_max_offset: 700000
+
 data_model: # supported names
   grid:
     unstructured_dim: ["cell", "points"]

{grid_process-1.0.0 → grid_process-1.1.0}/grid_process/config/library_settings.py

@@ -55,6 +55,7 @@ class LibrarySettings(BaseSettings):
     )
     cache_size: CacheSizeSettings
     data_model: DataModelSettings
+    lv95_max_offset: float = Field(gt=0)
 
     @classmethod
     def settings_customise_sources(

{grid_process-1.0.0 → grid_process-1.1.0}/grid_process/data_model.py

@@ -10,12 +10,15 @@ Module with utility functions to deal with the numerous supported data models
 
 import logging
 import json
+from functools import lru_cache
 from typing import Any, TypeVar, Literal, overload
 from collections.abc import Callable
 
 import numpy as np
 from numpy.typing import NDArray
 from pyproj import CRS
+from pyproj.crs import DerivedGeographicCRS
+from pyproj.crs.coordinate_operation import PoleRotationNetCDFCFConversion
 import geopandas as gpd
 import xarray as xr
 
@@ -203,12 +206,6 @@ def set_grid_mapping(ds: xr.Dataset) -> xr.Dataset:
     data variable encoding the grid mapping is removed.
     """
 
-    def serializable_mapping() -> dict[str, Any]:
-        return {
-            key: float(value) if isinstance(value, np.floating) else value
-            for key, value in ds_out[mapname].attrs.items()
-        }
-
     ds_out = ds.copy(deep=False)
     grid_map = set()
     for varname, da in ds_out.items():
@@ -224,7 +221,7 @@ def set_grid_mapping(ds: xr.Dataset) -> xr.Dataset:
 
     mapname = grid_map.pop()
     if {"grid_mapping", "crs"}.isdisjoint(ds_out.attrs) and mapname in ds_out:
-        ds_out.attrs["grid_mapping"] = json.dumps(serializable_mapping())
+        ds_out.attrs["grid_mapping"] = _grid_mapping_json(ds_out[mapname].attrs)
 
     return ds_out.drop_vars(mapname, errors="ignore")
 
@@ -452,23 +449,106 @@ def parse_grid_mapping(grid_mapping: str | dict[str, Any]) -> CRS:
     See also
     --------
     :func:`get_grid_crs`
+
+    Note
+    ----
+    CF ``latitude_longitude`` and ``rotated_latitude_longitude`` mappings without ellipsoid parameters other than
+    ``earth_radius`` (as NWP models on a sphere) are built directly on WGS84, which is much faster than
+    pyproj.CRS.from_cf(). The result is the same as ``sphere_as_wgs84(pyproj.CRS.from_cf(grid_mapping))``, see
+    :func:`~grid_process.helpers.sphere_as_wgs84`.
     """
 
+    # pyproj.CRS.from_cf() takes ~0.3 s for mappings with ellipsoid parameters (custom datum): cached on the JSON
     if isinstance(grid_mapping, str):
-        grid_mapping = json.loads(grid_mapping)
+        return _parse_grid_mapping_str(grid_mapping)
+    try:
+        return _parse_grid_mapping_str(_grid_mapping_json(grid_mapping))
+    except TypeError:  # not serializable: parse without cache
+        return _parse_grid_mapping_dict(grid_mapping)
+
+
+def _grid_mapping_json(grid_mapping: dict[str, Any]) -> str:
+    """
+    JSON string of a grid mapping dictionary, with sorted keys and numpy scalars and arrays (e.g. attributes read
+    from a netCDF file such as "standard_parallel") converted to numbers and lists.
+    """
+
+    return json.dumps(grid_mapping, sort_keys=True, default=lambda value: value.tolist())
+
+
+@lru_cache(maxsize=32)
+def _parse_grid_mapping_str(grid_mapping: str) -> CRS:
+    """
+    Cached parsing of a JSON "grid_mapping" attribute, see :func:`parse_grid_mapping`.
+    """
+
+    return _parse_grid_mapping_dict(json.loads(grid_mapping))
+
+
+def _parse_grid_mapping_dict(grid_mapping: dict[str, Any]) -> CRS:
+    """
+    Parsing of a "grid_mapping" dictionary, see :func:`parse_grid_mapping`.
+    """
+
     if "epsg_code" in grid_mapping:
-        return CRS.from_string(grid_mapping["epsg_code"])  # type: ignore[index]
+        return CRS.from_string(grid_mapping["epsg_code"])
+
+    if _ELLIPSOID_KEYS.isdisjoint(grid_mapping):
+        mapping_name = grid_mapping.get("grid_mapping_name")
+        if mapping_name == "latitude_longitude":
+            return CRS.from_epsg(4326)
+        if mapping_name == "rotated_latitude_longitude":
+            return DerivedGeographicCRS(
+                base_crs=CRS.from_epsg(4326),
+                conversion=PoleRotationNetCDFCFConversion(
+                    grid_north_pole_latitude=grid_mapping["grid_north_pole_latitude"],
+                    grid_north_pole_longitude=grid_mapping["grid_north_pole_longitude"],
+                    north_pole_grid_longitude=grid_mapping.get("north_pole_grid_longitude", 0.0),
+                ),
+            )
+
+    return CRS.from_cf(grid_mapping)
+
+
+# CF grid mapping attributes defining the ellipsoid or datum (other than earth_radius for a sphere)
+_ELLIPSOID_KEYS = frozenset(
+    (
+        "semi_major_axis",
+        "semi_minor_axis",
+        "inverse_flattening",
+        "reference_ellipsoid_name",
+        "horizontal_datum_name",
+        "geographic_crs_name",
+        "prime_meridian_name",
+        "longitude_of_prime_meridian",
+        "towgs84",
+        "crs_wkt",
+        "spatial_ref",  # WKT written by GDAL
+    )
+)
+
+
+def is_metric_crs(crs: CRS) -> bool:
+    """
+    Whether ``crs`` is a projected CRS with all axes in meters (e.g. LV95, LV03 or UTM), i.e. in which euclidean
+    distances can be computed directly from the coordinates. The axis units are compared by their conversion factor
+    to the meter, so that units written with another name (e.g. "m" or "Meter" in a WKT without authority) are
+    recognized, while kilometers or feet are not.
+    """
 
-    return CRS.from_cf(grid_mapping)  # type: ignore[arg-type]
+    return bool(crs.is_projected) and all(axis.unit_conversion_factor == 1.0 for axis in crs.axis_info)
 
 
 def get_coords_name_crs(
-    grid: xr.Dataset | xr.DataArray, spatial_dims: tuple[str, ...] | None = None
+    grid: xr.Dataset | xr.DataArray,
+    spatial_dims: tuple[str, ...] | None = None,
+    prefer_projected: bool = False,
 ) -> tuple[str, str, CRS]:
     """
     Get correct coordinate names and CRS according to supported conventions. When
     non-dimensional longitude and latitude coordinates arrays are present, these
-    are used instead of the native spatial dimension coordinates.
+    are used instead of the native spatial dimension coordinates, unless ``prefer_projected``
+    is True and the native CRS is projected with axes in meters.
 
     Parameters
     ----------
@@ -480,6 +560,10 @@ def get_coords_name_crs(
         name of the ``grid`` spatial dimension(s). Has size one in case of
         unstructured grid otherwise contains dimension names in the order (y, x).
         If not provided, calls :func:`~grid_process.helpers.get_spatial_dims`
+    prefer_projected : bool, optional
+        If True, return the native spatial coordinates when their CRS is projected with axes in meters (e.g. LV95),
+        even if longitude and latitude coordinates are also present. Native coordinates in degrees (e.g. rotated
+        pole) still give the longitude and latitude coordinates when present. Default is False.
 
     Returns
     -------
@@ -491,6 +575,13 @@ def get_coords_name_crs(
         pyproj representation of input grid coordinate reference system
     """
 
+    if prefer_projected:
+        if spatial_dims is None:
+            spatial_dims = get_spatial_dims(grid)
+        native = _get_native_metric_coords(grid, spatial_dims)
+        if native is not None:
+            return native
+
     latlon = get_latlon_coords(grid)
     if latlon is not None:
         grid_y, grid_x = latlon
@@ -510,6 +601,35 @@ def get_coords_name_crs(
     raise ValueError("Missing spatial coordinates")
 
 
+def _get_native_metric_coords(
+    grid: xr.Dataset | xr.DataArray, spatial_dims: tuple[str, ...]
+) -> tuple[str, str, CRS] | None:
+    """
+    Names of the native (y, x) spatial coordinates and their CRS if this CRS is projected with axes in meters,
+    otherwise None. Used by :func:`get_coords_name_crs` with ``prefer_projected=True``.
+    """
+
+    if len(spatial_dims) == 1:
+        try:
+            grid_y, grid_x = get_spatial_coords(grid)
+        except ValueError:
+            return None
+    else:
+        grid_y, grid_x = spatial_dims
+        if grid_y not in grid.coords or grid_x not in grid.coords:
+            return None
+
+    if (grid_y, grid_x) == get_latlon_coords(grid, nondim=False):
+        return None
+
+    try:
+        crs = get_grid_crs(grid)
+    except ValueError:  # no CRS attribute: only longitude and latitude can be used
+        return None
+
+    return (grid_y, grid_x, crs) if is_metric_crs(crs) else None
+
+
 def is_decreasing1d(coords: xr.DataArray) -> bool:
     """
     Whether 1 dimensional ``coords`` are in decreasing order.