h5netcdf 1.6.4__tar.gz → 1.7.1__tar.gz

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.6.0
+    rev: v6.0.0
     hooks:
       - id: trailing-whitespace
       - id: end-of-file-fixer
@@ -10,16 +10,16 @@ repos:
       - id: debug-statements
       - id: mixed-line-ending
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.5.0'
+    rev: 'v0.12.12'
     hooks:
       - id: ruff
         args: [ "--fix" ]
   - repo: https://github.com/psf/black
-    rev: 24.4.2
+    rev: 25.1.0
     hooks:
       - id: black
   - repo: https://github.com/adamchainz/blacken-docs
-    rev: "1.18.0"
+    rev: "1.20.0"
     hooks:
       - id: blacken-docs
         additional_dependencies:

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/CHANGELOG.rst

@@ -1,9 +1,28 @@
 Change Log
 ----------
 
+Version 1.7.1 (October 16th, 2025):
+
+- Fix regression where attributes with list of strings were written with h5py low-level API instead of high-level API (:issue:`291`, :pull:`292`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_
+
+Version 1.7.0 (October 15th, 2025):
+
+- Fix unintentional changes in test suite (:pull:`277`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_
+- Create ENUM with low level API to keep order-by-value, add h5dump based tests (:pull:`285`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_ and `David Huard <https://github.com/huard>`_
+- Fix interoperability issues between netcdf4/h5netcdf, namely resizing variables using partial slices,
+creating string attributes with NULLTERM, proper attachment of _Netcdf4Coordinates and _Netcdf4Dimid as well as special string type fillvalues (:pull:`286`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_ and `David Huard <https://github.com/huard>`_
+- Add the `format` argument to `h5netcdf.File` and support for the `NETCDF4_CLASSIC` format (:issue:`280`, :pull:`283`).
+  By `David Huard <https://github.com/huard>`_
+- Do not return padded arrays for slices larger than variable shape (:issue:`287`, :pull:`288`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_
+
 Version 1.6.4 (August 5th, 2025):
 
-- Cleanup: pyupgrade --py39-plus
+- Cleanup: pyupgrade --py39-plus (:pull:`272`).
   By `Kurt Schwehr <https://github.com/schwehr>`_
 - Add better error messages when operating on a closed file (:issue:`274`, :pull:`275`).
   By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: h5netcdf
-Version: 1.6.4
+Version: 1.7.1
 Summary: netCDF4 via h5py
 Author-email: Stephan Hoyer <shoyer@gmail.com>, Kai Mühlbauer <kmuehlbauer@wradlib.org>
 Maintainer-email: h5netcdf developers <devteam@h5netcdf.org>
@@ -124,10 +124,10 @@ Usage
 -----
 
 h5netcdf has two APIs, a new API and a legacy API. Both interfaces currently
-reproduce most of the features of the netCDF interface, with the notable
-exception of support for operations that rename or delete existing objects.
-We simply haven't gotten around to implementing this yet. Patches
-would be very welcome.
+reproduce most of the features of the netCDF interface, including the ability
+to write NETCDF4 and NETCDF4_CLASSIC formatted files. Support for operations
+that rename or delete existing objects is still missing, and patches would be
+very welcome.
 
 New API
 ~~~~~~~

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/README.rst

@@ -64,10 +64,10 @@ Usage
 -----
 
 h5netcdf has two APIs, a new API and a legacy API. Both interfaces currently
-reproduce most of the features of the netCDF interface, with the notable
-exception of support for operations that rename or delete existing objects.
-We simply haven't gotten around to implementing this yet. Patches
-would be very welcome.
+reproduce most of the features of the netCDF interface, including the ability
+to write NETCDF4 and NETCDF4_CLASSIC formatted files. Support for operations
+that rename or delete existing objects is still missing, and patches would be
+very welcome.
 
 New API
 ~~~~~~~

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/doc/devguide.rst

@@ -13,6 +13,7 @@ Contributors
 - `Aleksandar Jelenak <https://github.com/ajelenak>`_
 - `Bas Couwenberg <https://github.com/sebastic>`_.
 - `Brett Naul <https://github.com/bnaul>`_
+- `David Huard <https://github.com/huard>`_
 - `Dion Häfner <https://github.com/dionhaefner>`_
 - `Drew Parsons <https://github.com/drew-parsons>`_
 - `Ezequiel Cimadevilla Alvarez <https://github.com/zequihg50>`_
@@ -49,7 +50,7 @@ Continuous Integration
 or a PullRequest branch several checks are performed:
 
 - Lint and style checks (``ruff``, ``black``)
-- Unit tests with latest ``h5py3`` (Python 3.9, 3.10, 3.11) facilitating GitHub Ubuntu worker
+- Unit tests with latest ``h5py3`` (and Python versions) facilitating GitHub Ubuntu worker
 - Documentation build, artifacts are made available to download
 - On release, source-tarball and universal wheel is uploaded to PyPI and documentation is made available
   on `h5netcdf GitHub Pages`_

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/h5netcdf/_version.py

@@ -1,7 +1,14 @@
 # file generated by setuptools-scm
 # don't change, don't track in version control
 
-__all__ = ["__version__", "__version_tuple__", "version", "version_tuple"]
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
 
 TYPE_CHECKING = False
 if TYPE_CHECKING:
@@ -9,13 +16,19 @@ if TYPE_CHECKING:
     from typing import Union
 
     VERSION_TUPLE = Tuple[Union[int, str], ...]
+    COMMIT_ID = Union[str, None]
 else:
     VERSION_TUPLE = object
+    COMMIT_ID = object
 
 version: str
 __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
+commit_id: COMMIT_ID
+__commit_id__: COMMIT_ID
 
-__version__ = version = '1.6.4'
-__version_tuple__ = version_tuple = (1, 6, 4)
+__version__ = version = '1.7.1'
+__version_tuple__ = version_tuple = (1, 7, 1)
+
+__commit_id__ = commit_id = 'g251854355'

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/h5netcdf/attrs.py

@@ -2,6 +2,8 @@ from collections.abc import MutableMapping
 
 import numpy as np
 
+from .utils import _create_string_attribute
+
 _HIDDEN_ATTRS = frozenset(
     [
         "REFERENCE_LIST",
@@ -17,10 +19,11 @@ _HIDDEN_ATTRS = frozenset(
 
 
 class Attributes(MutableMapping):
-    def __init__(self, h5attrs, check_dtype, h5py_pckg):
+    def __init__(self, h5attrs, check_dtype, h5py_pckg, format="NETCDF4"):
         self._h5attrs = h5attrs
         self._check_dtype = check_dtype
         self._h5py = h5py_pckg
+        self._format = format
 
     def __getitem__(self, key):
         if key in _HIDDEN_ATTRS:
@@ -83,7 +86,26 @@ class Attributes(MutableMapping):
             dtype = np.asarray(value).dtype
 
         self._check_dtype(dtype)
-        self._h5attrs[key] = value
+
+        is_plain_string = dtype.kind in {"S", "U"} and dtype.metadata is None
+        if (
+            is_plain_string  # for simple strings
+            and (not isinstance(value, list) and self._format == "NETCDF4_CLASSIC")
+            and not isinstance(value, self._h5py.Empty)
+            and self._h5py.__name__ == "h5py"
+        ):
+            # create with low level API to get fixed length strings
+            # as netcdf4-python/netcdf-c does
+            _create_string_attribute(self._h5attrs._id, key, value)
+        # always for CLASSIC mode
+        elif self._format == "NETCDF4_CLASSIC":
+            self._h5attrs[key] = np.atleast_1d(value)
+        else:
+            # netcdf4-python/netcdf-c writes non-string scalars as simple dataset
+            # converting to 1D
+            if np.isscalar(value) and dtype.kind not in {"S", "U"}:
+                value = np.atleast_1d(value)
+            self._h5attrs[key] = value
 
     def __delitem__(self, key):
         del self._h5attrs[key]

{h5netcdf-1.6.4 → h5netcdf-1.7.1}/h5netcdf/core.py

@@ -13,7 +13,15 @@ from packaging import version
 from . import __version__
 from .attrs import Attributes
 from .dimensions import Dimension, Dimensions
-from .utils import Frozen
+from .utils import (
+    CompatibilityError,
+    Frozen,
+    _commit_enum_type,
+    _create_classic_string_dataset,
+    _create_enum_dataset,
+    _create_enum_dataset_attribute,
+    _create_string_attribute,
+)
 
 try:
     import h5pyd
@@ -36,10 +44,6 @@ def _name_from_dimension(dim):
     return dim[0].name.split("/")[-1]
 
 
-class CompatibilityError(Exception):
-    """Raised when using features that are not part of the NetCDF4 API."""
-
-
 def _invalid_netcdf_feature(feature, allow):
     if not allow:
         msg = (
@@ -348,7 +352,9 @@ class BaseVariable(BaseObject):
             [self._parent._all_dimensions[d]._dimid for d in dims],
             "int32",
         )
-        if len(coord_ids) > 1:
+        # add _Netcdf4Coordinates for multi-dimensional coordinate variables
+        # or for (one-dimensional) coordinates
+        if len(coord_ids) >= 1:
             self._h5ds.attrs["_Netcdf4Coordinates"] = coord_ids
 
     def _ensure_dim_id(self):
@@ -361,12 +367,21 @@ class BaseVariable(BaseObject):
                 self._h5ds.attrs["_Netcdf4Dimid"] = dim.attrs["_Netcdf4Dimid"]
 
     def _maybe_resize_dimensions(self, key, value):
-        """Resize according to given (expanded) key with respect to variable dimensions"""
+        """Resize according to given (expanded) key with respect to variable dimensions.
+
+        Parameters
+        ----------
+        key : Tuple[slice]
+            Indexing key
+        value : array-like
+            Values to be written.
+        """
         new_shape = ()
-        v = None
+        v = np.asarray(value)
         for i, dim in enumerate(self.dimensions):
             # is unlimited dimensions (check in all dimensions)
             if self._parent._all_dimensions[dim].isunlimited():
+                current_dim_size = len(self._parent._all_dimensions[dim])
                 if key[i].stop is None:
                     # if stop is None, get dimensions from value,
                     # they must match with variable dimension
@@ -375,16 +390,25 @@ class BaseVariable(BaseObject):
                     if v.ndim == self.ndim:
                         new_max = max(v.shape[i], self._h5ds.shape[i])
                     elif v.ndim == 0:
-                        # for scalars we take the current dimension size (check in all dimensions
+                        # for scalar values we take the current dimension size
+                        # (check in all dimensions)
                         new_max = self._parent._all_dimensions[dim].size
+                        # but for compatibility with netcdf4-python/netcdf-c
+                        # we set at least 1
+                        if new_max == 0:
+                            new_max = 1
                     else:
                         raise IndexError("shape of data does not conform to slice")
+                # if slice stop is negative, we need to check the value size
+                elif key[i].stop < 0:
+                    new_max = v.shape[i] - key[i].stop
                 else:
                     new_max = max(key[i].stop, self._h5ds.shape[i])
                 # resize unlimited dimension if needed but no other variables
                 # this is in line with `netcdf4-python` which only resizes
                 # the dimension and this variable
-                if self._parent._all_dimensions[dim].size < new_max:
+                # todo: check above assumptions with latest netcdf4-python/netcdf-c
+                if current_dim_size < new_max and self.name == dim:
                     self._parent.resize_dimension(dim, new_max)
                 new_shape += (new_max,)
             else:
@@ -412,8 +436,19 @@ class BaseVariable(BaseObject):
                 string_info
                 and string_info.length is not None
                 and string_info.length > 1
-            ) or enum_info:
+            ):
+                # fixed length string
                 value = fillvalue
+            elif string_info and string_info.length is None:
+                # variable length string
+                value = fillvalue
+            elif enum_info:
+                value = fillvalue
+                if self._root._h5py.__name__ == "h5py":
+                    _create_enum_dataset_attribute(
+                        self, "_FillValue", value, self.datatype
+                    )
+                    return
             else:
                 value = self.dtype.type(fillvalue)
 
@@ -499,16 +534,17 @@ class BaseVariable(BaseObject):
                 for k in key0
             ]
             # second convert to max shape
+            # we take the minimum of shape vs max_index to not return
+            # slices larger than expected data
             max_shape = tuple(
-                [
-                    shape[i] if k is None else max(h5ds_shape[i], k)
-                    for i, k in enumerate(max_index)
-                ]
+                s if k is None else min(s, k) for s, k in zip(shape, max_index)
             )
 
             # check if hdf5 dataset dimensions are smaller than
             # their respective netcdf dimensions
             sdiff = [d0 - d1 for d0, d1 in zip(max_shape, h5ds_shape)]
+            # set negative values to zero
+            sdiff = np.maximum(sdiff, 0)
             # create padding only if hdf5 dataset is smaller than netcdf dimension
             if sum(sdiff):
                 padding = [(0, s) for s in sdiff]
@@ -583,13 +619,28 @@ class BaseVariable(BaseObject):
         ):
             self._h5ds[key] = value.view(view)
         else:
-            self._h5ds[key] = value
+            # write with low-level API for CLASSIC format
+            if (
+                self._root._format == "NETCDF4_CLASSIC"
+                and self.dtype.kind in ["S", "U"]
+                and self._root._h5py.__name__ == "h5py"
+            ):
+                # h5py expects np.ndarray
+                value = np.asanyarray(value)
+                self._h5ds.id.write(
+                    h5py.h5s.ALL, h5py.h5s.ALL, value, mtype=self._h5ds.id.get_type()
+                )
+            else:
+                self._h5ds[key] = value
 
     @property
     def attrs(self):
         """Return variable attributes."""
         return Attributes(
-            self._h5ds.attrs, self._root._check_valid_netcdf_dtype, self._root._h5py
+            self._h5ds.attrs,
+            self._root._check_valid_netcdf_dtype,
+            self._root._h5py,
+            format=self._root._format,
         )
 
     _cls_name = "h5netcdf.Variable"
@@ -979,6 +1030,13 @@ class Group(Mapping):
 
     @dimensions.setter
     def dimensions(self, value):
+        if self._format == "NETCDF4_CLASSIC":
+            unlimited_dims = list(filter(lambda s: s in [None, 0], value.values()))
+            if len(unlimited_dims) > 1:
+                raise CompatibilityError(
+                    "NETCDF4_CLASSIC format only allows one unlimited dimension."
+                )
+
         for k, v in self._all_dimensions.maps[0].items():
             if k in value:
                 if v != value[k]:
@@ -1104,8 +1162,10 @@ class Group(Mapping):
         # dimension scale without a corresponding variable.
         # Keep the references, to re-attach later
         refs = None
+        dimid = None
         if h5name in self._dimensions and h5name in self._h5group:
             refs = self._dimensions[name]._scale_refs
+            dimid = self._dimensions[name]._h5ds.attrs.get("_Netcdf4Dimid", None)
             self._dimensions[name]._detach_scale()
             del self._h5group[name]
 
@@ -1115,15 +1175,28 @@ class Group(Mapping):
         fillvalue, h5fillvalue = _check_fillvalue(self, fillvalue, dtype)
 
         # create hdf5 variable
-        self._h5group.create_dataset(
-            h5name,
-            shape,
-            dtype=dtype,
-            data=data,
-            chunks=chunks,
-            fillvalue=h5fillvalue,
-            **kwargs,
-        )
+        # for classic format string types write with low level API
+        if (
+            self._root._format == "NETCDF4_CLASSIC"
+            and np.dtype(dtype).kind in ["S", "U"]
+            and self._root._h5py.__name__ == "h5py"
+        ):
+            _create_classic_string_dataset(
+                self._h5group._id, h5name, data, shape, chunks
+            )
+        elif self._root._h5py.__name__ == "h5py" and isinstance(dtype, EnumType):
+            # use low level API for creating ENUMS
+            _create_enum_dataset(self, h5name, shape, dtype, h5fillvalue)
+        else:
+            self._h5group.create_dataset(
+                h5name,
+                shape,
+                dtype=dtype,
+                data=data,
+                chunks=chunks,
+                fillvalue=h5fillvalue,
+                **kwargs,
+            )
 
         # create variable class instance
         variable = self._variable_cls(self, h5name, dimensions)
@@ -1135,9 +1208,12 @@ class Group(Mapping):
 
         # Re-create dim-scale and re-attach references to coordinate variable.
         if name in self._all_dimensions and h5name in self._h5group:
-            self._all_dimensions[name]._create_scale()
+            if dimid is not None:
+                self._all_dimensions[name]._create_scale(dimid=dimid)
             if refs is not None:
                 self._all_dimensions[name]._attach_scale(refs)
+            # re-attach coords for dimension scales
+            variable._attach_coords()
 
         # In case of data variables attach dim_scales and coords.
         if name in self.variables and h5name not in self._dimensions:
@@ -1146,9 +1222,12 @@ class Group(Mapping):
 
         # This is a bit of a hack, netCDF4 attaches _Netcdf4Dimid to every variable
         # when a variable is first written to, after variable creation.
-        # Here we just attach it to every variable on creation.
-        # Todo: get this consistent with netcdf-c/netcdf4-python
-        variable._ensure_dim_id()
+        # Last known behaviour since netcdf4-python 1.7.2 and netcdf-c 4.9.2
+        if (None in maxshape and maxshape[0] is not None) or (
+            None not in maxshape
+            and len(variable._h5ds.attrs.get("_Netcdf4Coordinates", [])) >= 1
+        ):
+            variable._ensure_dim_id()
 
         # add fillvalue attribute to variable
         if fillvalue is not None:
@@ -1211,7 +1290,6 @@ class Group(Mapping):
         var : h5netcdf.Variable
             Variable class instance
         """
-
         # if root-variable
         if name.startswith("/"):
             # handling default fillvalues for legacyapi
@@ -1220,6 +1298,12 @@ class Group(Mapping):
 
             if fillvalue is None and isinstance(self._parent._root, Dataset):
                 fillvalue = _get_default_fillvalue(dtype)
+
+            if self._root._format == "NETCDF4_CLASSIC" and len(dimensions) == 0:
+                raise CompatibilityError(
+                    "NETCDF4_CLASSIC format does not allow variables without dimensions."
+                )
+
             return self._root.create_variable(
                 name[1:],
                 dimensions,
@@ -1343,7 +1427,10 @@ class Group(Mapping):
     @property
     def attrs(self):
         return Attributes(
-            self._h5group.attrs, self._root._check_valid_netcdf_dtype, self._root._h5py
+            self._h5group.attrs,
+            self._root._check_valid_netcdf_dtype,
+            self._root._h5py,
+            format=self._root._format,
         )
 
     _cls_name = "h5netcdf.Group"
@@ -1397,8 +1484,14 @@ class Group(Mapping):
         enum_dict: dict
             A Python dictionary containing the Enum field/value pairs.
         """
-        et = self._root._h5py.enum_dtype(enum_dict, basetype=datatype)
-        self._h5group[datatype_name] = et
+        # to correspond with netcdf4-python/netcdf-c we need to create
+        # with low level API, to keep enums ordered by value
+        # works only for h5py
+        if self._root._h5py.__name__ == "h5py":
+            _commit_enum_type(self, datatype_name, enum_dict, datatype)
+        else:
+            et = self._root._h5py.enum_dtype(enum_dict, basetype=datatype)
+            self._h5group[datatype_name] = et
         # create enumtype class instance
         enumtype = self._enumtype_cls(self, datatype_name)
         self._enumtypes[datatype_name] = enumtype
@@ -1442,7 +1535,15 @@ class Group(Mapping):
 
 
 class File(Group):
-    def __init__(self, path, mode="r", invalid_netcdf=False, phony_dims=None, **kwargs):
+    def __init__(
+        self,
+        path,
+        mode="r",
+        format="NETCDF4",
+        invalid_netcdf=False,
+        phony_dims=None,
+        **kwargs,
+    ):
         """NetCDF4 file constructor.
 
         Parameters
@@ -1454,6 +1555,10 @@ class File(Group):
         mode: "r", "r+", "a", "w"
             A valid file access mode. Defaults to "r".
 
+        format: "NETCDF4", "NETCDF4_CLASSIC"
+            The format of the file to create. Only relevant when creating a new
+            file (mode "w"). Defaults to "NETCDF4".
+
         invalid_netcdf: bool
             Allow writing netCDF4 with data types and attributes that would
             otherwise not generate netCDF4 files that can be read by other
@@ -1570,8 +1675,16 @@ class File(Group):
         else:
             self._closed = False
 
+        if self._preexisting_file:
+            format = (
+                "NETCDF4_CLASSIC"
+                if self._h5file.attrs.get("_nc3_strict")
+                else "NETCDF4"
+            )
+
         self._filename = self._h5file.filename
         self._mode = mode
+        self._format = format
         self._writable = mode != "r"
         self._root_ref = weakref.ref(self)
         self._h5path = "/"
@@ -1588,6 +1701,9 @@ class File(Group):
                     "phony_dims='access' for per access naming."
                 )
 
+        if format not in ["NETCDF4", "NETCDF4_CLASSIC"]:
+            raise ValueError(f"unknown format {format!r}")
+
         # string decoding
         if "legacy" in self._cls_name:
             if self.decode_vlen_strings is not None:
@@ -1638,6 +1754,11 @@ class File(Group):
             description = "boolean"
         elif self._h5py.check_dtype(ref=dtype) is not None:
             description = "reference"
+        elif (
+            dtype in [int, np.int64, np.uint64, np.uint32, np.uint16, np.uint8]
+            and self._format == "NETCDF4_CLASSIC"
+        ):
+            description = f"{dtype} (CLASSIC)"
         else:
             description = None
 
@@ -1659,6 +1780,10 @@ class File(Group):
     def parent(self):
         return None
 
+    @property
+    def data_model(self):
+        return self._format
+
     @property
     def _root(self):
         return self
@@ -1672,12 +1797,20 @@ class File(Group):
                     f"hdf5={self._h5py.version.hdf5_version},"
                     f"{self._h5py.__name__}={self._h5py.__version__}"
                 )
-                self.attrs._h5attrs["_NCProperties"] = np.array(
-                    _NC_PROPERTIES,
-                    dtype=self._h5py.string_dtype(
-                        encoding="ascii", length=len(_NC_PROPERTIES)
-                    ),
-                )
+                if self._format == "NETCDF4_CLASSIC" and self._h5py.__name__ == "h5py":
+                    _create_string_attribute(
+                        self.attrs._h5attrs._id, "_NCProperties", _NC_PROPERTIES
+                    )
+                else:
+                    self.attrs._h5attrs["_NCProperties"] = np.array(
+                        _NC_PROPERTIES,
+                        dtype=self._h5py.string_dtype(
+                            encoding="ascii", length=len(_NC_PROPERTIES)
+                        ),
+                    )
+                if self._format == "NETCDF4_CLASSIC":
+                    self.attrs._h5attrs["_nc3_strict"] = np.array(1, np.int32)
+
             if self.invalid_netcdf:
                 # see https://github.com/h5netcdf/h5netcdf/issues/165
                 # warn user if .nc file extension is used for invalid netcdf features
@@ -1692,6 +1825,8 @@ class File(Group):
                 # remove _NCProperties if invalid_netcdf if exists
                 if "_NCProperties" in self.attrs._h5attrs:
                     del self.attrs._h5attrs["_NCProperties"]
+                if "_nc3_strict" in self.attrs._h5attrs:
+                    del self.attrs._h5attrs["_nc3_strict"]
 
     sync = flush