h5yaml 0.2.0__tar.gz → 0.3.0__tar.gz

{h5yaml-0.2.0 → h5yaml-0.3.0}/.github/workflows/python-package.yml

@@ -37,4 +37,5 @@ jobs:
         flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Test with pytest
       run: |
-        pytest
+        pytest tests/test_yaml_h5.py tests/test_from_yaml.py
+        pytest tests/test_yaml_nc.py

{h5yaml-0.2.0 → h5yaml-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: h5yaml
-Version: 0.2.0
+Version: 0.3.0
 Summary: Use YAML configuration file to generate HDF5/netCDF4 formated files.
 Project-URL: Homepage, https://github.com/rmvanhees/h5_yaml
 Project-URL: Source, https://github.com/rmvanhees/h5_yaml
@@ -61,23 +61,39 @@ However, package `netCDF4` has some limitations, which `h5py` has not, for examp
 not allow variable-length variables to have a compound data-type.
 
 ## Installation
-Releases of the code, starting from version 0.1, will be made available via PyPI.
+The package `h5yaml` is available from PyPI. To install it use `pip`:
+
+> $ pip install [--user] h5yaml
+
+The module `h5yaml` requires Python3.10+ and Python modules: h5py (v3.14+), netCDF4 (v1.7+) and numpy (v2.0+).
+
+**Note**: the packages `h5py` and `netCDF4` come with their own HDF5 libraries. If these are different then they may
+collide and result in a *''HDF5 error''*.
+If this is the case then you have to install the development packages of HDF5 and netCDF4 (or compile them from source).
+And reinstall `h5py` and `netCDF4` using the commands:
+
+> $ pip uninstall h5py; pip install --no-binary=h5py h5py
+> $ pip uninstall netCDF4; pip install --no-binary=netCDF4 netCDF4
 
 ## Usage
 
 The YAML file should be structured as follows:
 
- * The top level are: 'groups', 'dimensions', 'compounds' and 'variables'
+ * The top level are: 'groups', 'dimensions', 'compounds', 'variables', 'attrs\_global' and 'attrs\_groups'.
+ * > 'attrs\_global' and 'attrs\_groups' are added in version 0.3.0
+ * The names of the attributes, groups, dimensions, compounds and viariable should be specified as PosixPaths, however:
+   * The names of groups should never start with a slash (always erlative to root);
+   * All other elements which are stored in root should also not start with a slash;
+   * But these elements require a starting slash (absolute paths) when they are stored not the root. 
  * The section 'groups' are optional, but you should provide each group you want to use
    in your file. The 'groups' section in the YAML file may look like this:
-
    ```
    groups:
      - engineering_data
      - image_attributes
      - navigation_data
-     - processing_control
      - science_data
+     - processing_control/input_data
    ```
 
  * The section 'dimensions' is obligatory, you should define the dimensions for each
@@ -133,14 +149,6 @@ The YAML file should be structured as follows:
        dark_offs: [f4, '1', dark-offset]
    ```
 
-   Alternatively, provide a list with names of YAML files which contain the definitions
-   of the compounds.
-
-   ```
-   compounds:
-     - h5_nomhk_tm.yaml
-     - h5_science_hk.yaml
-   ```
  * The 'variables' are defined by their data-type ('_dtype') and dimensions ('_dims'),
    and optionally chunk sizes ('_chunks'), compression ('_compression'), variable length
    ('_vlen'). In addition, each variable can have as many attributes as you like,
@@ -148,6 +156,16 @@ The YAML file should be structured as follows:
 
    ```
    variables:
+     /science_data/detector_images:
+       _dtype: u2
+       _dims: [number_of_images, samples_per_image]
+	   _compression: 3
+       _FillValue: 65535
+       long_name: Detector pixel values
+       coverage_content_type: image
+       units: '1'
+       valid_min: 0
+       valid_max: 65534
      /image_attributes/nr_coadditions:
        _dtype: u2
        _dims: [number_of_images]
@@ -163,20 +181,18 @@ The YAML file should be structured as follows:
        units: seconds
      stats_163:
        _dtype: stats_dtype
-       _vlen: True
        _dims: [days]
+       _vlen: True
        comment: detector map statistics (MPS=163)
    ```
 
-### Notes and ToDo:
+### Notes and ToDo
 
- * The usage of older versions of h5py may result in broken netCDF4 files
- * Explain usage of parameter '_chunks', which is currently not correctly implemented.
- * Explain that the usage of variable length data-sets may break netCDF4 compatibility
+* The layout of a HDF5 or netCDF4 file can be complex. From version 0.3.0, you can split the file definition over several YAML files and provide a list with the names of YAML files as input to H5Yaml and NcYaml.  
 
 ## Support [TBW]
 
-## Roadmap
+## Road map
 
  * Release v0.1 : stable API to read your YAML files and generate the HDF5/netCDF4 file
 

{h5yaml-0.2.0 → h5yaml-0.3.0}/README.md

@@ -29,23 +29,39 @@ However, package `netCDF4` has some limitations, which `h5py` has not, for examp
 not allow variable-length variables to have a compound data-type.
 
 ## Installation
-Releases of the code, starting from version 0.1, will be made available via PyPI.
+The package `h5yaml` is available from PyPI. To install it use `pip`:
+
+> $ pip install [--user] h5yaml
+
+The module `h5yaml` requires Python3.10+ and Python modules: h5py (v3.14+), netCDF4 (v1.7+) and numpy (v2.0+).
+
+**Note**: the packages `h5py` and `netCDF4` come with their own HDF5 libraries. If these are different then they may
+collide and result in a *''HDF5 error''*.
+If this is the case then you have to install the development packages of HDF5 and netCDF4 (or compile them from source).
+And reinstall `h5py` and `netCDF4` using the commands:
+
+> $ pip uninstall h5py; pip install --no-binary=h5py h5py
+> $ pip uninstall netCDF4; pip install --no-binary=netCDF4 netCDF4
 
 ## Usage
 
 The YAML file should be structured as follows:
 
- * The top level are: 'groups', 'dimensions', 'compounds' and 'variables'
+ * The top level are: 'groups', 'dimensions', 'compounds', 'variables', 'attrs\_global' and 'attrs\_groups'.
+ * > 'attrs\_global' and 'attrs\_groups' are added in version 0.3.0
+ * The names of the attributes, groups, dimensions, compounds and viariable should be specified as PosixPaths, however:
+   * The names of groups should never start with a slash (always erlative to root);
+   * All other elements which are stored in root should also not start with a slash;
+   * But these elements require a starting slash (absolute paths) when they are stored not the root. 
  * The section 'groups' are optional, but you should provide each group you want to use
    in your file. The 'groups' section in the YAML file may look like this:
-
    ```
    groups:
      - engineering_data
      - image_attributes
      - navigation_data
-     - processing_control
      - science_data
+     - processing_control/input_data
    ```
 
  * The section 'dimensions' is obligatory, you should define the dimensions for each
@@ -101,14 +117,6 @@ The YAML file should be structured as follows:
        dark_offs: [f4, '1', dark-offset]
    ```
 
-   Alternatively, provide a list with names of YAML files which contain the definitions
-   of the compounds.
-
-   ```
-   compounds:
-     - h5_nomhk_tm.yaml
-     - h5_science_hk.yaml
-   ```
  * The 'variables' are defined by their data-type ('_dtype') and dimensions ('_dims'),
    and optionally chunk sizes ('_chunks'), compression ('_compression'), variable length
    ('_vlen'). In addition, each variable can have as many attributes as you like,
@@ -116,6 +124,16 @@ The YAML file should be structured as follows:
 
    ```
    variables:
+     /science_data/detector_images:
+       _dtype: u2
+       _dims: [number_of_images, samples_per_image]
+	   _compression: 3
+       _FillValue: 65535
+       long_name: Detector pixel values
+       coverage_content_type: image
+       units: '1'
+       valid_min: 0
+       valid_max: 65534
      /image_attributes/nr_coadditions:
        _dtype: u2
        _dims: [number_of_images]
@@ -131,20 +149,18 @@ The YAML file should be structured as follows:
        units: seconds
      stats_163:
        _dtype: stats_dtype
-       _vlen: True
        _dims: [days]
+       _vlen: True
        comment: detector map statistics (MPS=163)
    ```
 
-### Notes and ToDo:
+### Notes and ToDo
 
- * The usage of older versions of h5py may result in broken netCDF4 files
- * Explain usage of parameter '_chunks', which is currently not correctly implemented.
- * Explain that the usage of variable length data-sets may break netCDF4 compatibility
+* The layout of a HDF5 or netCDF4 file can be complex. From version 0.3.0, you can split the file definition over several YAML files and provide a list with the names of YAML files as input to H5Yaml and NcYaml.  
 
 ## Support [TBW]
 
-## Roadmap
+## Road map
 
  * Release v0.1 : stable API to read your YAML files and generate the HDF5/netCDF4 file